brd: module loaded loop: module loaded virtio_blk virtio2: [vda] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) virtio_blk virtio3: [vdb] 55264 512-byte logical blocks (28.3 MB/27.0 MiB) i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 serio: i8042 KBD port at 0x60,0x64 irq 1 serio: i8042 AUX port at 0x60,0x64 irq 12 mousedev: PS/2 mouse device common for all mice device-mapper: uevent: version 1.0.3 input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input0 device-mapper: ioctl: 4.39.0-ioctl (2018-04-03) initialised: dm-devel@redhat.com NET: Registered protocol family 17 Key type dns_resolver registered sched_clock: Marking stable (580012311, 0)->(938987672, -358975361) registered taskstats version 1 EXT4-fs (vda): mounted filesystem without journal. Opts: (null) VFS: Mounted root (ext4 filesystem) readonly on device 254:0. devtmpfs: mounted debug: unmapping init [mem 0xffffffff82556000-0xffffffff8284afff] Write protecting the kernel read-only data: 16384k debug: unmapping init [mem 0xffff880001a0d000-0xffff880001bfffff] debug: unmapping init [mem 0xffff880001eda000-0xffff880001ffffff] rodata_test: all tests were successful random: fast init done systemd[1]: systemd v241-8.git9ef65cb.fc30 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=hybrid) systemd[1]: Detected virtualization kvm. systemd[1]: Detected architecture x86-64. Welcome to Fedora 30 (Thirty)! systemd[1]: Set hostname to . systemd[1]: File /usr/lib/systemd/system/systemd-journald.service:12 configures an IP firewall (IPAddressDeny=any), but the local system does not support BPF/cgroup based firewalling. systemd[1]: Proceeding WITHOUT firewalling in effect! (This warning is only shown for the first loaded unit using IP firewalling.) random: systemd: uninitialized urandom read (16 bytes read) systemd[1]: Set up automount Arbitrary Executable File Formats File System Automount Point. [ OK ] Set up automount Arbitrary…s File System Automount Point. random: systemd: uninitialized urandom read (16 bytes read) systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. random: systemd: uninitialized urandom read (16 bytes read) systemd[1]: Reached target Slices. [ OK ] Reached target Slices. systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Dispatch Password …ts to Console Directory Watch. [ OK ] Created slice system-getty.slice. [ OK ] Started Forward Password R…uests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on Journal Socket. Mounting Huge Pages File System... Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Control Socket. Starting Journal Service... [ OK ] Reached target Swap. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Load Kernel Modules... Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Load Kernel Modules. Starting Apply Kernel Variables... [ OK ] Started Apply Kernel Variables. EXT4-fs (vda): re-mounted. Opts: (null) [ OK ] Started Remount Root and Kernel File Systems. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /tmp... [ OK ] Mounted /tmp. [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Restore /run/initramfs on shutdown... Starting Load/Save Random Seed... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Load/Save Random Seed. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Journal Service. Starting Flush Journal to Persistent Storage... systemd-journald[1208]: Received request to flush runtime journal from PID 1 [ OK ] Started udev Kernel Device Manager. [ OK ] Started Flush Journal to Persistent Storage. Starting Create Volatile Files and Directories... [ OK ] Started Create Volatile Files and Directories. virtio_net virtio0 enp0s2: renamed from eth0 [ OK ] Found device /dev/hvc0. Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Entropy Daemon based on the HAVEGE algorithm. [ OK ] Reached target sshd-keygen.target. Starting Login Service... Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. Starting D-Bus System Message Bus... random: crng init done random: 7 urandom warning(s) missed due to ratelimiting [ OK ] Started D-Bus System Message Bus. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Permit User Sessions... Starting OpenSSH server daemon... [ OK ] Started Permit User Sessions. Starting Hostname Service... [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on hvc0. [ OK ] Reached target Login Prompts. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target Multi-User System. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Hostname Service. Fedora 30 (Thirty) Kernel 4.18.0 on an x86_64 (hvc0) localhost login: libcfs: loading out-of-tree module taints kernel. LNet: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 Lustre: DEBUG MARKER: tmp.fDiY2FNJWK: executing check_logdir /tmp/ltest-logs Lustre: DEBUG MARKER: tmp.fDiY2FNJWK: executing yml_node Lustre: DEBUG MARKER: Client: Lustre version: 2.13.52_139_g7605833 Lustre: DEBUG MARKER: MDS: Lustre version: 2.13.52_139_g7605833 Lustre: DEBUG MARKER: OSS: Lustre version: 2.13.52_139_g7605833 Lustre: DEBUG MARKER: excepting tests: 14 55d 77 28 Lustre: DEBUG MARKER: skipping tests SLOW=no: 33a Lustre: DEBUG MARKER: tmp.fDiY2FNJWK: executing set_hostid Lustre: Lustre: Build Version: 2.13.52_139_g7605833 LNet: Added LNI 192.168.122.98@tcp [8/256/0/180] LNet: Accept secure, port 988 Lustre: Echo OBD driver; http://www.lustre.org/ LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro ------------[ cut here ]------------ DEBUG_LOCKS_WARN_ON(current->lockdep_recursion) WARNING: CPU: 1 PID: 7478 at kernel/locking/lockdep.c:3229 __lockdep_init_map+0x16a/0x180 Modules linked in: lustre(O) ofd(O) osp(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) lov(O) mdc(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) CPU: 1 PID: 7478 Comm: mount.lustre Tainted: G O --------- --- 4.18.0 #32 Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 RIP: 0010:__lockdep_init_map+0x16a/0x180 Code: e8 ab 2c 2f 00 85 c0 74 80 8b 05 61 fd 57 02 85 c0 0f 85 72 ff ff ff 48 c7 c6 66 b6 e0 81 48 c7 c7 c1 a9 df 81 e8 25 ac fb ff <0f> 0b e9 58 ff ff ff 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 RSP: 0018:ffff880207cc7750 EFLAGS: 00010282 RAX: 000000000000002f RBX: 0000000000000000 RCX: 0000000000000006 RDX: 0000000000000007 RSI: 0000000000000001 RDI: ffff880216dd52d0 RBP: ffff8801fde21898 R08: 0000000000000001 R09: 0000000000000001 R10: 0000000000000001 R11: ffffffff83480d53 R12: ffffffffa0a9ccc0 R13: 0000000000000002 R14: 0000000000000001 R15: ffff8801fde21898 FS: 00007fadc3ef5880(0000) GS:ffff880216c00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007ffda8774000 CR3: 0000000204686000 CR4: 00000000000006a0 Call Trace: ? ldiskfs_enable_quotas+0x107/0x200 [ldiskfs] ? ldiskfs_fill_super+0x3395/0x3550 [ldiskfs] ? mount_bdev+0x176/0x1a0 ? ldiskfs_calculate_overhead+0x470/0x470 [ldiskfs] ? mount_fs+0x2d/0x15a ? vfs_kern_mount.part.0+0x47/0x140 ? osd_mount+0x47b/0xcb0 [osd_ldiskfs] ? osd_device_alloc+0x347/0x8c0 [osd_ldiskfs] ? class_setup+0x5f1/0xa30 [obdclass] ? class_process_config+0x158a/0x30f0 [obdclass] ? do_lcfg+0x15a/0x4c0 [obdclass] ? do_lcfg+0x15a/0x4c0 [obdclass] ? cache_alloc_debugcheck_after+0x138/0x150 ? __kmalloc+0x20c/0x2e0 ? do_lcfg+0x22d/0x4c0 [obdclass] ? lustre_start_simple+0x73/0x1d0 [obdclass] ? osd_start+0x59d/0x760 [obdclass] ? simple_strtoull+0x2b/0x50 ? target_name2index+0x88/0xb0 [obdclass] ? server_fill_super+0x22c/0xf80 [obdclass] ? lustre_fill_super+0xe9f/0x29f0 [obdclass] ? sget_userns+0x47a/0x4f0 ? lustre_common_put_super+0xb70/0xb70 [obdclass] ? mount_nodev+0x3c/0x90 ? mount_fs+0x2d/0x15a ? vfs_kern_mount.part.0+0x47/0x140 ? do_mount+0x1d6/0xd70 ? kmem_cache_alloc_trace+0x260/0x2c0 ? ksys_mount+0x79/0xc0 ? __x64_sys_mount+0x1c/0x20 ? do_syscall_64+0x4b/0x1a0 ? entry_SYSCALL_64_after_hwframe+0x6a/0xdf irq event stamp: 13397 hardirqs last enabled at (13397): [] kmem_cache_alloc_trace+0xba/0x2c0 hardirqs last disabled at (13396): [] kmem_cache_alloc_trace+0x76/0x2c0 softirqs last enabled at (12800): [] sync_inodes_sb+0xc5/0x430 softirqs last disabled at (12796): [] wb_queue_work+0x44/0x1b0 ---[ end trace b3d8f88e6baaca63 ]--- LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: osd-ldiskfs create tunables for lustre-MDT0000 ------------[ cut here ]------------ do not call blocking ops when !TASK_RUNNING; state=402 set at [<00000000d2020ccf>] prepare_to_wait_event+0x4f/0x110 WARNING: CPU: 0 PID: 4694 at kernel/sched/core.c:6127 __might_sleep+0x67/0x70 Modules linked in: lustre(O) ofd(O) osp(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) lov(O) mdc(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) CPU: 0 PID: 4694 Comm: ptlrpcd_rcv Tainted: G W O --------- --- 4.18.0 #32 Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 RIP: 0010:__might_sleep+0x67/0x70 Code: 41 5c 41 5d e9 3a fe ff ff 48 8b 90 d0 1c 00 00 48 c7 c7 98 ac e0 81 c6 05 d2 7a 19 01 01 48 8b 70 10 48 89 d1 e8 28 7f fd ff <0f> 0b eb c8 0f 1f 44 00 00 85 ff 75 0a 65 48 8b 04 25 40 4e 01 00 RSP: 0018:ffff88020eb3bd98 EFLAGS: 00010292 RAX: 0000000000000073 RBX: ffffffffa02271e0 RCX: 0000000000000006 RDX: 0000000000000007 RSI: 0000000000000001 RDI: ffff880216bd52d0 RBP: ffffffff81e0b409 R08: 0000000000000000 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000017 R13: 0000000000000000 R14: 0000000000000001 R15: ffff880210d94f10 FS: 0000000000000000(0000) GS:ffff880216a00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007fb697181000 CR3: 000000020e873000 CR4: 00000000000006b0 Call Trace: down_read+0x18/0xa0 keys_fill+0x15/0x100 [obdclass] lu_env_refill+0x32/0x70 [obdclass] ptlrpcd_check+0x43a/0x540 [ptlrpc] ptlrpcd+0x461/0x4c0 [ptlrpc] ? wait_woken+0x90/0x90 kthread+0x100/0x140 ? ptlrpcd_check+0x540/0x540 [ptlrpc] ? kthread_flush_work_fn+0x10/0x10 ret_from_fork+0x24/0x30 irq event stamp: 44 hardirqs last enabled at (43): [] restore_regs_and_return_to_kernel+0x0/0x22 hardirqs last disabled at (44): [] __schedule+0xb0/0xb00 softirqs last enabled at (0): [] copy_process.part.0+0x326/0x1c90 softirqs last disabled at (0): [<0000000000000000>] (null) ---[ end trace b3d8f88e6baaca64 ]--- Lustre: MGS: Connection restored to 07f579d6-983f-4 (at 192.168.122.98@tcp) Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space Lustre: lustre-MDT0000: new disk, initializing Lustre: couldn't connect with -EAGAIN Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt Lustre: lustre-MDT0000 ready for connections LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: osd-ldiskfs create tunables for lustre-MDT0001 Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 Lustre: srv-lustre-MDT0001: No data found on store. Initialize space Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0001: new disk, initializing Lustre: lustre-MDT0000: Connection restored to 07f579d6-983f-4 (at 192.168.122.98@tcp) Lustre: Skipped 1 previous similar message Lustre: 00000000146ab451 lustre-MDT0000_UUID: changing import state from CONNECTING to FULL Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] Lustre: lustre-MDT0001 ready for connections LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc Lustre: osd-ldiskfs create tunables for lustre-OST0000 Lustre: lustre-OST0000: new disk, initializing Lustre: srv-lustre-OST0000: No data found on store. Initialize space Lustre: lustre-MDT0000: Connection restored to 07f579d6-983f-4 (at 192.168.122.98@tcp) Lustre: Skipped 1 previous similar message Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc Lustre: osd-ldiskfs create tunables for lustre-OST0001 Lustre: lustre-OST0001: new disk, initializing Lustre: couldn't connect with -EAGAIN Lustre: couldn't connect with -EAGAIN Lustre: lustre-OST0000: Connection restored to 07f579d6-983f-4 (at 192.168.122.98@tcp) Lustre: Skipped 3 previous similar messages Lustre: couldn't connect with -EAGAIN Lustre: couldn't connect with -EAGAIN Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] Lustre: lustre-MDT0000: Connection restored to 07f579d6-983f-4 (at 192.168.122.98@tcp) Lustre: Skipped 4 previous similar messages Lustre: Mounted lustre-client Lustre: Mounted lustre-client Lustre: DEBUG MARKER: Using TIMEOUT=20 Lustre: DEBUG MARKER: == sanityn test 1: Check attribute updates on 2 mount points ========================================= 03:17:18 (1583187438) Lustre: DEBUG MARKER: == sanityn test 2a: check cached attribute updates on 2 mtpt's ======================================= 03:17:18 (1583187438) Lustre: DEBUG MARKER: == sanityn test 2b: check cached attribute updates on 2 mtpt's ======================================= 03:17:19 (1583187439) Lustre: DEBUG MARKER: == sanityn test 2c: check cached attribute updates on 2 mtpt's root ================================== 03:17:19 (1583187439) Lustre: DEBUG MARKER: == sanityn test 2d: check cached attribute updates on 2 mtpt's root ================================== 03:17:19 (1583187439) Lustre: DEBUG MARKER: == sanityn test 2e: check chmod on root is propagated to others ====================================== 03:17:19 (1583187439) Lustre: DEBUG MARKER: == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ==================================== 03:17:20 (1583187440) Lustre: DEBUG MARKER: == sanityn test 2g: check blocks update on sync write ================================================ 03:17:20 (1583187440) Lustre: DEBUG MARKER: == sanityn test 3: symlink on one mtpt, readlink on another ========================================== 03:17:21 (1583187441) Lustre: DEBUG MARKER: == sanityn test 4: fstat validation on multiple mount points ========================================= 03:17:21 (1583187441) Lustre: DEBUG MARKER: == sanityn test 5: create a file on one mount, truncate it on the other ============================== 03:17:23 (1583187443) Lustre: DEBUG MARKER: == sanityn test 6: remove of open file on other node ================================================= 03:17:23 (1583187443) Lustre: DEBUG MARKER: == sanityn test 7: remove of open directory on other node ============================================ 03:17:24 (1583187444) Lustre: DEBUG MARKER: == sanityn test 8: remove of open special file on other node ========================================= 03:17:24 (1583187444) Lustre: DEBUG MARKER: == sanityn test 9a: append of file with sub-page size on multiple mounts ============================= 03:17:24 (1583187444) Lustre: DEBUG MARKER: == sanityn test 9b: append to striped sparse file ==================================================== 03:17:25 (1583187445) Lustre: lfs: using old ioctl(LL_IOC_LOV_GETSTRIPE) on [0x200000402:0xf:0x0], use llapi_layout_get_by_path() Lustre: DEBUG MARKER: == sanityn test 10a: write of file with sub-page size on multiple mounts ============================= 03:17:25 (1583187445) Lustre: DEBUG MARKER: == sanityn test 10b: write of file with sub-page size on multiple mounts ============================= 03:17:26 (1583187446) Lustre: DEBUG MARKER: == sanityn test 11: execution of file opened for write should return error =========================== 03:17:26 (1583187446) Lustre: DEBUG MARKER: == sanityn test 12: test lock ordering (link, stat, unlink) ========================================== 03:17:27 (1583187447) Lustre: DEBUG MARKER: start dir: /mnt/lustre/lockdir=144115205289279510 file: /mnt/lustre/lockdir/lockfile=144115205289279508 Lustre: DEBUG MARKER: == sanityn test 13: test directory page revocation =================================================== 03:19:34 (1583187574) Lustre: DEBUG MARKER: SKIP: sanityn test_14aa skipping excluded test 14aa (base 14) Lustre: DEBUG MARKER: SKIP: sanityn test_14ab skipping excluded test 14ab (base 14) Lustre: DEBUG MARKER: SKIP: sanityn test_14b skipping excluded test 14b (base 14) Lustre: DEBUG MARKER: SKIP: sanityn test_14c skipping excluded test 14c (base 14) Lustre: DEBUG MARKER: SKIP: sanityn test_14d skipping excluded test 14d (base 14) Lustre: DEBUG MARKER: == sanityn test 15: test out-of-space with multiple writers ========================================== 03:19:35 (1583187575) Lustre: DEBUG MARKER: SKIP: sanityn test_15 oos2.sh: 4792636kB free gt MAXFREE 800000kB, increase 800000 (or reduce test fs size) to proceed Lustre: DEBUG MARKER: == sanityn test 16a: 2500 iterations of dual-mount fsx =============================================== 03:19:47 (1583187587) WARNING: CPU: 0 PID: 7956 at /home/alexey/master-mine/ldiskfs/namei.c:3439 ldiskfs_orphan_add+0xcd/0x220 [ldiskfs] Modules linked in: lustre(O) ofd(O) osp(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) lov(O) mdc(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) CPU: 0 PID: 7956 Comm: ll_ost_io00_001 Tainted: G W O --------- --- 4.18.0 #32 Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 RIP: 0010:ldiskfs_orphan_add+0xcd/0x220 [ldiskfs] Code: ad 0d 00 00 48 c7 c6 90 1e a6 a0 4c 89 ef e8 6a 19 fe ff eb 03 45 31 e4 48 83 c4 20 44 89 e0 5b 5d 41 5c 41 5d 41 5e 41 5f c3 <0f> 0b e9 7a ff ff ff 48 8d 54 24 08 48 89 de 4c 89 f7 e8 9c eb 00 RSP: 0018:ffff8801e6a33c40 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff8801da970968 RCX: 0000000000000000 RDX: ffffffff00000001 RSI: ffff8801da970968 RDI: ffff8801da970968 RBP: ffff8801ea2ec000 R08: 0000000000000000 R09: 0000000000000001 R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 R13: ffff8801e9c81000 R14: ffff880205353050 R15: ffff8801da970968 FS: 0000000000000000(0000) GS:ffff880216a00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00005582f239fb40 CR3: 00000001d4112000 CR4: 00000000000006b0 Call Trace: ? up_write+0x17/0x80 ? unmap_mapping_pages+0x4e/0x100 osd_punch+0x168/0x6f0 [osd_ldiskfs] ofd_object_punch+0x789/0x1330 [ofd] ofd_punch_hdl+0x67c/0xa50 [ofd] tgt_request_handle+0x3fe/0x1770 [ptlrpc] ptlrpc_main+0x117c/0x3170 [ptlrpc] kthread+0x100/0x140 ? ptlrpc_register_service+0x15a0/0x15a0 [ptlrpc] ? kthread_flush_work_fn+0x10/0x10 ret_from_fork+0x24/0x30 irq event stamp: 0 hardirqs last enabled at (0): [<0000000000000000>] (null) hardirqs last disabled at (0): [] copy_process.part.0+0x326/0x1c90 softirqs last enabled at (0): [] copy_process.part.0+0x326/0x1c90 softirqs last disabled at (0): [<0000000000000000>] (null) ---[ end trace b3d8f88e6baaca65 ]--- Lustre: DEBUG MARKER: sanityn test_16a: @@@@@@ FAIL: fsx with O_DIRECT failed. Lustre: DEBUG MARKER: == sanityn test 16b: 2500 iterations of dual-mount fsx at small size ================================= 03:20:26 (1583187626) Lustre: DEBUG MARKER: == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ================ 03:20:41 (1583187641) ofd: 'obdfilter.*.read_cache_enabled' is deprecated, use 'osd-*.read_cache_enabled' instead Lustre: DEBUG MARKER: == sanityn test 16d: Verify DIO and buffer IO with two clients ======================================= 03:21:08 (1583187668) Lustre: DEBUG MARKER: == sanityn test 17: resource creation/LVB creation race ============================================== 03:21:16 (1583187676) LustreError: 7512:0:(fail.c:130:__cfs_fail_timeout_set()) cfs_fail_timeout id 30a sleeping for 2000ms LustreError: 7512:0:(fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 30a awake Lustre: DEBUG MARKER: == sanityn test 18: mmap sanity check ================================================================ 03:21:18 (1583187678) Lustre: DEBUG MARKER: == sanityn test 19: test concurrent uncached read races ============================================== 03:21:24 (1583187684) ofd: 'obdfilter.*.readcache_max_filesize' is deprecated, use 'osd-*.readcache_max_filesize' instead Lustre: DEBUG MARKER: loop 5 Lustre: DEBUG MARKER: loop 10 Lustre: DEBUG MARKER: loop 15 Lustre: DEBUG MARKER: loop 20 Lustre: DEBUG MARKER: == sanityn test 20: test extra readahead page left in cache ========================================== 03:21:33 (1583187693) Lustre: DEBUG MARKER: == sanityn test 21: Try to remove mountpoint on another dir ========================================= 03:21:33 (1583187693) Lustre: DEBUG MARKER: == sanityn test 23: others should see updated atime while another read=============================== 03:21:34 (1583187694) Lustre: lustre-OST0000-osc-ffff8801dd526000: disconnect after 23s idle Lustre: lustre-OST0000: Connection restored to 07f579d6-983f-4 (at 192.168.122.98@tcp) Lustre: Skipped 7 previous similar messages Lustre: DEBUG MARKER: == sanityn test 24a: lfs df [-ih] [path] test ======================================================== 03:22:35 (1583187755) Lustre: DEBUG MARKER: == sanityn test 24b: lfs df should show both filesystems ============================================= 03:22:36 (1583187756) Lustre: DEBUG MARKER: == sanityn test 25a: change ACL on one mountpoint be seen on another ================================= 03:22:36 (1583187756) Lustre: DEBUG MARKER: == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ================ 03:22:37 (1583187757) Lustre: DEBUG MARKER: == sanityn test 26a: allow mtime to get older ======================================================== 03:22:37 (1583187757) Lustre: DEBUG MARKER: == sanityn test 26b: sync mtime between ost and mds ================================================== 03:22:39 (1583187759) Lustre: DEBUG MARKER: == sanityn test 27: align non-overlapping extent locks from request ================================== 03:22:41 (1583187761) Lustre: DEBUG MARKER: dd 1 started Lustre: DEBUG MARKER: dd 2 started Lustre: DEBUG MARKER: dd 3 finished Lustre: DEBUG MARKER: SKIP: sanityn test_28 skipping ALWAYS excluded test 28 Lustre: DEBUG MARKER: == sanityn test 30: recreate file race =============================================================== 03:22:43 (1583187763) bash[13363]: segfault at 2d6b0 ip 000000000002d6b0 sp 00007ffca5356748 error 14 in bash[5578ea0f4000+2d000] Code: Bad RIP value. Lustre: DEBUG MARKER: == sanityn test 31a: voluntary cancel / blocking ast race============================================= 03:22:46 (1583187766) Lustre: *** cfs_fail_loc=314, val=0*** Lustre: DEBUG MARKER: == sanityn test 31b: voluntary OST cancel / blocking ast race========================================= 03:22:46 (1583187766) Lustre: DEBUG MARKER: SKIP: sanityn test_31b local OST Lustre: DEBUG MARKER: == sanityn test 32a: lockless truncate =============================================================== 03:22:46 (1583187766) Lustre: DEBUG MARKER: checking cached lockless truncate Lustre: DEBUG MARKER: checking not cached lockless truncate Lustre: DEBUG MARKER: disabled lockless truncate Lustre: DEBUG MARKER: == sanityn test 32b: lockless i/o ==================================================================== 03:22:47 (1583187767) LustreError: 11-0: lustre-OST0001-osc-ffff8801d9a50000: operation ldlm_enqueue to node 192.168.122.98@tcp failed: rc = -87 Lustre: DEBUG MARKER: SKIP: sanityn test_33a skipping SLOW test 33a Lustre: DEBUG MARKER: == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ================= 03:22:48 (1583187768) Lustre: DEBUG MARKER: SKIP: sanityn test_33b Need two or more clients, have 1 Lustre: DEBUG MARKER: == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-Lock-Cancel =========================== 03:22:48 (1583187768) Lustre: DEBUG MARKER: == sanityn test 33d: DNE distributed operation should trigger COS ==================================== 03:22:50 (1583187770) Lustre: DEBUG MARKER: == sanityn test 33e: DNE local operation shouldn't trigger COS ======================================= 03:22:58 (1583187778) Lustre: DEBUG MARKER: SKIP: sanityn test_33e Need two or more clients, have 1 Lustre: DEBUG MARKER: == sanityn test 34: no lock timeout under IO ========================================================= 03:22:58 (1583187778) Lustre: *** cfs_fail_loc=512, val=0*** LustreError: 11595:0:(fail.c:130:__cfs_fail_timeout_set()) cfs_fail_timeout id 512 sleeping for 4000ms Lustre: *** cfs_fail_loc=512, val=0*** Lustre: Skipped 2 previous similar messages Lustre: *** cfs_fail_loc=512, val=0*** Lustre: *** cfs_fail_loc=512, val=0*** Lustre: Skipped 8 previous similar messages Lustre: *** cfs_fail_loc=512, val=0*** Lustre: Skipped 1 previous similar message Lustre: *** cfs_fail_loc=512, val=0*** LustreError: 7951:0:(fail.c:130:__cfs_fail_timeout_set()) cfs_fail_timeout id 512 sleeping for 4000ms Lustre: Skipped 3 previous similar messages LustreError: 7951:0:(fail.c:130:__cfs_fail_timeout_set()) Skipped 17 previous similar messages LustreError: 11595:0:(fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 512 awake Lustre: *** cfs_fail_loc=512, val=0*** Lustre: Skipped 24 previous similar messages Lustre: *** cfs_fail_loc=512, val=0*** Lustre: Skipped 3 previous similar messages LustreError: 11819:0:(fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 512 awake LustreError: 11819:0:(fail.c:141:__cfs_fail_timeout_set()) Skipped 11 previous similar messages LustreError: 11595:0:(fail.c:130:__cfs_fail_timeout_set()) cfs_fail_timeout id 512 sleeping for 4000ms LustreError: 11595:0:(fail.c:130:__cfs_fail_timeout_set()) Skipped 1 previous similar message Lustre: *** cfs_fail_loc=512, val=0*** Lustre: Skipped 27 previous similar messages LustreError: 11595:0:(fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 512 awake LustreError: 11595:0:(fail.c:141:__cfs_fail_timeout_set()) Skipped 7 previous similar messages LustreError: 11595:0:(fail.c:130:__cfs_fail_timeout_set()) cfs_fail_timeout id 512 sleeping for 4000ms LustreError: 11595:0:(fail.c:130:__cfs_fail_timeout_set()) Skipped 18 previous similar messages Lustre: lustre-OST0001-osc-ffff8801d9a50000: disconnect after 23s idle Lustre: Skipped 3 previous similar messages LustreError: 11595:0:(fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 512 awake LustreError: 11595:0:(fail.c:141:__cfs_fail_timeout_set()) Skipped 18 previous similar messages Lustre: lustre-OST0001-osc-ffff8801dd526000: disconnect after 21s idle Lustre: *** cfs_fail_loc=512, val=0*** Lustre: Skipped 53 previous similar messages Lustre: *** cfs_fail_loc=512, val=0*** LustreError: 7495:0:(fail.c:130:__cfs_fail_timeout_set()) cfs_fail_timeout id 512 sleeping for 4000ms LustreError: 7495:0:(fail.c:130:__cfs_fail_timeout_set()) Skipped 33 previous similar messages LustreError: 7495:0:(fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 512 awake LustreError: 7495:0:(fail.c:141:__cfs_fail_timeout_set()) Skipped 33 previous similar messages Lustre: lustre-OST0001: Connection restored to 07f579d6-983f-4 (at 192.168.122.98@tcp) Lustre: Skipped 3 previous similar messages LustreError: 7500:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 2s: evicting client at 192.168.122.98@tcp ns: filter-lustre-OST0001_UUID lock: 00000000e70add96/0x43e0d791411d453e lrc: 3/0,0 mode: PW/PW res: [0x25:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->4095) flags: 0x60000400000020 nid: 192.168.122.98@tcp remote: 0x43e0d791411d4530 expref: 8 pid: 16852 timeout: 429 lvb_type: 0 Lustre: lustre-OST0001-osc-ffff8801dd526000: Connection to lustre-OST0001 (at 192.168.122.98@tcp) was lost; in progress operations using this service will wait for recovery to complete LustreError: 167-0: lustre-OST0001-osc-ffff8801dd526000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. Lustre: 4695:0:(llite_lib.c:2850:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.122.98@tcp:/lustre/fid: [0x200000402:0x6a:0x0]// may get corrupted (rc -5) LustreError: 7500:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 1s: evicting client at 192.168.122.98@tcp ns: filter-lustre-OST0000_UUID lock: 00000000ac752b7e/0x43e0d791411d44c0 lrc: 3/0,0 mode: PR/PR res: [0x26:0x0:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->4095) flags: 0x60000400030020 nid: 192.168.122.98@tcp remote: 0x43e0d791411d44b9 expref: 8 pid: 11819 timeout: 433 lvb_type: 0 Lustre: lustre-OST0000-osc-ffff8801dd526000: Connection to lustre-OST0000 (at 192.168.122.98@tcp) was lost; in progress operations using this service will wait for recovery to complete LustreError: 167-0: lustre-OST0000-osc-ffff8801dd526000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. LustreError: 16857:0:(fail.c:136:__cfs_fail_timeout_set()) cfs_fail_timeout interrupted Lustre: DEBUG MARKER: tmp.fDiY2FNJWK: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8801d9a50000.ost_server_uuid osc.lustre-OST0000-osc-ffff8801dd526000.ost_server_uuid 40 Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8801d9a50000.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: tmp.fDiY2FNJWK: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8801d9a50000.ost_server_uuid osc.lustre-OST0001-osc-ffff8801dd526000.ost_server_uuid 40 Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8801d9a50000.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: tmp.fDiY2FNJWK: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8801d9a50000.ost_server_uuid osc.lustre-OST0000-osc-ffff8801dd526000.ost_server_uuid 40 Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8801d9a50000.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: tmp.fDiY2FNJWK: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8801d9a50000.ost_server_uuid osc.lustre-OST0001-osc-ffff8801dd526000.ost_server_uuid 40 Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8801d9a50000.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: tmp.fDiY2FNJWK: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8801d9a50000.ost_server_uuid osc.lustre-OST0000-osc-ffff8801dd526000.ost_server_uuid 40 Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8801d9a50000.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: tmp.fDiY2FNJWK: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8801d9a50000.ost_server_uuid osc.lustre-OST0001-osc-ffff8801dd526000.ost_server_uuid 40 Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8801d9a50000.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ============================== 03:23:37 (1583187817) Lustre: DEBUG MARKER: Race attempt 0 Lustre: DEBUG MARKER: Wait for 18244 18246 for 60 sec... Lustre: lustre-OST0000-osc-ffff8801d9a50000: disconnect after 21s idle Lustre: DEBUG MARKER: == sanityn test 36: handle ESTALE/open-unlink correctly ============================================== 03:24:39 (1583187879) Lustre: DEBUG MARKER: start test - cycle (0) Lustre: lustre-OST0000: Connection restored to 07f579d6-983f-4 (at 192.168.122.98@tcp) Lustre: Skipped 7 previous similar messages Lustre: DEBUG MARKER: start test - cycle (1) Lustre: DEBUG MARKER: start test - cycle (2) Lustre: DEBUG MARKER: start test - cycle (3) Lustre: DEBUG MARKER: start test - cycle (4) Lustre: DEBUG MARKER: start test - cycle (5) Lustre: DEBUG MARKER: start test - cycle (6) Lustre: DEBUG MARKER: start test - cycle (7) Lustre: DEBUG MARKER: start test - cycle (8) Lustre: DEBUG MARKER: start test - cycle (9) Lustre: DEBUG MARKER: start test - cycle (10) Lustre: DEBUG MARKER: == sanityn test 37: check i_size is not updated for directory on close (bug 18695) =================== 03:26:35 (1583187995) Lustre: DEBUG MARKER: == sanityn test 39a: test from 11063 ================================================================= 03:26:44 (1583188004) Lustre: DEBUG MARKER: == sanityn test 39b: 11063 problem 1 ================================================================= 03:26:45 (1583188005) Lustre: DEBUG MARKER: == sanityn test 39c: check truncate mtime update ===================================================== 03:26:46 (1583188006) Lustre: DEBUG MARKER: == sanityn test 39d: sync write should update mtime ================================================== 03:26:48 (1583188008) Lustre: *** cfs_fail_loc=411, val=0*** Lustre: DEBUG MARKER: == sanityn test 40a: pdirops: create vs others ======================================================= 03:26:48 (1583188008) LustreError: 7973:0:(fail.c:130:__cfs_fail_timeout_set()) cfs_fail_timeout id 145 sleeping for 15000ms LustreError: 7973:0:(fail.c:130:__cfs_fail_timeout_set()) Skipped 55 previous similar messages LustreError: 7973:0:(fail.c:136:__cfs_fail_timeout_set()) cfs_fail_timeout interrupted LustreError: 7973:0:(fail.c:136:__cfs_fail_timeout_set()) Skipped 17 previous similar messages Lustre: DEBUG MARKER: == sanityn test 40b: pdirops: open|create and others ================================================= 03:26:54 (1583188014) LustreError: 10525:0:(fail.c:136:__cfs_fail_timeout_set()) cfs_fail_timeout interrupted Lustre: DEBUG MARKER: == sanityn test 40c: pdirops: link and others ======================================================== 03:27:00 (1583188020) LustreError: 16853:0:(fail.c:136:__cfs_fail_timeout_set()) cfs_fail_timeout interrupted Lustre: DEBUG MARKER: == sanityn test 40d: pdirops: unlink and others ====================================================== 03:27:06 (1583188026) LustreError: 16853:0:(fail.c:136:__cfs_fail_timeout_set()) cfs_fail_timeout interrupted Lustre: DEBUG MARKER: == sanityn test 40e: pdirops: rename and others ====================================================== 03:27:11 (1583188031) Lustre: DEBUG MARKER: == sanityn test 41a: pdirops: create vs mkdir ======================================================== 03:27:16 (1583188036) Lustre: DEBUG MARKER: == sanityn test 41b: pdirops: create vs create ======================================================= 03:27:19 (1583188039) LustreError: 16856:0:(fail.c:136:__cfs_fail_timeout_set()) cfs_fail_timeout interrupted LustreError: 16856:0:(fail.c:136:__cfs_fail_timeout_set()) Skipped 2 previous similar messages Lustre: DEBUG MARKER: == sanityn test 41c: pdirops: create vs link ========================================================= 03:27:21 (1583188041) Lustre: DEBUG MARKER: == sanityn test 41d: pdirops: create vs unlink ======================================================= 03:27:23 (1583188043) LustreError: 10525:0:(fail.c:130:__cfs_fail_timeout_set()) cfs_fail_timeout id 145 sleeping for 15000ms LustreError: 10525:0:(fail.c:130:__cfs_fail_timeout_set()) Skipped 7 previous similar messages Lustre: DEBUG MARKER: == sanityn test 41e: pdirops: create and rename (tgt) ================================================ 03:27:25 (1583188045) Lustre: DEBUG MARKER: == sanityn test 41f: pdirops: create and rename (src) ================================================ 03:27:28 (1583188048) Lustre: DEBUG MARKER: == sanityn test 41g: pdirops: create vs getattr ====================================================== 03:27:30 (1583188050) Lustre: DEBUG MARKER: == sanityn test 41h: pdirops: create vs readdir ====================================================== 03:27:32 (1583188052) Lustre: DEBUG MARKER: == sanityn test 42a: pdirops: mkdir vs mkdir ========================================================= 03:27:35 (1583188055) Lustre: DEBUG MARKER: == sanityn test 42b: pdirops: mkdir vs create ======================================================== 03:27:37 (1583188057) LustreError: 7512:0:(fail.c:136:__cfs_fail_timeout_set()) cfs_fail_timeout interrupted LustreError: 7512:0:(fail.c:136:__cfs_fail_timeout_set()) Skipped 7 previous similar messages Lustre: DEBUG MARKER: == sanityn test 42c: pdirops: mkdir vs link ========================================================== 03:27:39 (1583188059) Lustre: DEBUG MARKER: == sanityn test 42d: pdirops: mkdir vs unlink ======================================================== 03:27:42 (1583188062) Lustre: DEBUG MARKER: == sanityn test 42e: pdirops: mkdir and rename (tgt) ================================================= 03:27:44 (1583188064) Lustre: DEBUG MARKER: == sanityn test 42f: pdirops: mkdir and rename (src) ================================================= 03:27:47 (1583188067) Lustre: DEBUG MARKER: == sanityn test 42g: pdirops: mkdir vs getattr ======================================================= 03:27:49 (1583188069) Lustre: DEBUG MARKER: == sanityn test 42h: pdirops: mkdir vs readdir ======================================================= 03:27:51 (1583188071) Lustre: DEBUG MARKER: == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ============================================== 03:27:53 (1583188073) Lustre: DEBUG MARKER: == sanityn test 43b: pdirops: unlink vs create ======================================================= 03:28:05 (1583188085) Lustre: DEBUG MARKER: == sanityn test 43c: pdirops: unlink vs link ========================================================= 03:28:07 (1583188087) Lustre: DEBUG MARKER: == sanityn test 43d: pdirops: unlink vs unlink ======================================================= 03:28:10 (1583188090) LustreError: 16858:0:(fail.c:136:__cfs_fail_timeout_set()) cfs_fail_timeout interrupted LustreError: 16858:0:(fail.c:136:__cfs_fail_timeout_set()) Skipped 8 previous similar messages Lustre: DEBUG MARKER: == sanityn test 43e: pdirops: unlink and rename (tgt) ================================================ 03:28:12 (1583188092) Lustre: DEBUG MARKER: == sanityn test 43f: pdirops: unlink and rename (src) ================================================ 03:28:15 (1583188095) Lustre: DEBUG MARKER: == sanityn test 43g: pdirops: unlink vs getattr ====================================================== 03:28:17 (1583188097) Lustre: DEBUG MARKER: == sanityn test 43h: pdirops: unlink vs readdir ====================================================== 03:28:19 (1583188099) Lustre: DEBUG MARKER: == sanityn test 43i: pdirops: unlink vs remote mkdir ================================================= 03:28:21 (1583188101) Lustre: DEBUG MARKER: == sanityn test 43j: racy mkdir return EEXIST ======================================================== 03:28:23 (1583188103) LustreError: 16856:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 167 sleeping LustreError: 16859:0:(libcfs_fail.h:174:cfs_race()) cfs_fail_race id 167 waking LustreError: 16856:0:(libcfs_fail.h:172:cfs_race()) cfs_fail_race id 167 awake: rc=0 LustreError: 16859:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 167 sleeping LustreError: 16859:0:(libcfs_fail.h:169:cfs_race()) Skipped 12 previous similar messages LustreError: 16859:0:(libcfs_fail.h:174:cfs_race()) cfs_fail_race id 167 waking LustreError: 16859:0:(libcfs_fail.h:174:cfs_race()) Skipped 13 previous similar messages LustreError: 16857:0:(libcfs_fail.h:172:cfs_race()) cfs_fail_race id 167 awake: rc=0 LustreError: 16857:0:(libcfs_fail.h:172:cfs_race()) Skipped 13 previous similar messages LustreError: 16857:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 167 sleeping LustreError: 16857:0:(libcfs_fail.h:169:cfs_race()) Skipped 26 previous similar messages LustreError: 16858:0:(libcfs_fail.h:174:cfs_race()) cfs_fail_race id 167 waking LustreError: 16858:0:(libcfs_fail.h:174:cfs_race()) Skipped 27 previous similar messages LustreError: 16857:0:(libcfs_fail.h:172:cfs_race()) cfs_fail_race id 167 awake: rc=0 LustreError: 16857:0:(libcfs_fail.h:172:cfs_race()) Skipped 27 previous similar messages LustreError: 16858:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 167 sleeping LustreError: 16858:0:(libcfs_fail.h:169:cfs_race()) Skipped 57 previous similar messages Lustre: DEBUG MARKER: == sanityn test 44a: pdirops: rename tgt vs mkdir ==================================================== 03:28:27 (1583188107) LustreError: 16857:0:(fail.c:130:__cfs_fail_timeout_set()) cfs_fail_timeout id 146 sleeping for 10000ms LustreError: 16857:0:(fail.c:130:__cfs_fail_timeout_set()) Skipped 20 previous similar messages Lustre: DEBUG MARKER: == sanityn test 44b: pdirops: rename tgt vs create =================================================== 03:28:29 (1583188109) Lustre: DEBUG MARKER: == sanityn test 44c: pdirops: rename tgt vs link ===================================================== 03:28:32 (1583188112) Lustre: DEBUG MARKER: == sanityn test 44d: pdirops: rename tgt vs unlink =================================================== 03:28:34 (1583188114) Lustre: DEBUG MARKER: == sanityn test 44e: pdirops: rename tgt and rename (tgt) ============================================ 03:28:36 (1583188116) Lustre: DEBUG MARKER: == sanityn test 44f: pdirops: rename tgt and rename (src) ============================================ 03:28:38 (1583188118) Lustre: DEBUG MARKER: == sanityn test 44g: pdirops: rename tgt vs getattr ================================================== 03:28:41 (1583188121) Lustre: DEBUG MARKER: == sanityn test 44h: pdirops: rename tgt vs readdir ================================================== 03:28:43 (1583188123) Lustre: DEBUG MARKER: == sanityn test 44i: pdirops: rename tgt vs remote mkdir ============================================= 03:28:45 (1583188125) Lustre: DEBUG MARKER: == sanityn test 45a: rename,mkdir doesn't return -EEXIST ============================================= 03:28:48 (1583188128) Lustre: DEBUG MARKER: == sanityn test 45b: pdirops: rename src vs create =================================================== 03:29:03 (1583188143) Lustre: DEBUG MARKER: == sanityn test 45c: pdirops: rename src vs link ===================================================== 03:29:05 (1583188145) Lustre: DEBUG MARKER: == sanityn test 45d: pdirops: rename src vs unlink =================================================== 03:29:07 (1583188147) Lustre: DEBUG MARKER: == sanityn test 45e: pdirops: rename src and rename (tgt) ============================================ 03:29:09 (1583188149) Lustre: DEBUG MARKER: == sanityn test 45f: pdirops: rename src and rename (src) ============================================ 03:29:12 (1583188152) Lustre: DEBUG MARKER: == sanityn test 45g: pdirops: rename src vs getattr ================================================== 03:29:14 (1583188154) LustreError: 16858:0:(fail.c:136:__cfs_fail_timeout_set()) cfs_fail_timeout interrupted LustreError: 16858:0:(fail.c:136:__cfs_fail_timeout_set()) Skipped 19 previous similar messages Lustre: DEBUG MARKER: == sanityn test 45h: pdirops: unlink vs readdir ====================================================== 03:29:16 (1583188156) Lustre: DEBUG MARKER: == sanityn test 45i: pdirops: rename src vs remote mkdir ============================================= 03:29:18 (1583188158) Lustre: DEBUG MARKER: == sanityn test 46a: pdirops: link vs mkdir ========================================================== 03:29:21 (1583188161) Lustre: DEBUG MARKER: == sanityn test 46b: pdirops: link vs create ========================================================= 03:29:23 (1583188163) Lustre: DEBUG MARKER: == sanityn test 46c: pdirops: link vs link =========================================================== 03:29:26 (1583188166) Lustre: DEBUG MARKER: == sanityn test 46d: pdirops: link vs unlink ========================================================= 03:29:29 (1583188169) Lustre: DEBUG MARKER: == sanityn test 46e: pdirops: link and rename (tgt) ================================================== 03:29:31 (1583188171) Lustre: DEBUG MARKER: == sanityn test 46f: pdirops: link and rename (src) ================================================== 03:29:34 (1583188174) Lustre: DEBUG MARKER: == sanityn test 46g: pdirops: link vs getattr ======================================================== 03:29:36 (1583188176) Lustre: DEBUG MARKER: == sanityn test 46h: pdirops: link vs readdir ======================================================== 03:29:39 (1583188179) Lustre: DEBUG MARKER: == sanityn test 46i: pdirops: link vs remote mkdir =================================================== 03:29:41 (1583188181) Lustre: DEBUG MARKER: == sanityn test 47a: pdirops: remote mkdir vs mkdir ================================================== 03:29:43 (1583188183) Lustre: DEBUG MARKER: == sanityn test 47b: pdirops: remote mkdir vs create ================================================= 03:29:46 (1583188186) Lustre: DEBUG MARKER: == sanityn test 47c: pdirops: remote mkdir vs link =================================================== 03:29:49 (1583188189) Lustre: DEBUG MARKER: == sanityn test 47d: pdirops: remote mkdir vs unlink ================================================= 03:29:52 (1583188192) Lustre: DEBUG MARKER: == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================== 03:29:54 (1583188194) Lustre: DEBUG MARKER: == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================== 03:29:56 (1583188196) Lustre: DEBUG MARKER: == sanityn test 47g: pdirops: remote mkdir vs getattr ================================================ 03:29:58 (1583188198) Lustre: DEBUG MARKER: == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ================================================ 03:30:01 (1583188201) LustreError: 8711:0:(fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 410 awake LustreError: 8711:0:(fail.c:141:__cfs_fail_timeout_set()) Skipped 37 previous similar messages Lustre: DEBUG MARKER: == sanityn test 51a: layout lock: refresh layout should work ========================================= 03:30:06 (1583188206) Lustre: DEBUG MARKER: == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ================ 03:30:09 (1583188209) Lustre: DEBUG MARKER: == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned =============== 03:30:16 (1583188216) Lustre: DEBUG MARKER: == sanityn test 51d: layout lock: losing layout lock should clean up memory map region =============== 03:30:20 (1583188220) Lustre: DEBUG MARKER: == sanityn test 54: rename locking =================================================================== 03:30:22 (1583188222) Lustre: lustre-OST0001-osc-ffff8801dd526000: disconnect after 23s idle LustreError: 10525:0:(fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 153 awake LustreError: 10525:0:(fail.c:141:__cfs_fail_timeout_set()) Skipped 7 previous similar messages LustreError: 10525:0:(fail.c:130:__cfs_fail_timeout_set()) cfs_fail_timeout id 154 sleeping for 5000ms LustreError: 10525:0:(fail.c:130:__cfs_fail_timeout_set()) Skipped 37 previous similar messages Lustre: DEBUG MARKER: == sanityn test 55a: rename vs unlink target dir ===================================================== 03:30:43 (1583188243) Lustre: DEBUG MARKER: == sanityn test 55b: rename vs unlink source dir ===================================================== 03:30:49 (1583188249) Lustre: DEBUG MARKER: == sanityn test 55c: rename vs unlink orphan target dir ============================================== 03:30:54 (1583188254) Lustre: DEBUG MARKER: SKIP: sanityn test_55d skipping excluded test 55d Lustre: DEBUG MARKER: == sanityn test 60: Verify data_version behaviour ==================================================== 03:31:05 (1583188265) Lustre: lustre-OST0001: Connection restored to 07f579d6-983f-4 (at 192.168.122.98@tcp) Lustre: DEBUG MARKER: == sanityn test 70a: cd directory Lustre: DEBUG MARKER: == sanityn test 70b: remove files after calling rm_entry ============================================= 03:31:06 (1583188266) Lustre: DEBUG MARKER: == sanityn test 71a: correct file map just after write operation is finished ========================= 03:31:06 (1583188266) Lustre: DEBUG MARKER: == sanityn test 71b: check fiemap support for stripecount > 1 ======================================== 03:31:07 (1583188267) Lustre: DEBUG MARKER: == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ======================= 03:31:07 (1583188267) Lustre: DEBUG MARKER: == sanityn test 73: getxattr should not cause xattr lock cancellation ================================ 03:31:08 (1583188268) Lustre: DEBUG MARKER: == sanityn test 74: flock deadlock: different mounts ================================================= 03:31:08 (1583188268) LustreError: 11-0: lustre-MDT0000-mdc-ffff8801dd526000: operation ldlm_enqueue to node 192.168.122.98@tcp failed: rc = -35 Lustre: DEBUG MARKER: == sanityn test 75: osc: upcall after unuse lock====================================================== 03:31:13 (1583188273) Lustre: DEBUG MARKER: == sanityn test 76: Verify MDT open_files listing ==================================================== 03:31:18 (1583188278) Lustre: lustre-OST0000-osc-ffff8801d9a50000: disconnect after 23s idle Lustre: DEBUG MARKER: SKIP: sanityn test_77a skipping excluded test 77a (base 77) Lustre: DEBUG MARKER: SKIP: sanityn test_77b skipping excluded test 77b (base 77) Lustre: DEBUG MARKER: SKIP: sanityn test_77c skipping excluded test 77c (base 77) Lustre: DEBUG MARKER: SKIP: sanityn test_77d skipping excluded test 77d (base 77) Lustre: DEBUG MARKER: SKIP: sanityn test_77e skipping excluded test 77e (base 77) Lustre: DEBUG MARKER: SKIP: sanityn test_77f skipping excluded test 77f (base 77) Lustre: DEBUG MARKER: SKIP: sanityn test_77g skipping excluded test 77g (base 77) Lustre: DEBUG MARKER: SKIP: sanityn test_77h skipping excluded test 77h (base 77) Lustre: DEBUG MARKER: SKIP: sanityn test_77i skipping excluded test 77i (base 77) Lustre: DEBUG MARKER: SKIP: sanityn test_77j skipping excluded test 77j (base 77) Lustre: DEBUG MARKER: SKIP: sanityn test_77ja skipping excluded test 77ja (base 77) Lustre: DEBUG MARKER: SKIP: sanityn test_77k skipping excluded test 77k (base 77) Lustre: DEBUG MARKER: SKIP: sanityn test_77l skipping excluded test 77l (base 77) Lustre: DEBUG MARKER: SKIP: sanityn test_77m skipping excluded test 77m (base 77) Lustre: DEBUG MARKER: SKIP: sanityn test_77n skipping excluded test 77n (base 77) Lustre: DEBUG MARKER: == sanityn test 78: Enable policy and specify tunings right away ===================================== 03:32:04 (1583188324) BUG: sleeping function called from invalid context at /home/alexey/master-mine/libcfs/libcfs/hash.c:1156 in_atomic(): 1, irqs_disabled(): 0, pid: 13502, name: lctl INFO: lockdep is turned off. CPU: 0 PID: 13502 Comm: lctl Tainted: G W O --------- --- 4.18.0 #32 Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 Call Trace: dump_stack+0x85/0xc0 ___might_sleep.cold+0xac/0xbc ? cfs_hash_putref+0x25c/0x4a0 [libcfs] ? nrs_orr_stop+0x60/0x240 [ptlrpc] ? nrs_policy_stop0+0x30/0x1b0 [ptlrpc] ? nrs_policy_stop_primary.isra.0+0x173/0x1c0 [ptlrpc] ? nrs_policy_start_locked+0x487/0x600 [ptlrpc] ? nrs_policy_find_locked+0x31/0x50 [ptlrpc] ? nrs_policy_ctl+0x186/0x280 [ptlrpc] ? ptlrpc_nrs_policy_control+0xea/0x280 [ptlrpc] ? ptlrpc_lprocfs_nrs_seq_write+0x439/0x540 [ptlrpc] ? full_proxy_write+0x4b/0x70 ? __vfs_write+0x1f/0x160 ? rcu_sync_lockdep_assert+0x9/0x50 ? __sb_start_write+0x13f/0x1a0 ? vfs_write+0x183/0x1b0 ? vfs_write+0xba/0x1b0 ? ksys_write+0x3d/0xa0 ? do_syscall_64+0x4b/0x1a0 ? entry_SYSCALL_64_after_hwframe+0x6a/0xdf Lustre: DEBUG MARKER: == sanityn test 79: xattr: intent error ============================================================== 03:32:05 (1583188325) LustreError: 16858:0:(fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 160 awake LustreError: 16858:0:(fail.c:141:__cfs_fail_timeout_set()) Skipped 6 previous similar messages Lustre: *** cfs_fail_loc=131, val=0*** Lustre: DEBUG MARKER: == sanityn test 80a: migrate directory when some children is being opened ============================ 03:32:16 (1583188336) Lustre: 7511:0:(osd_internal.h:1249:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 502 < left 521, rollback = 2 Lustre: 7511:0:(osd_handler.c:1796:osd_trans_dump_creds()) create: 3/24/5, destroy: 0/0/0 Lustre: 7511:0:(osd_handler.c:1803:osd_trans_dump_creds()) attr_set: 2/2/0, xattr_set: 7/521/0 Lustre: 7511:0:(osd_handler.c:1810:osd_trans_dump_creds()) write: 5/136/0, punch: 0/0/0, quota 6/6/0 Lustre: 7511:0:(osd_handler.c:1820:osd_trans_dump_creds()) insert: 12/195/5, delete: 0/0/0 Lustre: 7511:0:(osd_handler.c:1827:osd_trans_dump_creds()) ref_add: 6/6/0, ref_del: 0/0/0 Lustre: DEBUG MARKER: == sanityn test 80b: Accessing directory during migration ============================================ 03:32:19 (1583188339) Lustre: 10525:0:(osd_internal.h:1249:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 500 < left 688, rollback = 2 Lustre: 10525:0:(osd_internal.h:1249:osd_trans_exec_op()) Skipped 3 previous similar messages Lustre: 10525:0:(osd_handler.c:1796:osd_trans_dump_creds()) create: 3/24/6, destroy: 0/0/0 Lustre: 10525:0:(osd_handler.c:1796:osd_trans_dump_creds()) Skipped 3 previous similar messages Lustre: 10525:0:(osd_handler.c:1803:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 7/688/0 Lustre: 10525:0:(osd_handler.c:1803:osd_trans_dump_creds()) Skipped 3 previous similar messages Lustre: 10525:0:(osd_handler.c:1810:osd_trans_dump_creds()) write: 5/136/0, punch: 0/0/0, quota 6/6/0 Lustre: 10525:0:(osd_handler.c:1810:osd_trans_dump_creds()) Skipped 3 previous similar messages Lustre: 10525:0:(osd_handler.c:1820:osd_trans_dump_creds()) insert: 12/215/5, delete: 1/1/1 Lustre: 10525:0:(osd_handler.c:1820:osd_trans_dump_creds()) Skipped 3 previous similar messages Lustre: 10525:0:(osd_handler.c:1827:osd_trans_dump_creds()) ref_add: 5/5/0, ref_del: 0/0/0 Lustre: 10525:0:(osd_handler.c:1827:osd_trans_dump_creds()) Skipped 3 previous similar messages Lustre: DEBUG MARKER: == sanityn test 81a: rename and stat under striped directory ========================================= 03:32:20 (1583188340) Lustre: DEBUG MARKER: == sanityn test 81b: rename under striped directory doesn't deadlock ================================= 03:32:21 (1583188341) Lustre: lustre-OST0000-osc-ffff8801d9a50000: disconnect after 22s idle Lustre: Skipped 1 previous similar message Lustre: DEBUG MARKER: == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================== 03:32:53 (1583188373) Lustre: DEBUG MARKER: == sanityn test 83: access striped directory while it is being created/unlinked ====================== 03:32:54 (1583188374) Lustre: DEBUG MARKER: == sanityn test 84: 0-nlink race in lu_object_find() ================================================= 03:34:54 (1583188494) LustreError: 7510:0:(libcfs_fail.h:196:cfs_race_wait()) cfs_race id 60b sleeping LustreError: 7511:0:(libcfs_fail.h:212:cfs_race_wakeup()) cfs_fail_race id 60b waking LustreError: 7510:0:(libcfs_fail.h:199:cfs_race_wait()) cfs_fail_race id 60b awake: rc=0 Lustre: DEBUG MARKER: == sanityn test 90: open/create and unlink striped directory ========================================= 03:35:00 (1583188500) Lustre: lustre-OST0000: Connection restored to 07f579d6-983f-4 (at 192.168.122.98@tcp) Lustre: Skipped 2 previous similar messages Lustre: dir [0x240000402:0x771:0x0] stripe 0 readdir failed: -2, directory is partially accessed! Lustre: DEBUG MARKER: == sanityn test 91: chmod and unlink striped directory =============================================== 03:38:00 (1583188680) Lustre: lustre-OST0000-osc-ffff8801d9a50000: disconnect after 23s idle Lustre: DEBUG MARKER: == sanityn test 92: create remote directory under orphan directory =================================== 03:41:01 (1583188861) Lustre: DEBUG MARKER: == sanityn test 93: alloc_rr should not allocate on same ost ========================================= 03:41:02 (1583188862) LustreError: 16853:0:(fail.c:130:__cfs_fail_timeout_set()) cfs_fail_timeout id 163 sleeping for 2000ms LustreError: 16853:0:(fail.c:130:__cfs_fail_timeout_set()) Skipped 6 previous similar messages LustreError: 16853:0:(fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 163 awake Lustre: DEBUG MARKER: == sanityn test 94: signal vs CP callback race ======================================================= 03:41:07 (1583188867) Lustre: lustre-OST0000: Connection restored to 07f579d6-983f-4 (at 192.168.122.98@tcp) Lustre: DEBUG MARKER: write Lustre: DEBUG MARKER: kill 3518 Lustre: DEBUG MARKER: == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ===================== 03:41:16 (1583188876) Lustre: DEBUG MARKER: SKIP: sanityn test_100a Reserved for glimpse-ahead Lustre: DEBUG MARKER: == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ====================== 03:41:16 (1583188876) Lustre: DEBUG MARKER: == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ============================= 03:41:17 (1583188877) Lustre: DEBUG MARKER: == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ==================== 03:41:17 (1583188877) Lustre: DEBUG MARKER: == sanityn test 100e: DoM: read on open and file size ================================================ 03:41:18 (1583188878) Lustre: DEBUG MARKER: == sanityn test 101a: Discard DoM data on unlink ===================================================== 03:41:18 (1583188878) Lustre: DEBUG MARKER: sanityn test_101a: @@@@@@ FAIL: Found WRITE RPC but expect none Lustre: DEBUG MARKER: == sanityn test 101b: Discard DoM data on rename ===================================================== 03:41:19 (1583188879) Lustre: DEBUG MARKER: == sanityn test 101c: Discard DoM data on close-unlink =============================================== 03:41:20 (1583188880) Lustre: DEBUG MARKER: == sanityn test 102: Test open by handle of unlinked file ============================================ 03:41:21 (1583188881) Lustre: DEBUG MARKER: == sanityn test 103: Test size correctness with lockahead ============================================ 03:41:22 (1583188882) Lustre: *** cfs_fail_loc=415, val=0*** Lustre: DEBUG MARKER: == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close =========================== 03:41:27 (1583188887) Lustre: DEBUG MARKER: == sanityn test 105: Glimpse and lock cancel race ==================================================== 03:41:42 (1583188902) Lustre: DEBUG MARKER: cleanup: ====================================================== Lustre: DEBUG MARKER: == sanityn test complete, duration 1593 sec ========================================================== 03:43:04 (1583188984) Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request Lustre: Unmounted lustre-client Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request Lustre: Skipped 3 previous similar messages Lustre: Unmounted lustre-client LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 192.168.122.98@tcp failed: rc = -107 Lustre: 00000000146ab451 lustre-MDT0000_UUID: changing import state from FULL to DISCONN Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 192.168.122.98@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: 00000000146ab451 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING Lustre: lustre-MDT0000: Not available for connect from 192.168.122.98@tcp (stopping) Lustre: 00000000146ab451 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN Lustre: 00000000146ab451 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 192.168.122.98@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 192.168.122.98@tcp (stopping) Lustre: 00000000146ab451 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN Lustre: 00000000146ab451 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING Lustre: lustre-MDT0000: Not available for connect from 192.168.122.98@tcp (stopping) Lustre: Skipped 3 previous similar messages Lustre: 00000000146ab451 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 2. Is it stuck? Lustre: server umount lustre-MDT0000 complete Lustre: 00000000146ab451 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.122.98@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.122.98@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: 00000000146ab451 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN LustreError: 10526:0:(ldlm_lockd.c:2475:ldlm_cancel_handler()) ldlm_cancel from 192.168.122.98@tcp arrived at 1583189063 with bad export cookie 4891146214184186323 Lustre: 00000000146ab451 lustre-MDT0000_UUID: changing import state from DISCONN to CLOSED LustreError: 166-1: MGC192.168.122.98@tcp: Connection to MGS (at 192.168.122.98@tcp) was lost; in progress operations using this service will fail LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.122.98@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 192.168.122.98@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 192.168.122.98@tcp (stopping) Lustre: Skipped 3 previous similar messages Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-OST0000 complete Lustre: server umount lustre-OST0001 complete LNet: Removed LNI 192.168.122.98@tcp