[ 0.518666] random: fast init done [ 0.518806] random: crng init done [ 0.536402] brd: module loaded [ 0.544758] loop: module loaded [ 0.558558] virtio_blk virtio3: [vda] 1526344 512-byte logical blocks (781 MB/745 MiB) [ 0.558722] vda: detected capacity change from 0 to 781488128 [ 0.561972] virtio_blk virtio4: [vdb] 82120 512-byte logical blocks (42.0 MB/40.1 MiB) [ 0.562223] vdb: detected capacity change from 0 to 42045440 [ 0.566776] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 0.575499] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 0.575627] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 0.575908] device-mapper: uevent: version 1.0.3 [ 0.576172] device-mapper: ioctl: 4.43.0-ioctl (2020-10-01) initialised: dm-devel@redhat.com [ 0.577168] NET: Registered protocol family 10 [ 0.578061] Segment Routing with IPv6 [ 0.578192] NET: Registered protocol family 17 [ 0.579898] sched_clock: Marking stable (576777375, 0)->(960372838, -383595463) [ 0.580246] registered taskstats version 1 [ 0.581691] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready [ 0.610230] Sending DHCP requests ., OK [ 0.650592] IP-Config: Got DHCP answer from 192.168.120.1, my address is 192.168.121.234 [ 0.650652] IP-Config: Complete: [ 0.650686] device=eth0, hwaddr=9a:e1:28:25:d1:f3, ipaddr=192.168.121.234, mask=255.255.248.0, gw=192.168.120.1 [ 0.650751] host=192.168.121.234, domain=, nis-domain=(none) [ 0.650795] bootserver=192.168.120.1, rootserver=192.168.120.1, rootpath= [ 0.650796] nameserver0=192.168.120.1 [ 0.654108] VFS: Mounted root (squashfs filesystem) readonly on device 254:0. [ 0.660514] devtmpfs: mounted [ 0.660637] debug: unmapping init [mem 0xffffffff87c03000-0xffffffff87dfffff] [ 0.660770] debug: unmapping init [mem 0xffffffff871d0000-0xffffffff874bdfff] [ 0.680614] Write protecting the kernel read-only data: 14336k [ 0.680947] debug: unmapping init [mem 0xffff8f7523808000-0xffff8f75239fffff] [ 0.681057] debug: unmapping init [mem 0xffff8f7523c5b000-0xffff8f7523dfffff] [ 0.985759] systemd[1]: /etc/systemd/system.conf:69: Invalid log level'error': Invalid argument [ 0.986626] systemd[1]: systemd 239 (239-51.el8_5.2) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 0.987055] systemd[1]: Detected virtualization kvm. [ 0.987149] systemd[1]: Detected architecture x86-64. Welcome to CentOS Linux 8! [ 0.988264] systemd[1]: Set hostname to . [ 1.426646] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. [ 1.441318] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 1.500564] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. [ 1.518879] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ 1.519780] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ 1.523911] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Configure read-only root support... Mounting Kernel Debug File System... Mounting /tmp... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Network is Online. [ OK ] Set up automount Arbitrary Executab…rmats File System Automount Point. [ 1.597489] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready Starting Load Kernel Modules... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Starting Journal Service... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted /tmp. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Load Kernel Modules. Starting Apply Kernel Variables... Starting Create Static Device Nodes in /dev... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [[ 2.097593] systemd-sysctl[767]: Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory [ 2.098354] systemd-sysctl[767]: Couldn't write 'fq_codel' to 'net/core/default_qdisc', ignoring: No such file or directory  OK ] Started Journal Service. [ 2.210091] systemd[1]: Started udev Coldplug all Devices. [ 2.266279] systemd-udevd[773]: Specified user 'tss' unknown [ 2.274054] systemd-udevd[773]: Specified group 'tss' unknown [ OK ] Started udev Coldplug all Devices. [ 2.314222] systemd[1]: Started udev Kernel Device Manager. [ OK ] Started udev Kernel Device Manager. [ 2.653042] systemd-udevd[1032]: Using default interface naming scheme 'rhel-8.0'. [ 2.686596] systemd-udevd[1032]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable. [ 2.686978] systemd-udevd[1032]: Error changing net interface name 'eth0' to 'enp0s2': Device or resource busy [ 2.687259] systemd-udevd[1032]: could not rename interface '2' from 'eth0' to 'enp0s2': Device or resource busy [ 2.866401] systemd-udevd[1043]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable. [ OK ] Started Configure read-only root support. [ 3.748960] systemd[1]: Started Configure read-only root support. [[ 3.749624] systemd[1]: Reached target Local File Systems.  OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ 3.768681] systemd[1]: Starting Mark the need to relabel after reboot... [ 3.769007] systemd[1]: Starting Create Volatile Files and Directories... [ 3.799774] touch[1144]: touch: cannot touch '/.autorelabel': Read-only file system [ OK ] Started Mark the need to relabel after reboot. [ 3.814994] systemd[1]: Started Mark the need to relabel after reboot. [ 3.838593] systemd-tmpfiles[1145]: /var/spool does not exist and cannot be created as the file system is read-only. [ 3.838897] systemd[1]: Started Create Volatile Files and Directories. [ OK ] Started Create Volatile Files and Directories. [ 3.847752] systemd[1]: Starting RPC Bind... Starting RPC Bind... [ 3.848518] systemd[1]: Reached target System Initialization. [ OK ] Reached target System Initialization. [ 3.865633] systemd[1]: Listening on D-Bus System Message Bus Socket. [ OK ] Listening on D-Bus System Message Bus Socket. [ 3.866115] systemd[1]: Reached target Sockets. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ 3.866682] systemd[1]: Reached target Basic System. Starting Permit User Sessions... [ 3.914121] systemd[ OK ] Started D-Bus System Message Bus. [1]: Starting Permit User Sessions... [ 3.919968] systemd[1]: Started D-Bus System Message Bus. [ 3.931095] systemd[1]: Reached target sshd-keygen.target. [ OK ] Reached target sshd-keygen.target. Starting OpenSSH server daemon... Starting /etc/rc.d/rc.local Compatibility... [ OK ] Started Permit User Sessions. [ 3.955449] systemd[1]: Starting OpenSSH server daemon... [ 4.018216] systemd[1]: Starting /etc/rc.d/rc.local Compatibility... [ 4.019502] systemd[1]: Started Permit User Sessions. [ 4.088434] [systemd[1]: Started /etc/rc.d/rc.local Compatibility.  OK ] Started /etc/rc.d/rc.local Compatibility. [ OK ] Started OpenSSH server daemon.[ 4.096544] systemd[1]: Started OpenSSH server daemon. [ 4.097011] systemd[ OK ] [1]: Started RPC Bind. Started RPC Bind. [[ 4.098247] systemd[1]: Reached target Multi-User System.  OK ] Reached target Multi-User System. [ 4.140334] systemd[1]: Startup finished in 947ms (kernel) + 3.192s (userspace) = 4.140s. [ 7.312300] /dev/vdb: Can't open blockdev [ 8.020773] spl: loading out-of-tree module taints kernel. [ 8.048178] systemd-udevd[773]: Specified user 'tss' unknown [ 8.048531] systemd-udevd[773]: Specified group 'tss' unknown [ 8.440287] ZFS: Loaded module v2.1.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 14.215435] libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 [ 15.149850] Lustre: DEBUG MARKER: tmp.136UuqwYDI: executing check_logdir /tmp/ltest-logs [ 15.784583] Lustre: DEBUG MARKER: tmp.136UuqwYDI: executing yml_node [ 16.884484] Lustre: DEBUG MARKER: Client: 2.15.57 [ 17.037094] Lustre: DEBUG MARKER: MDS: 2.15.57 [ 17.207446] Lustre: DEBUG MARKER: OSS: 2.15.57 [ 17.395752] Lustre: DEBUG MARKER: excepting tests: 17m 27D 32 43 45 51f 56xc 65a 65k 77g 101g 102 123ac 134a 209 225 231a 255c 400a 413 420 422 801 802 817 129 160s 901 398b 803a 42a 42c 42b 118c 118d 407 123ad 817 411 130b 130c 130d 130e 130f 130g 312 [ 17.481898] Lustre: DEBUG MARKER: skipping tests SLOW=no: 27m 60i 64b 68 71 135 136 230d 300o 51b [ 18.969501] Lustre: DEBUG MARKER: tmp.136UuqwYDI: executing set_hostid [ 19.114026] systemd-udevd[773]: Specified user 'tss' unknown [ 19.119705] systemd-udevd[773]: Specified group 'tss' unknown [ 19.157499] systemd-udevd[2896]: Using default interface naming scheme 'rhel-8.0'. [ 19.521921] Lustre: Lustre: Build Version: 2.15.57 [ 19.648864] LNet: Added LNI 192.168.121.234@tcp [8/256/0/180] [ 19.649241] LNet: Accept secure, port 988 [ 20.323532] Lustre: Echo OBD driver; http://www.lustre.org/ [ 21.105422] systemd[1]: usr-sbin-mount.lustre.mount: Succeeded. [ 30.595403] systemd-udevd[773]: Specified user 'tss' unknown [ 30.597502] systemd-udevd[773]: Specified group 'tss' unknown [ 30.774634] systemd-udevd[5382]: Using default interface naming scheme 'rhel-8.0'. [ 33.403346] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 34.656938] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 34.841360] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 [ 34.886968] Lustre: lustre-MDT0000: new disk, initializing [ 35.017338] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 35.047358] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 38.037917] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 [ 38.065615] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 [ 38.065817] Lustre: Skipped 1 previous similar message [ 38.091448] Lustre: lustre-MDT0001: new disk, initializing [ 38.392798] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 38.450089] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 38.450434] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 41.794997] Lustre: lustre-OST0000: new disk, initializing [ 41.803045] Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 [ 41.870215] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 43.081548] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 43.091518] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 43.183843] ------------[ cut here ]------------ [ 43.184269] do not call blocking ops when !TASK_RUNNING; state=1 set at [<00000000731f4978>] prepare_to_wait_event+0x76/0x100 [ 43.184549] WARNING: CPU: 0 PID: 6754 at kernel/sched/core.c:6700 __might_sleep+0x63/0x70 [ 43.184730] Modules linked in: lustre(O) ofd(O) osp(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_zfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) zfs(O) zunicode(O) zzstd(O) zlua(O) zcommon(O) znvpair(O) zavl(O) icp(O) spl(O) [ 43.185319] CPU: 0 PID: 6754 Comm: lod0000_rec0001 Tainted: G O --------- - - 4.18.0 #2 [ 43.185521] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 43.185745] RIP: 0010:__might_sleep+0x63/0x70 [ 43.185866] Code: 5b 5d 41 5c e9 4e ff ff ff 48 8b 90 48 1d 00 00 48 c7 c7 18 fb ba 86 c6 05 17 11 e9 00 01 48 8b 70 10 48 89 d1 e8 23 f3 fd ff <0f> 0b eb ca 66 0f 1f 84 00 00 00 00 00 85 ff 75 0a 65 48 8b 04 25 [ 43.186262] RSP: 0018:ffff8f7602bf3998 EFLAGS: 00010292 [ 43.186379] RAX: 0000000000000071 RBX: ffffffff86bbe474 RCX: 0000000000000007 [ 43.186556] RDX: 0000000000000007 RSI: 0000000000000006 RDI: ffff8f762cbe5450 [ 43.186729] RBP: 0000000000000202 R08: 0000000b5f1d63a5 R09: 0000000000000000 [ 43.186901] R10: 0000000000000001 R11: 00000000ffffffff R12: 0000000000000000 [ 43.187074] R13: ffff8f760a6ee100 R14: ffff8f760ebefa80 R15: ffff8f7602ab4a80 [ 43.187247] FS: 0000000000000000(0000) GS:ffff8f762ca00000(0000) knlGS:0000000000000000 [ 43.187419] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 43.187570] CR2: 00007fa590e668a0 CR3: 000000031a054000 CR4: 00000000000006b0 [ 43.187745] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 43.187918] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 43.188093] Call Trace: [ 43.188220] ? null_alloc_repbuf+0x137/0x2d0 [ptlrpc] [ 43.188348] __kmalloc+0xfd/0x1b0 [ 43.188475] null_alloc_repbuf+0x137/0x2d0 [ptlrpc] [ 43.188636] ptl_send_rpc+0x76d/0x1220 [ptlrpc] [ 43.188763] ? __lock_acquire.isra.16+0x210/0x5b0 [ 43.188913] ? ptlrpc_check_set+0x1ff9/0x3180 [ptlrpc] [ 43.189066] ptlrpc_check_set+0x2023/0x3180 [ptlrpc] [ 43.189220] ptlrpc_set_wait+0x45c/0x760 [ptlrpc] [ 43.189346] ? wait_woken+0xa0/0xa0 [ 43.189467] ptlrpc_queue_wait+0x7f/0x230 [ptlrpc] [ 43.189611] osp_remote_sync+0x134/0x1b0 [osp] [ 43.189741] osp_attr_get+0x570/0x810 [osp] [ 43.189841] osp_object_init+0x1a0/0x2d0 [osp] [ 43.190009] lu_object_start.isra.7+0x66/0xf0 [obdclass] [ 43.190169] lu_object_find_at+0x4e8/0xb20 [obdclass] [ 43.190300] ? _kstrtoull+0x2b/0x70 [ 43.190413] dt_locate_at+0x13/0xa0 [obdclass] [ 43.190559] llog_osd_get_cat_list+0xe0/0xde0 [obdclass] [ 43.190706] lod_sub_prep_llog+0x13d/0x838 [lod] [ 43.190839] ? lod_sub_cancel_llog+0x8d0/0x8d0 [lod] [ 43.190971] lod_sub_recovery_thread+0xd8/0xae0 [lod] [ 43.191098] ? __kthread_parkme+0x33/0x90 [ 43.191198] ? lod_sub_cancel_llog+0x8d0/0x8d0 [lod] [ 43.191323] kthread+0x129/0x140 [ 43.191417] ? kthread_flush_work_fn+0x10/0x10 [ 43.191542] ret_from_fork+0x1f/0x30 [ 43.191638] ---[ end trace 2e0a432d1825ff77 ]--- [ 43.318583] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 45.265055] Lustre: lustre-OST0001: new disk, initializing [ 45.279101] Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 [ 45.324625] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 47.319707] Lustre: Mounted lustre-client [ 49.050674] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 49.215591] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 49.230181] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 49.537613] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 50.869106] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 56.523110] Lustre: DEBUG MARKER: == sanity test 0a: touch; rm ============================= 09:42:59 (1693302179) [ 57.149500] Lustre: DEBUG MARKER: == sanity test 0b: chmod 0755 /mnt/lustre ======================================================================================= 09:43:00 (1693302180) [ 57.711089] Lustre: DEBUG MARKER: == sanity test 0c: check import proc ===================== 09:43:00 (1693302180) [ 58.294067] Lustre: DEBUG MARKER: == sanity test 0d: check export proc ======================================================================================= 09:43:01 (1693302181) [ 59.193113] Lustre: DEBUG MARKER: == sanity test 0e: Enable DNE MDT balancing for mkdir in the ROOT ========================================================== 09:43:02 (1693302182) [ 59.794557] Lustre: DEBUG MARKER: == sanity test 1: mkdir; remkdir; rmdir ================== 09:43:02 (1693302182) [ 60.447789] Lustre: DEBUG MARKER: == sanity test 2: mkdir; touch; rmdir; check file ======== 09:43:03 (1693302183) [ 60.955941] Lustre: DEBUG MARKER: == sanity test 3: mkdir; touch; rmdir; check dir ========= 09:43:04 (1693302184) [ 61.606669] Lustre: DEBUG MARKER: == sanity test 4: mkdir; touch dir/file; rmdir; checkdir (expect error) ========================================================== 09:43:04 (1693302184) [ 62.261278] Lustre: DEBUG MARKER: == sanity test 5: mkdir .../d5 .../d5/d2; chmod .../d5/d2 ========================================================== 09:43:05 (1693302185) [ 63.083000] Lustre: DEBUG MARKER: == sanity test 6a: touch f6a; chmod f6a; runas -u 500 -g 500 chmod f6a (should return error) ============================================================ 09:43:06 (1693302186) [ 63.544864] Lustre: DEBUG MARKER: == sanity test 6c: touch f6c; chown f6c; runas -u 500 -g 500 chown f6c (should return error) ============================================================ 09:43:06 (1693302186) [ 64.058038] Lustre: DEBUG MARKER: == sanity test 6e: touch+chgrp ; runas -u 500 -g 500 chgrp (should return error) ========================================================== 09:43:07 (1693302187) [ 64.593860] Lustre: DEBUG MARKER: == sanity test 6g: verify new dir in sgid dir inherits group ========================================================== 09:43:07 (1693302187) [ 65.483386] Lustre: DEBUG MARKER: == sanity test 6h: runas -u 500 -g 500 chown RUNAS_ID.0 .../ (should return error) ========================================================== 09:43:08 (1693302188) [ 66.169496] Lustre: DEBUG MARKER: == sanity test 7a: mkdir .../d7; mcreate .../d7/f; chmod .../d7/f ============================================================== 09:43:09 (1693302189) [ 66.877746] Lustre: DEBUG MARKER: == sanity test 7b: mkdir .../d7; mcreate d7/f2; echo foo > d7/f2 =============================================================== 09:43:10 (1693302190) [ 67.579782] Lustre: DEBUG MARKER: == sanity test 8: mkdir .../d8; touch .../d8/f; chmod .../d8/f ================================================================= 09:43:10 (1693302190) [ 68.310976] Lustre: DEBUG MARKER: == sanity test 9: mkdir .../d9 .../d9/d2 .../d9/d2/d3 ========================================================================== 09:43:11 (1693302191) [ 69.262443] Lustre: DEBUG MARKER: == sanity test 10: mkdir .../d10 .../d10/d2; touch .../d10/d2/f ================================================================ 09:43:12 (1693302192) [ 70.141261] Lustre: DEBUG MARKER: == sanity test 11: mkdir .../d11 d11/d2; chmod .../d11/d2 ====================================================================== 09:43:13 (1693302193) [ 71.042349] Lustre: DEBUG MARKER: == sanity test 12: touch .../d12/f; chmod .../d12/f .../d12/f ================================================================== 09:43:14 (1693302194) [ 71.958848] Lustre: DEBUG MARKER: == sanity test 13: creat .../d13/f; dd .../d13/f; > .../d13/f ================================================================== 09:43:15 (1693302195) [ 72.463734] Lustre: DEBUG MARKER: == sanity test 14: touch .../d14/f; rm .../d14/f; rm .../d14/f ================================================================= 09:43:15 (1693302195) [ 73.191972] Lustre: DEBUG MARKER: == sanity test 15: touch .../d15/f; mv .../d15/f .../d15/f2 ==================================================================== 09:43:16 (1693302196) [ 74.035071] Lustre: DEBUG MARKER: == sanity test 16: touch .../d16/f; rm -rf .../d16/f ===== 09:43:17 (1693302197) [ 74.810669] Lustre: DEBUG MARKER: == sanity test 17a: symlinks: create, remove (real) ====== 09:43:17 (1693302197) [ 75.546656] Lustre: DEBUG MARKER: == sanity test 17b: symlinks: create, remove (dangling) == 09:43:18 (1693302198) [ 76.290883] Lustre: DEBUG MARKER: == sanity test 17c: symlinks: open dangling (should return error) ========================================================== 09:43:19 (1693302199) [ 77.095923] Lustre: DEBUG MARKER: == sanity test 17d: symlinks: create dangling ============ 09:43:20 (1693302200) [ 77.931655] Lustre: DEBUG MARKER: == sanity test 17e: symlinks: create recursive symlink (should return error) ========================================================== 09:43:21 (1693302201) [ 78.680592] Lustre: DEBUG MARKER: == sanity test 17f: symlinks: long and very long symlink name ========================================================== 09:43:21 (1693302201) [ 79.499570] Lustre: DEBUG MARKER: == sanity test 17g: symlinks: really long symlink name and inode boundaries ========================================================== 09:43:22 (1693302202) [ 80.445617] Lustre: DEBUG MARKER: == sanity test 17h: create objects: lov_free_memmd() doesn't lbug ========================================================== 09:43:23 (1693302203) [ 81.137010] Lustre: DEBUG MARKER: == sanity test 17i: don't panic on short symlink (should return error) ========================================================== 09:43:24 (1693302204) [ 81.408120] Lustre: *** cfs_fail_loc=143, val=0*** [ 81.408351] LustreError: 11582:0:(symlink.c:106:ll_readlink_internal()) lustre: inode [0x240000402:0x2f:0x0]: symlink length 33 not expected 35 [ 81.922257] Lustre: DEBUG MARKER: == sanity test 17k: symlinks: rsync with xattrs enabled == 09:43:25 (1693302205) [ 82.585456] Lustre: DEBUG MARKER: == sanity test 17l: Ensure lgetxattr's returned xattr size is consistent ========================================================== 09:43:25 (1693302205) [ 83.099789] Lustre: DEBUG MARKER: SKIP: sanity test_17m skipping excluded test 17m [ 83.231458] Lustre: DEBUG MARKER: == sanity test 17n: run e2fsck against master/slave MDT which contains remote dir ========================================================== 09:43:26 (1693302206) [ 83.309360] Lustre: DEBUG MARKER: SKIP: sanity test_17n ldiskfs only test [ 83.442577] Lustre: DEBUG MARKER: == sanity test 17o: stat file with incompat LMA feature == 09:43:26 (1693302206) [ 83.707197] systemd[1]: mnt-lustre\x2dmds1.mount: Succeeded. [ 83.771290] Lustre: Failing over lustre-MDT0000 [ 83.914818] Lustre: server umount lustre-MDT0000 complete [ 84.470777] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 84.471530] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 84.477731] Lustre: Skipped 3 previous similar messages [ 89.510591] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 89.518404] LustreError: Skipped 8 previous similar messages [ 90.790130] Lustre: 3177:0:(client.c:2309:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1693302207/real 1693302207] req@00000000d6c82bf7 x1775555987016064/t0(0) o400->MGC192.168.121.234@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1693302214 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 uid:0 gid:0 job:'kworker/u4:2.0' [ 90.790356] LustreError: 166-1: MGC192.168.121.234@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 90.791640] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 90.792906] LustreError: Skipped 4 previous similar messages [ 95.831331] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 95.832966] LustreError: Skipped 4 previous similar messages [ 96.872524] Lustre: Evicted from MGS (at 192.168.121.234@tcp) after server handle changed from 0x8c25e6449edfa7eb to 0x8c25e6449edfecc5 [ 96.873055] Lustre: MGC192.168.121.234@tcp: Connection restored to (at 0@lo) [ 96.979987] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 98.331811] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 101.992205] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.121.234@tcp (at 0@lo) [ 102.016528] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 102.025906] Lustre: lustre-OST0000: deleting orphan objects from 0x280000401:9 to 0x280000401:33 [ 102.025943] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000401:10 to 0x2c0000401:33 [ 102.491586] Lustre: DEBUG MARKER: tmp.136UuqwYDI: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 102.682142] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 103.751183] Lustre: *** cfs_fail_loc=194, val=0*** [ 103.751351] Lustre: 8009:0:(osd_object.c:476:osd_check_lma()) lustre-MDT0000: unsupported incompat LMA feature(s) 0x0 for fid = [0x200000401:0x1e:0x0] [ 103.751625] LustreError: 11-0: lustre-MDT0000-mdc-ffff8f75fa73f000: operation ldlm_enqueue to node 0@lo failed: rc = -95 [ 104.229179] Lustre: DEBUG MARKER: == sanity test 18: touch .../f ; ls ... ======================================================================================== 09:43:47 (1693302227) [ 104.838623] Lustre: DEBUG MARKER: == sanity test 19a: touch .../f19 ; ls -l ... ; rm .../f19 ===================================================================== 09:43:47 (1693302227) [ 105.441121] Lustre: DEBUG MARKER: == sanity test 19b: ls -l .../f19 (should return error) ======================================================================== 09:43:48 (1693302228) [ 105.823970] Lustre: DEBUG MARKER: == sanity test 19c: runas -u 500 -g 500 touch .../f19 (should return error) ============================================================ 09:43:49 (1693302229) [ 106.206210] Lustre: DEBUG MARKER: == sanity test 19d: cat .../f19 (should return error) ======================================================================== 09:43:49 (1693302229) [ 106.542220] Lustre: DEBUG MARKER: == sanity test 20: touch .../f ; ls -l ... =============== 09:43:49 (1693302229) [ 106.943473] Lustre: DEBUG MARKER: == sanity test 21: write to dangling link ================ 09:43:50 (1693302230) [ 107.413447] Lustre: DEBUG MARKER: == sanity test 22: unpack tar archive as non-root user === 09:43:50 (1693302230) [ 108.025475] Lustre: DEBUG MARKER: == sanity test 23a: O_CREAT|O_EXCL in subdir ============= 09:43:51 (1693302231) [ 108.468867] Lustre: DEBUG MARKER: == sanity test 23b: O_APPEND check ======================= 09:43:51 (1693302231) [ 108.921801] Lustre: DEBUG MARKER: == sanity test 23c: O_APPEND size checks for tiny writes ========================================================== 09:43:52 (1693302232) [ 109.473860] Lustre: DEBUG MARKER: == sanity test 23d: file offset is correct after appending writes ========================================================== 09:43:52 (1693302232) [ 109.827778] Lustre: DEBUG MARKER: == sanity test 24a: rename file to non-existent target === 09:43:52 (1693302232) [ 110.480298] Lustre: DEBUG MARKER: == sanity test 24b: rename file to existing target ======= 09:43:53 (1693302233) [ 111.238587] Lustre: DEBUG MARKER: == sanity test 24c: rename directory to non-existent target ========================================================== 09:43:54 (1693302234) [ 112.041400] Lustre: DEBUG MARKER: == sanity test 24d: rename directory to existing target == 09:43:55 (1693302235) [ 112.850402] Lustre: DEBUG MARKER: == sanity test 24e: touch .../R5a/f; rename .../R5a/f .../R5b/g ================================================================ 09:43:55 (1693302235) [ 113.536356] Lustre: DEBUG MARKER: == sanity test 24f: touch .../R6a/f R6b/g; mv .../R6a/f .../R6b/g ============================================================== 09:43:56 (1693302236) [ 114.368029] Lustre: DEBUG MARKER: == sanity test 24g: mkdir .../R7a/d; .../R7b/d; mv .../R7a/d .../R7b/e ================================================================ 09:43:57 (1693302237) [ 114.922962] Lustre: DEBUG MARKER: == sanity test 24h: mkdir .../R8a/d; .../R8a/e; .../R8b/d; .../R8b/e; rename .../R8a/d .../R8b/e ========================================================== 09:43:58 (1693302238) [ 115.511250] Lustre: DEBUG MARKER: == sanity test 24i: rename file to dir error: touch f ; mkdir a ; rename f a ========================================================== 09:43:58 (1693302238) [ 116.318393] Lustre: DEBUG MARKER: == sanity test 24j: source does not exist ====================================================================================== 09:43:59 (1693302239) [ 116.960923] Lustre: DEBUG MARKER: == sanity test 24k: touch .../R11a/f; mv .../R11a/f .../R11a/d ================================================================= 09:44:00 (1693302240) [ 117.726763] Lustre: DEBUG MARKER: == sanity test 24l: Renaming a file to itself ================================================================================== 09:44:00 (1693302240) [ 118.115617] Lustre: DEBUG MARKER: == sanity test 24m: Renaming a file to a hard link to itself =================================================================== 09:44:01 (1693302241) [ 118.584145] Lustre: DEBUG MARKER: == sanity test 24n: Statting the old file after renaming (Posix rename 2) ========================================================== 09:44:01 (1693302241) [ 119.004676] Lustre: DEBUG MARKER: == sanity test 24o: rename of files during htree split === 09:44:02 (1693302242) [ 162.187042] Lustre: DEBUG MARKER: == sanity test 24p: mkdir .../R12a; .../R12b; rename .../R12a .../R12b ========================================================== 09:44:45 (1693302285) [ 162.829651] Lustre: DEBUG MARKER: == sanity test 24q: mkdir .../R13a; .../R13b; open R13b rename R13a R13b ============================================================= 09:44:46 (1693302286) [ 163.367196] Lustre: DEBUG MARKER: == sanity test 24r: mkdir .../R14a/b; rename .../R14a .../R14a/b =============================================================== 09:44:46 (1693302286) [ 163.848420] Lustre: DEBUG MARKER: == sanity test 24s: mkdir .../R15a/b/c; rename .../R15a .../R15a/b/c =========================================================== 09:44:47 (1693302287) [ 164.449267] Lustre: DEBUG MARKER: == sanity test 24t: mkdir .../R16a/b/c; rename .../R16a/b/c .../R16a =========================================================== 09:44:47 (1693302287) [ 164.996658] Lustre: DEBUG MARKER: == sanity test 24u: create stripe file =================== 09:44:48 (1693302288) [ 165.826000] Lustre: DEBUG MARKER: == sanity test 24v: list large directory (test hash collision, b=17560) ========================================================== 09:44:49 (1693302289) [ 199.056244] Lustre: DEBUG MARKER: == sanity test 24w: Reading a file larger than 4Gb ======= 09:45:22 (1693302322) [ 199.549359] Lustre: DEBUG MARKER: == sanity test 24x: cross MDT rename/link ================ 09:45:22 (1693302322) [ 200.532577] Lustre: DEBUG MARKER: == sanity test 24y: rename/link on the same dir should succeed ========================================================== 09:45:23 (1693302323) [ 201.316218] Lustre: DEBUG MARKER: == sanity test 24z: cross-MDT rename is done as cp ======= 09:45:24 (1693302324) [ 202.004804] Lustre: DEBUG MARKER: == sanity test 24A: readdir() returns correct number of entries. ========================================================== 09:45:25 (1693302325) [ 223.266157] Lustre: DEBUG MARKER: == sanity test 24B: readdir for striped dir return correct number of entries ========================================================== 09:45:46 (1693302346) [ 223.865455] Lustre: DEBUG MARKER: == sanity test 24C: check .. in striped dir ============== 09:45:47 (1693302347) [ 224.650439] Lustre: DEBUG MARKER: == sanity test 24E: cross MDT rename/link ================ 09:45:47 (1693302347) [ 224.734157] Lustre: DEBUG MARKER: SKIP: sanity test_24E needs >= 4 MDTs [ 224.871840] Lustre: DEBUG MARKER: == sanity test 24F: hash order vs readdir (LU-11330) ===== 09:45:48 (1693302348) [ 233.408169] Lustre: DEBUG MARKER: == sanity test 24G: migrate symlink in rename ============ 09:45:56 (1693302356) [ 234.262204] Lustre: DEBUG MARKER: == sanity test 24H: repeat FLD_QUERY rpc ================= 09:45:57 (1693302357) [ 234.411643] Lustre: DEBUG MARKER: SKIP: sanity test_24H MDT1 should be on another node [ 234.622994] Lustre: DEBUG MARKER: == sanity test 25a: create file in symlinked directory ========================================================================= 09:45:57 (1693302357) [ 235.439966] Lustre: DEBUG MARKER: == sanity test 25b: lookup file in symlinked directory ========================================================================= 09:45:58 (1693302358) [ 235.965224] Lustre: DEBUG MARKER: == sanity test 26a: multiple component symlink ================================================================================= 09:45:59 (1693302359) [ 236.925238] Lustre: DEBUG MARKER: == sanity test 26b: multiple component symlink at end of lookup ================================================================ 09:45:59 (1693302359) [ 237.602164] Lustre: DEBUG MARKER: == sanity test 26c: chain of symlinks ==================== 09:46:00 (1693302360) [ 238.318352] Lustre: DEBUG MARKER: == sanity test 26d: create multiple component recursive symlink ========================================================== 09:46:01 (1693302361) [ 238.823356] Lustre: DEBUG MARKER: == sanity test 26e: unlink multiple component recursive symlink ========================================================== 09:46:01 (1693302361) [ 239.218944] Lustre: DEBUG MARKER: == sanity test 26f: rm -r of a directory which has recursive symlink ========================================================== 09:46:02 (1693302362) [ 240.151477] Lustre: DEBUG MARKER: == sanity test 27a: one stripe file ====================== 09:46:03 (1693302363) [ 240.695254] Lustre: DEBUG MARKER: == sanity test 27b: create and write to two stripe file == 09:46:03 (1693302363) [ 241.255467] Lustre: DEBUG MARKER: == sanity test 27ca: one stripe on specified OST ========= 09:46:04 (1693302364) [ 241.965724] Lustre: DEBUG MARKER: == sanity test 27cb: two stripes on specified OSTs ======= 09:46:05 (1693302365) [ 242.840789] Lustre: DEBUG MARKER: == sanity test 27cc: two stripes on the same OST ========= 09:46:05 (1693302365) [ 243.742859] Lustre: DEBUG MARKER: == sanity test 27cd: four stripes on two OSTs ============ 09:46:06 (1693302366) [ 244.916700] Lustre: DEBUG MARKER: == sanity test 27ce: more stripes than OSTs with -o ====== 09:46:07 (1693302367) [ 245.728090] Lustre: DEBUG MARKER: == sanity test 27cf: 'setstripe -o' on inactive OSTs should return error ========================================================== 09:46:08 (1693302368) [ 245.804110] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 246.888619] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 246.889093] Lustre: Skipped 1 previous similar message [ 246.889413] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 246.889663] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 246.890104] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.121.234@tcp (at 0@lo) [ 246.890249] Lustre: Skipped 4 previous similar messages [ 247.514182] Lustre: DEBUG MARKER: == sanity test 27cg: 1000 shouldn't cause too many credits ========================================================== 09:46:10 (1693302370) [ 247.669485] Lustre: DEBUG MARKER: SKIP: sanity test_27cg ldiskfs only test [ 247.935930] Lustre: DEBUG MARKER: == sanity test 27d: create file with default settings ==== 09:46:11 (1693302371) [ 248.550347] Lustre: DEBUG MARKER: == sanity test 27e: setstripe existing file (should return error) ========================================================== 09:46:11 (1693302371) [ 249.187542] Lustre: DEBUG MARKER: == sanity test 27f: setstripe with bad stripe size (should return error) ========================================================== 09:46:12 (1693302372) [ 249.719561] Lustre: DEBUG MARKER: == sanity test 27g: /mnt/build/lustre/tests/../utils/lfs getstripe with no objects ========================================================== 09:46:12 (1693302372) [ 250.543053] Lustre: DEBUG MARKER: == sanity test 27ga: /mnt/build/lustre/tests/../utils/lfs getstripe with missing file (should return error) ========================================================== 09:46:13 (1693302373) [ 251.081511] Lustre: DEBUG MARKER: == sanity test 27i: /mnt/build/lustre/tests/../utils/lfs getstripe with some objects ========================================================== 09:46:14 (1693302374) [ 251.585041] Lustre: DEBUG MARKER: == sanity test 27j: setstripe with bad stripe offset (should return error) ========================================================== 09:46:14 (1693302374) [ 252.029237] Lustre: DEBUG MARKER: == sanity test 27k: limit i_blksize for broken user apps ========================================================== 09:46:15 (1693302375) [ 252.682824] Lustre: DEBUG MARKER: == sanity test 27l: check setstripe permissions (should return error) ========================================================== 09:46:15 (1693302375) [ 253.112943] Lustre: DEBUG MARKER: SKIP: sanity test_27m skipping SLOW test 27m [ 253.246717] Lustre: DEBUG MARKER: == sanity test 27n: create file with some full OSTs ====== 09:46:16 (1693302376) [ 261.990267] Lustre: *** cfs_fail_loc=215, val=0*** [ 274.993330] Lustre: DEBUG MARKER: == sanity test 27o: create file with all full OSTs (should error) ========================================================== 09:46:38 (1693302398) [ 281.350227] Lustre: *** cfs_fail_loc=215, val=1*** [ 283.510434] Lustre: *** cfs_fail_loc=215, val=1*** [ 283.510585] Lustre: Skipped 2 previous similar messages [ 284.790332] Lustre: *** cfs_fail_loc=215, val=0*** [ 286.390242] Lustre: *** cfs_fail_loc=215, val=1*** [ 297.204680] Lustre: DEBUG MARKER: == sanity test 27oo: don't let few threads to reserve too many objects ========================================================== 09:47:00 (1693302420) [ 324.259929] systemd[1]: mnt-lustre\x2dost1.mount: Succeeded. [ 324.300447] Lustre: Failing over lustre-OST0000 [ 324.341680] Lustre: server umount lustre-OST0000 complete [ 325.909603] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 325.909897] Lustre: Skipped 1 previous similar message [ 325.948182] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 326.513845] Lustre: DEBUG MARKER: == sanity test 27p: append to a truncated file with some full OSTs ========================================================== 09:47:29 (1693302449) [ 327.916582] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 327.981649] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 327.983180] Lustre: lustre-OST0000-osc-ffff8f75fa73f000: Connection restored to 192.168.121.234@tcp (at 0@lo) [ 327.983350] Lustre: Skipped 1 previous similar message [ 348.150236] Lustre: *** cfs_fail_loc=215, val=0*** [ 348.150352] Lustre: Skipped 3 previous similar messages [ 351.110157] Lustre: lustre-OST0000-osc-ffff8f75fa73f000: disconnect after 23s idle [ 361.067716] Lustre: DEBUG MARKER: == sanity test 27q: append to truncated file with all OSTs full (should error) ========================================================== 09:48:04 (1693302484) [ 367.030180] Lustre: *** cfs_fail_loc=215, val=1*** [ 375.513772] Lustre: *** cfs_fail_loc=215, val=0*** [ 383.577734] Lustre: DEBUG MARKER: == sanity test 27r: stripe file with some full OSTs (shouldn't LBUG) =========================================================== 09:48:26 (1693302506) [ 393.510299] Lustre: *** cfs_fail_loc=215, val=0*** [ 393.510450] Lustre: Skipped 8 previous similar messages [ 405.581981] Lustre: DEBUG MARKER: == sanity test 27s: lsm_xfersize overflow (should error) (bug 10725) ========================================================== 09:48:48 (1693302528) [ 405.977499] Lustre: DEBUG MARKER: == sanity test 27t: check that utils parse path correctly ========================================================== 09:48:49 (1693302529) [ 406.420768] Lustre: DEBUG MARKER: == sanity test 27u: skip object creation on OSC w/o objects ========================================================== 09:48:49 (1693302529) [ 406.758591] Lustre: *** cfs_fail_loc=139, val=0*** [ 407.260777] Lustre: *** cfs_fail_loc=139, val=0*** [ 407.260888] Lustre: Skipped 624 previous similar messages [ 408.261186] Lustre: *** cfs_fail_loc=139, val=0*** [ 408.261302] Lustre: Skipped 1303 previous similar messages [ 425.726160] Lustre: DEBUG MARKER: == sanity test 27v: skip object creation on slow OST ===== 09:49:08 (1693302548) [ 427.510268] Lustre: *** cfs_fail_loc=215, val=1*** [ 457.762842] Lustre: DEBUG MARKER: == sanity test 27w: check /mnt/build/lustre/tests/../utils/lfs setstripe -S and getstrip -d options ========================================================== 09:49:40 (1693302580) [ 458.199704] Lustre: DEBUG MARKER: == sanity test 27wa: check /mnt/build/lustre/tests/../utils/lfs setstripe -c -i options ========================================================== 09:49:41 (1693302581) [ 458.715306] Lustre: DEBUG MARKER: == sanity test 27x: create files while OST0 is degraded == 09:49:41 (1693302581) [ 469.329503] Lustre: DEBUG MARKER: == sanity test 27y: create files while OST0 is degraded and the rest inactive ========================================================== 09:49:52 (1693302592) [ 469.651736] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 469.651848] Lustre: Skipped 1 previous similar message [ 480.114590] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 480.114762] Lustre: Skipped 2 previous similar messages [ 480.114965] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 480.115036] Lustre: Skipped 1 previous similar message [ 480.115150] LustreError: 167-0: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 480.115379] LustreError: Skipped 1 previous similar message [ 480.115603] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 192.168.121.234@tcp (at 0@lo) [ 480.115756] Lustre: Skipped 2 previous similar messages [ 490.725819] Lustre: DEBUG MARKER: == sanity test 27z: check SEQ/OID on the MDT and OST filesystems ========================================================== 09:50:13 (1693302613) [ 496.158314] Lustre: DEBUG MARKER: check file /mnt/lustre/d27z.sanity/f27z.sanity-1 [ 496.317926] Lustre: DEBUG MARKER: FID seq 0x200000402, oid 0x2273 ver 0x0 [ 496.409793] Lustre: DEBUG MARKER: LOV seq 0x200000402, oid 0x2273, count: 1 [ 496.515812] Lustre: DEBUG MARKER: want: stripe:0 ost:0 oid:440/0x1b8 seq:0x280000401 [ 496.582888] systemd[1]: mnt-lustre\x2dost1.mount: Succeeded. [ 496.607178] Lustre: Failing over lustre-OST0000 [ 496.636073] Lustre: server umount lustre-OST0000 complete [ 497.271681] Lustre: lustre-OST0000-osc-ffff8f75fa73f000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 497.271980] Lustre: Skipped 1 previous similar message [ 497.272544] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 497.272800] LustreError: Skipped 7 previous similar messages [ 497.752950] systemd[1]: mnt-lustre\x2dost1.mount: Succeeded. [ 498.950816] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 500.324788] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 500.384442] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 500.385521] Lustre: lustre-OST0000-osc-ffff8f75fa73f000: Connection restored to 192.168.121.234@tcp (at 0@lo) [ 500.385711] Lustre: Skipped 1 previous similar message [ 500.638806] Lustre: DEBUG MARKER: check file /mnt/lustre/d27z.sanity/f27z.sanity-2 [ 500.776266] Lustre: DEBUG MARKER: FID seq 0x240000402, oid 0x4790 ver 0x0 [ 500.865001] Lustre: DEBUG MARKER: LOV seq 0x240000402, oid 0x4790, count: 2 [ 500.970989] Lustre: DEBUG MARKER: want: stripe:0 ost:1 oid:557/0x22d seq:0x2c0000400 [ 501.057467] systemd[1]: mnt-lustre\x2dost2.mount: Succeeded. [ 501.097698] Lustre: Failing over lustre-OST0001 [ 501.119148] Lustre: server umount lustre-OST0001 complete [ 502.410311] systemd[1]: mnt-lustre\x2dost2.mount: Succeeded. [ 503.531455] Lustre: lustre-OST0001: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 504.801368] Lustre: lustre-OST0001: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 504.877553] Lustre: lustre-OST0001: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 505.097409] Lustre: DEBUG MARKER: want: stripe:1 ost:0 oid:69/0x45 seq:0x280000400 [ 505.163233] systemd[1]: mnt-lustre\x2dost1.mount: Succeeded. [ 505.194914] Lustre: Failing over lustre-OST0000 [ 505.213602] Lustre: server umount lustre-OST0000 complete [ 505.430627] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 505.430834] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 505.431001] Lustre: Skipped 5 previous similar messages [ 505.431248] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 506.378307] systemd[1]: mnt-lustre\x2dost1.mount: Succeeded. [ 507.433988] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 509.276433] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 509.313426] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 509.800792] Lustre: DEBUG MARKER: == sanity test 27A: check filesystem-wide default LOV EA values ========================================================== 09:50:32 (1693302632) [ 510.305589] Lustre: DEBUG MARKER: == sanity test 27B: call setstripe on open unlinked file/rename victim ========================================================== 09:50:33 (1693302633) [ 510.916715] Lustre: DEBUG MARKER: == sanity test 27Ca: check full striping across all OSTs ========================================================== 09:50:34 (1693302634) [ 511.499905] Lustre: DEBUG MARKER: == sanity test 27Cb: more stripes than OSTs with -C ====== 09:50:34 (1693302634) [ 512.006461] Lustre: DEBUG MARKER: == sanity test 27Cc: fewer stripes than OSTs does not set overstriping ========================================================== 09:50:35 (1693302635) [ 512.515540] Lustre: DEBUG MARKER: == sanity test 27Cd: test maximum stripe count =========== 09:50:35 (1693302635) [ 512.738948] Lustre: *** cfs_fail_loc=210a, val=0*** [ 513.169791] Lustre: *** cfs_fail_loc=210a, val=0*** [ 513.252873] Lustre: *** cfs_fail_loc=210a, val=0*** [ 513.252979] Lustre: Skipped 48 previous similar messages [ 513.572127] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000bd0 [ 514.268243] Lustre: *** cfs_fail_loc=210a, val=0*** [ 514.268408] Lustre: Skipped 82 previous similar messages [ 514.309425] Lustre: *** cfs_fail_loc=210a, val=0*** [ 514.309537] Lustre: Skipped 17 previous similar messages [ 514.692196] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000bd1 [ 514.709575] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000bd0 [ 527.238342] Lustre: DEBUG MARKER: == sanity test 27Ce: test pool with overstriping ========= 09:50:50 (1693302650) [ 534.457419] Lustre: DEBUG MARKER: == sanity test 27Cf: test default inheritance with overstriping ========================================================== 09:50:57 (1693302657) [ 535.418124] Lustre: DEBUG MARKER: == sanity test 27Cg: test setstripe with wrong OST idx === 09:50:58 (1693302658) [ 535.767019] Lustre: DEBUG MARKER: SKIP: sanity test_27D skipping excluded test 27D [ 535.906258] Lustre: DEBUG MARKER: == sanity test 27E: check that default extended attribute size properly increases ========================================================== 09:50:59 (1693302659) [ 536.369893] Lustre: DEBUG MARKER: == sanity test 27F: Client resend delayed layout creation with non-zero size ========================================================== 09:50:59 (1693302659) [ 536.590129] systemd[1]: mnt-lustre\x2dost1.mount: Succeeded. [ 536.615579] Lustre: Failing over lustre-OST0000 [ 536.655893] Lustre: server umount lustre-OST0000 complete [ 536.922015] systemd[1]: mnt-lustre\x2dost2.mount: Succeeded. [ 536.970409] Lustre: Failing over lustre-OST0001 [ 537.130400] Lustre: server umount lustre-OST0001 complete [ 538.072459] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 538.072822] Lustre: Skipped 5 previous similar messages [ 538.073057] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 538.073371] LustreError: Skipped 1 previous similar message [ 538.088938] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 539.059130] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 539.123003] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to (at 0@lo) [ 539.123056] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 539.123232] Lustre: Skipped 8 previous similar messages [ 544.520103] Lustre: 3176:0:(client.c:2309:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1693302660/real 1693302660] req@00000000e4c08ac1 x1775555999848960/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1693302667 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 uid:0 gid:0 job:'kworker/u4:1.0' [ 546.630058] Lustre: 3177:0:(client.c:2309:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1693302660/real 1693302660] req@00000000dc168687 x1775555999849152/t0(0) o400->lustre-OST0001-osc-ffff8f75fa73f000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1693302667 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 uid:0 gid:0 job:'kworker/u4:1.0' [ 546.630347] Lustre: 3177:0:(client.c:2309:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 548.950064] Lustre: 5942:0:(client.c:2309:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1693302660/real 1693302660] req@0000000027cb5e51 x1775555999848704/t0(0) o10->lustre-OST0000-osc-MDT0000@0@lo:6/4 lens 440/432 e 0 to 1 dl 1693302672 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 uid:0 gid:0 job:'mdt00_002.0' [ 548.950354] LustreError: 5942:0:(osp_precreate.c:1764:osp_object_truncate()) can't punch object: -107 [ 548.950703] LustreError: 11-0: lustre-MDT0000-mdc-ffff8f75fa73f000: operation ldlm_enqueue to node 0@lo failed: rc = -107 [ 548.962753] LustreError: Skipped 1 previous similar message [ 548.963030] Lustre: lustre-MDT0000: Client 2b18f5a1-3a9c-46b9-9b9b-4e01995ee896 (at 0@lo) reconnecting [ 548.963187] Lustre: Skipped 1 previous similar message [ 549.403602] Lustre: DEBUG MARKER: == sanity test 27G: Clear OST pool from stripe =========== 09:51:12 (1693302672) [ 557.724634] Lustre: DEBUG MARKER: == sanity test 27H: Set specific OSTs stripe ============= 09:51:20 (1693302680) [ 557.822264] Lustre: DEBUG MARKER: SKIP: sanity test_27H needs >= 3 OSTs [ 557.975646] Lustre: DEBUG MARKER: == sanity test 27I: check that root dir striping does not break parent dir one ========================================================== 09:51:21 (1693302681) [ 565.144796] Lustre: DEBUG MARKER: == sanity test 27J: basic ops on file with foreign LOV === 09:51:28 (1693302688) [ 565.967439] Lustre: DEBUG MARKER: == sanity test 27K: basic ops on dir with foreign LMV ==== 09:51:29 (1693302689) [ 566.823829] Lustre: DEBUG MARKER: == sanity test 27L: lfs pool_list gives correct pool name ========================================================== 09:51:29 (1693302689) [ 571.753994] Lustre: DEBUG MARKER: == sanity test 27M: test O_APPEND striping =============== 09:51:34 (1693302694) [ 580.929293] Lustre: DEBUG MARKER: == sanity test 27N: lctl pool_list on separate MGS gives correct pool name ========================================================== 09:51:44 (1693302704) [ 581.081873] Lustre: DEBUG MARKER: SKIP: sanity test_27N needs separate MGS/MDT [ 581.258513] Lustre: DEBUG MARKER: == sanity test 27O: basic ops on foreign file of symlink type ========================================================== 09:51:44 (1693302704) [ 582.112080] Lustre: DEBUG MARKER: == sanity test 27P: basic ops on foreign dir of foreign_symlink type ========================================================== 09:51:45 (1693302705) [ 582.894465] Lustre: DEBUG MARKER: == sanity test 27Q: llapi_file_get_stripe() works on symlinks ========================================================== 09:51:46 (1693302706) [ 583.562162] Lustre: DEBUG MARKER: == sanity test 27R: test max_stripecount limitation when stripe count is set to -1 ========================================================== 09:51:46 (1693302706) [ 584.302833] Lustre: DEBUG MARKER: == sanity test 27T: no eio on close on partial write due to enosp ========================================================== 09:51:47 (1693302707) [ 584.408100] Lustre: DEBUG MARKER: SKIP: sanity test_27T need ost1 and client on different nodes [ 584.579464] Lustre: DEBUG MARKER: == sanity test 27U: append pool and stripe count work with composite default layout ========================================================== 09:51:47 (1693302707) [ 601.847591] Lustre: DEBUG MARKER: == sanity test 28: create/mknod/mkdir with bad file types ====================================================================== 09:52:05 (1693302725) [ 602.413045] Lustre: DEBUG MARKER: == sanity test 29: IT_GETATTR regression ====================================================================================== 09:52:05 (1693302725) [ 603.770972] Lustre: DEBUG MARKER: first d29 [ 603.888139] Lustre: DEBUG MARKER: second d29 [ 603.989315] Lustre: DEBUG MARKER: done [ 604.385868] Lustre: DEBUG MARKER: == sanity test 30a: execute binary from Lustre (execve) ======================================================================== 09:52:07 (1693302727) [ 604.815306] Lustre: DEBUG MARKER: == sanity test 30b: execute binary from Lustre as non-root ===================================================================== 09:52:07 (1693302727) [ 605.266814] Lustre: DEBUG MARKER: == sanity test 30c: execute binary from Lustre without read perms ============================================================== 09:52:08 (1693302728) [ 605.753283] Lustre: DEBUG MARKER: == sanity test 30d: execute binary from Lustre while clear locks ========================================================== 09:52:08 (1693302728) [ 617.592761] Lustre: DEBUG MARKER: == sanity test 31a: open-unlink file ============================================================================================ 09:52:20 (1693302740) [ 617.955399] Lustre: DEBUG MARKER: == sanity test 31b: unlink file with multiple links while open ================================================================= 09:52:21 (1693302741) [ 618.342615] Lustre: DEBUG MARKER: == sanity test 31c: open-unlink file with multiple links ======================================================================= 09:52:21 (1693302741) [ 618.765699] Lustre: DEBUG MARKER: == sanity test 31d: remove of open directory =================================================================================== 09:52:21 (1693302741) [ 619.272823] Lustre: DEBUG MARKER: == sanity test 31e: remove of open non-empty directory ========================================================================= 09:52:22 (1693302742) [ 619.668336] Lustre: DEBUG MARKER: == sanity test 31f: remove of open directory with open-unlink file ============================================================= 09:52:22 (1693302742) [ 626.392731] Lustre: DEBUG MARKER: == sanity test 31g: cross directory link================== 09:52:29 (1693302749) [ 626.964893] Lustre: DEBUG MARKER: == sanity test 31h: cross directory link under child========================================================================= 09:52:30 (1693302750) [ 627.527467] Lustre: DEBUG MARKER: == sanity test 31i: cross directory link under parent========================================================================= 09:52:30 (1693302750) [ 628.061028] Lustre: DEBUG MARKER: == sanity test 31j: link for directory==================== 09:52:31 (1693302751) [ 628.569822] Lustre: DEBUG MARKER: == sanity test 31k: link to file: the same, non-existing, dir========================================================================= 09:52:31 (1693302751) [ 629.094565] Lustre: DEBUG MARKER: == sanity test 31l: link to file: target dir has trailing slash ========================================================== 09:52:32 (1693302752) [ 629.712635] Lustre: DEBUG MARKER: == sanity test 31m: link to file: the same, non-existing, dir========================================================================= 09:52:32 (1693302752) [ 630.300770] Lustre: DEBUG MARKER: == sanity test 31n: check link count of unlinked file ==== 09:52:33 (1693302753) [ 630.758054] Lustre: DEBUG MARKER: == sanity test 31o: duplicate hard links with same filename ========================================================== 09:52:33 (1693302753) [ 646.109902] Lustre: DEBUG MARKER: == sanity test 31p: remove of open striped directory ===== 09:52:49 (1693302769) [ 646.964296] Lustre: DEBUG MARKER: == sanity test 31q: create striped directory on specific MDTs ========================================================== 09:52:50 (1693302770) [ 647.090969] Lustre: DEBUG MARKER: SKIP: sanity test_31q needs >= 3 MDTs [ 647.324648] Lustre: DEBUG MARKER: == sanity test 31r: open-rename(replace) race ============ 09:52:50 (1693302770) [ 648.781478] Lustre: DEBUG MARKER: SKIP: sanity test_32a skipping excluded test 32a (base 32) [ 648.983678] Lustre: DEBUG MARKER: SKIP: sanity test_32b skipping excluded test 32b (base 32) [ 649.154990] Lustre: DEBUG MARKER: SKIP: sanity test_32c skipping excluded test 32c (base 32) [ 649.269859] Lustre: DEBUG MARKER: SKIP: sanity test_32d skipping excluded test 32d (base 32) [ 649.391434] Lustre: DEBUG MARKER: SKIP: sanity test_32e skipping excluded test 32e (base 32) [ 649.534783] Lustre: DEBUG MARKER: SKIP: sanity test_32f skipping excluded test 32f (base 32) [ 649.669176] Lustre: DEBUG MARKER: SKIP: sanity test_32g skipping excluded test 32g (base 32) [ 649.794813] Lustre: DEBUG MARKER: SKIP: sanity test_32h skipping excluded test 32h (base 32) [ 649.902330] Lustre: DEBUG MARKER: SKIP: sanity test_32i skipping excluded test 32i (base 32) [ 650.016446] Lustre: DEBUG MARKER: SKIP: sanity test_32j skipping excluded test 32j (base 32) [ 650.132998] Lustre: DEBUG MARKER: SKIP: sanity test_32k skipping excluded test 32k (base 32) [ 650.237858] Lustre: DEBUG MARKER: SKIP: sanity test_32l skipping excluded test 32l (base 32) [ 650.335477] Lustre: DEBUG MARKER: SKIP: sanity test_32m skipping excluded test 32m (base 32) [ 650.439499] Lustre: DEBUG MARKER: SKIP: sanity test_32n skipping excluded test 32n (base 32) [ 650.543718] Lustre: DEBUG MARKER: SKIP: sanity test_32o skipping excluded test 32o (base 32) [ 650.657632] Lustre: DEBUG MARKER: SKIP: sanity test_32p skipping excluded test 32p (base 32) [ 650.763520] Lustre: DEBUG MARKER: SKIP: sanity test_32q skipping excluded test 32q (base 32) [ 650.864180] Lustre: DEBUG MARKER: SKIP: sanity test_32r skipping excluded test 32r (base 32) [ 651.013123] Lustre: DEBUG MARKER: == sanity test 33aa: write file with mode 444 (should return error) ========================================================== 09:52:54 (1693302774) [ 651.135591] Lustre: DEBUG MARKER: 33_1 [ 651.225991] Lustre: DEBUG MARKER: 33_2 [ 651.606813] Lustre: DEBUG MARKER: == sanity test 33a: test open file(mode=0444) with O_RDWR (should return error) ========================================================== 09:52:54 (1693302774) [ 652.124672] Lustre: DEBUG MARKER: == sanity test 33b: test open file with malformed flags (No panic) ========================================================== 09:52:55 (1693302775) [ 652.606217] Lustre: DEBUG MARKER: == sanity test 33c: test write_bytes stats =============== 09:52:55 (1693302775) [ 653.111404] Lustre: DEBUG MARKER: == sanity test 33d: openfile with 444 modes and malformed flags under remote dir ========================================================== 09:52:56 (1693302776) [ 653.657351] Lustre: DEBUG MARKER: == sanity test 33e: mkdir and striped directory should have same mode ========================================================== 09:52:56 (1693302776) [ 654.407070] Lustre: DEBUG MARKER: == sanity test 33f: nonroot user can create, access, and remove a striped directory ========================================================== 09:52:57 (1693302777) [ 655.067461] Lustre: DEBUG MARKER: == sanity test 33g: nonroot user create already existing root created file ========================================================== 09:52:58 (1693302778) [ 655.523153] Lustre: DEBUG MARKER: == sanity test 33h: temp file is located on the same MDT as target (crush) ========================================================== 09:52:58 (1693302778) [ 686.453818] Lustre: DEBUG MARKER: == sanity test 33hh: temp file is located on the same MDT as target (crush2) ========================================================== 09:53:29 (1693302809) [ 716.139416] Lustre: DEBUG MARKER: == sanity test 33i: striped directory can be accessed when one MDT is down ========================================================== 09:53:59 (1693302839) [ 719.610687] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 719.610819] Lustre: Skipped 1 previous similar message [ 719.633447] LustreError: 57768:0:(file.c:246:ll_close_inode_openhandle()) lustre-clilmv-ffff8f75fa73f000: inode [0x240000402:0x4b7e:0x0] mdc close failed: rc = -108 [ 719.694281] LustreError: 57771:0:(mdc_request.c:1465:mdc_read_page()) lustre-MDT0001-mdc-ffff8f75fa73f000: [0x240000403:0x9f:0x0] lock enqueue fails: rc = -108 [ 719.694577] Lustre: dir [0x200000402:0x2eb7:0x0] stripe 1 readdir failed: -108, directory is partially accessed! [ 720.234215] Lustre: lustre-MDT0001-mdc-ffff8f75fa73f000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 720.234517] Lustre: Skipped 3 previous similar messages [ 720.239054] Lustre: lustre-MDT0001: Client 2b18f5a1-3a9c-46b9-9b9b-4e01995ee896 (at 0@lo) reconnecting [ 720.239355] LustreError: 167-0: lustre-MDT0001-mdc-ffff8f75fa73f000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 720.239687] LustreError: Skipped 1 previous similar message [ 720.241781] Lustre: lustre-MDT0001-mdc-ffff8f75fa73f000: Connection restored to 192.168.121.234@tcp (at 0@lo) [ 720.241950] Lustre: Skipped 6 previous similar messages [ 720.532653] Lustre: DEBUG MARKER: == sanity test 33j: lfs setdirstripe -D -i x,y,x should fail ========================================================== 09:54:03 (1693302843) [ 721.363460] Lustre: DEBUG MARKER: == sanity test 34a: truncate file that has not been opened ===================================================================== 09:54:04 (1693302844) [ 721.896591] Lustre: DEBUG MARKER: == sanity test 34b: O_RDONLY opening file doesn't create objects =============================================================== 09:54:05 (1693302845) [ 722.303659] Lustre: DEBUG MARKER: == sanity test 34c: O_RDWR opening file-with-size works ======================================================================== 09:54:05 (1693302845) [ 723.046082] Lustre: DEBUG MARKER: == sanity test 34d: write to sparse file ======================================================================================= 09:54:06 (1693302846) [ 723.774729] Lustre: DEBUG MARKER: == sanity test 34e: create objects, some with size and some without ============================================================ 09:54:06 (1693302846) [ 724.530202] Lustre: DEBUG MARKER: == sanity test 34f: read from a file with no objects until EOF ================================================================= 09:54:07 (1693302847) [ 725.299888] Lustre: DEBUG MARKER: == sanity test 34g: truncate long file ========================================================================================= 09:54:08 (1693302848) [ 726.020822] Lustre: DEBUG MARKER: == sanity test 34h: ftruncate file under grouplock should not block ========================================================== 09:54:09 (1693302849) [ 728.699058] Lustre: DEBUG MARKER: == sanity test 35a: exec file with mode 444 (should return and not leak) ========================================================== 09:54:11 (1693302851) [ 729.134953] Lustre: DEBUG MARKER: == sanity test 36a: MDS utime check (mknod, utime) ======= 09:54:12 (1693302852) [ 729.549635] Lustre: DEBUG MARKER: == sanity test 36b: OST utime check (open, utime) ======== 09:54:12 (1693302852) [ 729.954624] Lustre: DEBUG MARKER: == sanity test 36c: non-root MDS utime check (mknod, utime) ========================================================== 09:54:13 (1693302853) [ 730.568426] Lustre: DEBUG MARKER: == sanity test 36d: non-root OST utime check (open, utime) ========================================================== 09:54:13 (1693302853) [ 731.002699] Lustre: DEBUG MARKER: == sanity test 36e: utime on non-owned file (should return error) ========================================================== 09:54:14 (1693302854) [ 731.447161] Lustre: DEBUG MARKER: == sanity test 36f: utime on file racing with OST BRW write ==================================================================== 09:54:14 (1693302854) [ 731.556111] LustreError: 41798:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 214 sleeping for 5000ms [ 736.620055] LustreError: 41798:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 214 awake [ 737.324521] Lustre: DEBUG MARKER: == sanity test 36g: FMD cache expiry =============================================================================== 09:54:20 (1693302860) [ 760.790163] Lustre: lustre-OST0001-osc-ffff8f75fa73f000: disconnect after 24s idle [ 780.485647] Lustre: DEBUG MARKER: == sanity test 36h: utime on file racing with OST BRW write ==================================================================== 09:55:03 (1693302903) [ 782.637900] Lustre: DEBUG MARKER: == sanity test 36i: change mtime on striped directory ==== 09:55:05 (1693302905) [ 783.440473] Lustre: DEBUG MARKER: == sanity test 38: open a regular file with O_DIRECTORY should return -ENOTDIR ============================================================= 09:55:06 (1693302906) [ 784.085335] Lustre: DEBUG MARKER: == sanity test 39a: mtime changed on create ============== 09:55:07 (1693302907) [ 787.012686] Lustre: DEBUG MARKER: == sanity test 39b: mtime change on open, link, unlink, rename ================================================================ 09:55:10 (1693302910) [ 789.184106] Lustre: DEBUG MARKER: == sanity test 39c: mtime change on rename ===================================================================================== 09:55:12 (1693302912) [ 792.343240] Lustre: DEBUG MARKER: == sanity test 39d: create, utime, stat ======================================================================================== 09:55:15 (1693302915) [ 793.138798] Lustre: DEBUG MARKER: == sanity test 39e: create, stat, utime, stat ================================================================================== 09:55:16 (1693302916) [ 794.036151] Lustre: DEBUG MARKER: == sanity test 39f: create, stat, sleep, utime, stat =========================================================================== 09:55:17 (1693302917) [ 796.692829] Lustre: DEBUG MARKER: == sanity test 39g: write, chmod, stat ========================================================================================= 09:55:19 (1693302919) [ 799.545901] Lustre: DEBUG MARKER: == sanity test 39h: write, utime within one second, stat ======================================================================= 09:55:22 (1693302922) [ 801.427560] Lustre: DEBUG MARKER: == sanity test 39i: write, rename, stat ======================================================================================== 09:55:24 (1693302924) [ 803.005245] Lustre: DEBUG MARKER: == sanity test 39j: write, rename, close, stat ================================================================================= 09:55:26 (1693302926) [ 804.737175] LustreError: 61230:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 412 sleeping for 1000ms [ 805.840075] LustreError: 61230:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 412 awake [ 810.356067] Lustre: DEBUG MARKER: == sanity test 39k: write, utime, close, stat ================================================================================== 09:55:33 (1693302933) [ 812.970561] Lustre: DEBUG MARKER: == sanity test 39l: directory atime update ===================================================================================== 09:55:36 (1693302936) [ 820.125588] Lustre: DEBUG MARKER: == sanity test 39m: test atime and mtime before 1970 ===== 09:55:43 (1693302943) [ 822.899598] Lustre: DEBUG MARKER: == sanity test 39n: check that O_NOATIME is honored ====== 09:55:45 (1693302945) [ 833.908168] Lustre: DEBUG MARKER: == sanity test 39o: directory cached attributes updated after create ========================================================== 09:55:57 (1693302957) [ 834.737499] Lustre: DEBUG MARKER: == sanity test 39p: remote directory cached attributes updated after create ================================================================== 09:55:57 (1693302957) [ 835.817804] Lustre: DEBUG MARKER: == sanity test 39r: lazy atime update on OST ============= 09:55:58 (1693302958) [ 835.931899] Lustre: DEBUG MARKER: SKIP: sanity test_39r ldiskfs only test [ 836.186411] Lustre: DEBUG MARKER: == sanity test 39q: close won't zero out atime =========== 09:55:59 (1693302959) [ 836.771861] Lustre: DEBUG MARKER: == sanity test 39s: relatime is supported ================ 09:55:59 (1693302959) [ 837.086961] systemd[1]: mnt-lustre.mount: Succeeded. [ 837.716512] Lustre: Unmounted lustre-client [ 837.890267] Lustre: Mounted lustre-client [ 843.514341] systemd[1]: mnt-lustre.mount: Succeeded. [ 843.592651] Lustre: Unmounted lustre-client [ 843.716306] Lustre: Mounted lustre-client [ 844.669590] Lustre: DEBUG MARKER: == sanity test 40: failed open(O_TRUNC) doesn't truncate ======================================================================= 09:56:07 (1693302967) [ 845.366288] Lustre: DEBUG MARKER: == sanity test 41: test small file write + fstat =============================================================================== 09:56:08 (1693302968) [ 847.105824] Lustre: DEBUG MARKER: SKIP: sanity test_42a skipping ALWAYS excluded test 42a [ 847.288001] Lustre: DEBUG MARKER: SKIP: sanity test_42b skipping ALWAYS excluded test 42b [ 847.470399] Lustre: DEBUG MARKER: SKIP: sanity test_42c skipping ALWAYS excluded test 42c [ 847.696914] Lustre: DEBUG MARKER: == sanity test 42d: test complete truncate of file with cached dirty data ========================================================== 09:56:10 (1693302970) [ 849.232777] Lustre: DEBUG MARKER: == sanity test 42e: verify sub-RPC writes are not done synchronously ========================================================== 09:56:12 (1693302972) [ 1012.876237] Lustre: DEBUG MARKER: SKIP: sanity test_43A skipping excluded test 43A (base 43) [ 1013.192425] Lustre: DEBUG MARKER: SKIP: sanity test_43a skipping excluded test 43a (base 43) [ 1013.398914] Lustre: DEBUG MARKER: SKIP: sanity test_43b skipping excluded test 43b (base 43) [ 1013.663322] Lustre: DEBUG MARKER: SKIP: sanity test_43c skipping excluded test 43c (base 43) [ 1013.923897] Lustre: DEBUG MARKER: == sanity test 44A: zero length read from a sparse stripe ========================================================== 09:58:57 (1693303137) [ 1015.031194] Lustre: DEBUG MARKER: == sanity test 44a: test sparse pwrite ========================================================================================= 09:58:58 (1693303138) [ 1016.512004] Lustre: DEBUG MARKER: SKIP: sanity test_45 skipping excluded test 45 [ 1016.793561] Lustre: DEBUG MARKER: == sanity test 46: dirtying a previously written page ========================================================================== 09:58:59 (1693303139) [ 1017.907491] Lustre: DEBUG MARKER: == sanity test 48a: Access renamed working dir (should return errors)=========================================================== 09:59:00 (1693303140) [ 1019.906337] Lustre: DEBUG MARKER: == sanity test 48b: Access removed working dir (should return errors)=========================================================== 09:59:02 (1693303142) [ 1021.278883] Lustre: DEBUG MARKER: == sanity test 48c: Access removed working subdir (should return errors) ========================================================== 09:59:04 (1693303144) [ 1022.957812] Lustre: DEBUG MARKER: == sanity test 48d: Access removed parent subdir (should return errors) ========================================================== 09:59:06 (1693303146) [ 1023.944719] Lustre: DEBUG MARKER: == sanity test 48e: Access to recreated parent subdir (should return errors) ========================================================== 09:59:07 (1693303147) [ 1025.143088] Lustre: DEBUG MARKER: == sanity test 48f: non-zero nlink dir unlink won't LBUG() ========================================================== 09:59:08 (1693303148) [ 1025.345235] Lustre: DEBUG MARKER: SKIP: sanity test_48f needs different host for mdt1 mdt2 [ 1025.631236] Lustre: DEBUG MARKER: == sanity test 49: Change max_pages_per_rpc won't break osc extent ========================================================== 09:59:08 (1693303148) [ 1060.148680] Lustre: DEBUG MARKER: == sanity test 50: special situations: /proc symlinks ========================================================================= 09:59:43 (1693303183) [ 1060.997310] Lustre: DEBUG MARKER: == sanity test 51a: special situations: split htree with empty entry ============================================================ 09:59:44 (1693303184) [ 1062.441861] Lustre: DEBUG MARKER: SKIP: sanity test_51b skipping SLOW test 51b [ 1062.713426] Lustre: DEBUG MARKER: == sanity test 51d: check LOV round-robin OST object distribution ========================================================== 09:59:45 (1693303185) [ 1062.921092] Lustre: DEBUG MARKER: SKIP: sanity test_51d needs >= 3 OSTs [ 1063.224119] Lustre: DEBUG MARKER: == sanity test 51e: check file nlink limit =============== 09:59:46 (1693303186) [ 1063.381762] Lustre: DEBUG MARKER: SKIP: sanity test_51e ldiskfs only test [ 1063.630538] Lustre: DEBUG MARKER: SKIP: sanity test_51f skipping excluded test 51f [ 1063.929279] Lustre: DEBUG MARKER: == sanity test 52a: append-only flag test (should return errors) ========================================================== 09:59:46 (1693303186) [ 1064.292451] LustreError: 71209:0:(file.c:246:ll_close_inode_openhandle()) lustre-clilmv-ffff8f75d6b79000: inode [0x240000405:0x8cf:0x0] mdc close failed: rc = -1 [ 1064.292727] LustreError: 71209:0:(file.c:246:ll_close_inode_openhandle()) Skipped 274 previous similar messages [ 1065.372363] Lustre: DEBUG MARKER: == sanity test 52b: immutable flag test (should return errors) ================================================================= 09:59:48 (1693303188) [ 1066.432355] Lustre: DEBUG MARKER: == sanity test 53: verify that MDS and OSTs agree on pre-creation ============================================================== 09:59:49 (1693303189) [ 1068.209377] Lustre: DEBUG MARKER: == sanity test 54a: unix domain socket test ==================================================================================== 09:59:51 (1693303191) [ 1069.082113] Lustre: DEBUG MARKER: == sanity test 54b: char device works in lustre ================================================================================ 09:59:52 (1693303192) [ 1069.784913] Lustre: DEBUG MARKER: == sanity test 54c: block device works in lustre =============================================================================== 09:59:52 (1693303192) [ 1070.028212] blk_update_request: operation not supported error, dev loop3, sector 0 op 0x3:(DISCARD) flags 0x800 phys_seg 1 prio class 0 [ 1070.035003] blk_update_request: operation not supported error, dev loop3, sector 2048 op 0x3:(DISCARD) flags 0x800 phys_seg 1 prio class 0 [ 1070.147742] blk_update_request: operation not supported error, dev loop3, sector 8064 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 1070.149718] blk_update_request: operation not supported error, dev loop3, sector 42 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 1070.166282] blk_update_request: operation not supported error, dev loop3, sector 326 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 1070.481252] systemd[1]: mnt-lustre-d54c.sanity.mount: Succeeded. [ 1071.307694] Lustre: DEBUG MARKER: == sanity test 54d: fifo device works in lustre ================================================================================ 09:59:54 (1693303194) [ 1071.953245] Lustre: DEBUG MARKER: == sanity test 54e: console/tty device works in lustre ================================================================================ 09:59:55 (1693303195) aaaaaa [ 1072.753668] Lustre: DEBUG MARKER: == sanity test 56a: check /mnt/build/lustre/tests/../utils/lfs getstripe ========================================================== 09:59:55 (1693303195) [ 1074.375503] Lustre: DEBUG MARKER: == sanity test 56b: check /mnt/build/lustre/tests/../utils/lfs getdirstripe ========================================================== 09:59:57 (1693303197) [ 1075.823880] Lustre: DEBUG MARKER: == sanity test 56bb: check /mnt/build/lustre/tests/../utils/lfs getdirstripe layout is YAML ========================================================== 09:59:58 (1693303198) [ 1077.357472] Lustre: DEBUG MARKER: == sanity test 56c: check 'lfs df' showing device status ========================================================== 10:00:00 (1693303200) [ 1099.098803] Lustre: DEBUG MARKER: == sanity test 56d: 'lfs df -v' prints only configured devices ========================================================== 10:00:22 (1693303222) [ 1100.038698] Lustre: DEBUG MARKER: == sanity test 56e: 'lfs df' Handle non LustreFS [ 1101.187317] Lustre: DEBUG MARKER: == sanity test 56g: check lfs find -name ================= 10:00:24 (1693303224) [ 1103.008750] Lustre: DEBUG MARKER: == sanity test 56h: check lfs find ! -name =============== 10:00:26 (1693303226) [ 1103.926548] Lustre: DEBUG MARKER: == sanity test 56i: check 'lfs find -ost UUID' skips directories ========================================================== 10:00:27 (1693303227) [ 1104.897015] Lustre: DEBUG MARKER: == sanity test 56j: check lfs find -type d =============== 10:00:27 (1693303227) [ 1106.268350] Lustre: DEBUG MARKER: == sanity test 56k: check lfs find -type f =============== 10:00:29 (1693303229) [ 1107.165438] Lustre: DEBUG MARKER: == sanity test 56l: check lfs find -type b =============== 10:00:30 (1693303230) [ 1107.929919] Lustre: DEBUG MARKER: == sanity test 56m: check lfs find -type c =============== 10:00:30 (1693303230) [ 1108.755260] Lustre: DEBUG MARKER: == sanity test 56n: check lfs find -type l =============== 10:00:31 (1693303231) [ 1109.528483] Lustre: DEBUG MARKER: == sanity test 56o: check lfs find -mtime for old files == 10:00:32 (1693303232) [ 1111.548392] Lustre: DEBUG MARKER: == sanity test 56ob: check lfs find -atime -mtime -ctime with units ========================================================== 10:00:34 (1693303234) [ 1115.059498] Lustre: DEBUG MARKER: == sanity test 56oc: check lfs find -newerXY work ======== 10:00:38 (1693303238) [ 1210.961727] Lustre: DEBUG MARKER: == sanity test 56od: check lfs find -btime with units ==== 10:02:14 (1693303334) [ 1218.488479] Lustre: DEBUG MARKER: == sanity test 56p: check lfs find -uid and ! -uid ======= 10:02:21 (1693303341) [ 1220.918730] Lustre: DEBUG MARKER: == sanity test 56q: check lfs find -gid and ! -gid ======= 10:02:24 (1693303344) [ 1222.907579] Lustre: DEBUG MARKER: == sanity test 56r: check lfs find -size works =========== 10:02:25 (1693303345) [ 1225.271538] Lustre: DEBUG MARKER: == sanity test 56ra: check lfs find -size -lazy works for data on OSTs ========================================================== 10:02:28 (1693303348) [ 1229.432646] Lustre: DEBUG MARKER: == sanity test 56rb: check lfs find --size --ost/--mdt works ========================================================== 10:02:32 (1693303352) [ 1230.747601] Lustre: DEBUG MARKER: == sanity test 56rc: check lfs find --mdt-count/--mdt-hash works ========================================================== 10:02:33 (1693303353) [ 1233.205354] Lustre: DEBUG MARKER: == sanity test 56rd: check lfs find --printf special files ========================================================== 10:02:36 (1693303356) [ 1234.403024] Lustre: DEBUG MARKER: == sanity test 56s: check lfs find -stripe-count works === 10:02:37 (1693303357) [ 1236.277039] Lustre: DEBUG MARKER: == sanity test 56t: check lfs find -stripe-size works ==== 10:02:39 (1693303359) [ 1238.655068] Lustre: DEBUG MARKER: == sanity test 56u: check lfs find -stripe-index works === 10:02:41 (1693303361) [ 1240.240421] Lustre: DEBUG MARKER: == sanity test 56v: check 'lfs find -m match with lfs getstripe -m' ========================================================== 10:02:43 (1693303363) [ 1242.002724] Lustre: DEBUG MARKER: == sanity test 56wa: check lfs_migrate -c stripe_count works ========================================================== 10:02:45 (1693303365) [ 1260.966431] Lustre: DEBUG MARKER: == sanity test 56wb: check lfs_migrate pool support ====== 10:03:04 (1693303384) [ 1271.114120] Lustre: DEBUG MARKER: == sanity test 56wc: check unrecognized options for lfs_migrate are passed through ========================================================== 10:03:14 (1693303394) [ 1278.231946] Lustre: DEBUG MARKER: == sanity test 56wd: check lfs_migrate --rsync and --no-rsync work ========================================================== 10:03:21 (1693303401) [ 1279.609078] Lustre: DEBUG MARKER: == sanity test 56we: check lfs_migrate --non-direct|-D support ========================================================== 10:03:22 (1693303402) [ 1281.127796] Lustre: DEBUG MARKER: == sanity test 56x: lfs migration support ================ 10:03:24 (1693303404) [ 1282.436158] Lustre: DEBUG MARKER: == sanity test 56xa: lfs migration --block support ======= 10:03:25 (1693303405) [ 1283.438200] Lustre: DEBUG MARKER: == sanity test 56xb: lfs migration hard link support ===== 10:03:26 (1693303406) [ 1396.415663] Lustre: DEBUG MARKER: SKIP: sanity test_56xc skipping excluded test 56xc [ 1396.691961] Lustre: DEBUG MARKER: == sanity test 56xd: check lfs_migrate --yaml and --copy support ========================================================== 10:05:19 (1693303519) [ 1402.693372] Lustre: DEBUG MARKER: == sanity test 56xe: migrate a composite layout file ===== 10:05:25 (1693303525) [ 1409.624373] Lustre: DEBUG MARKER: == sanity test 56xf: FID is not lost during migration of a composite layout file ========================================================== 10:05:32 (1693303532) [ 1412.298674] Lustre: DEBUG MARKER: == sanity test 56xg: lfs migrate pool support ============ 10:05:35 (1693303535) [ 1442.697834] Lustre: DEBUG MARKER: == sanity test 56xh: lfs migrate bandwidth limitation support ========================================================== 10:06:05 (1693303565) [ 1470.016682] Lustre: DEBUG MARKER: == sanity test 56xi: lfs migrate stats support =========== 10:06:33 (1693303593) [ 1472.381057] Lustre: DEBUG MARKER: == sanity test 56xj: lfs migrate -b should not cause starvation of threads on OSS ========================================================== 10:06:35 (1693303595) [ 1479.864561] Lustre: DEBUG MARKER: == sanity test 56xk: lfs mirror resync bandwidth limitation support ========================================================== 10:06:42 (1693303602) [ 1486.284546] Lustre: DEBUG MARKER: == sanity test 56xl: lfs mirror resync stats support ===== 10:06:49 (1693303609) [ 1488.004842] Lustre: DEBUG MARKER: == sanity test 56y: lfs find -L raid0|released =========== 10:06:51 (1693303611) [ 1489.077170] Lustre: DEBUG MARKER: == sanity test 56z: lfs find should continue after an error ========================================================== 10:06:52 (1693303612) [ 1491.291750] Lustre: DEBUG MARKER: == sanity test 56aa: lfs find --size under striped dir === 10:06:54 (1693303614) [ 1497.289052] Lustre: DEBUG MARKER: == sanity test 56ab: lfs find --blocks =================== 10:07:00 (1693303620) [ 1501.255346] Lustre: DEBUG MARKER: == sanity test 56aca: check lfs find -perm with octal representation ========================================================== 10:07:04 (1693303624) [ 1506.175460] Lustre: DEBUG MARKER: == sanity test 56acb: check lfs find -perm with symbolic representation ========================================================== 10:07:09 (1693303629) [ 1508.503942] Lustre: DEBUG MARKER: == sanity test 56acc: check parsing error for lfs find -perm ========================================================== 10:07:11 (1693303631) [ 1509.407842] Lustre: DEBUG MARKER: == sanity test 56ba: test lfs find --component-end, -start, -count, and -flags ========================================================== 10:07:12 (1693303632) [ 1512.861245] Lustre: DEBUG MARKER: == sanity test 56ca: check lfs find --mirror-count|-N and --mirror-state ========================================================== 10:07:15 (1693303635) [ 1514.825873] Lustre: DEBUG MARKER: == sanity test 56da: test lfs find with long paths ======= 10:07:17 (1693303637) [ 1518.310685] Lustre: DEBUG MARKER: == sanity test 56ea: test lfs find -printf option ======== 10:07:21 (1693303641) [ 1530.851103] Lustre: DEBUG MARKER: == sanity test 56eb: check lfs getstripe on symlink ====== 10:07:33 (1693303653) [ 1531.786071] Lustre: DEBUG MARKER: == sanity test 56ec: check lfs getstripe,setstripe --hex --yaml ========================================================== 10:07:34 (1693303654) [ 1532.619313] Lustre: DEBUG MARKER: == sanity test 56eda: check lfs find --links ============= 10:07:35 (1693303655) [ 1533.658769] Lustre: DEBUG MARKER: == sanity test 56edb: check lfs find --links for directory striped on multiple MDTs ========================================================== 10:07:36 (1693303656) [ 1534.547213] Lustre: DEBUG MARKER: == sanity test 56ef: lfs find with multiple paths ======== 10:07:37 (1693303657) [ 1535.506911] Lustre: DEBUG MARKER: == sanity test 57a: verify MDS filesystem created with large inodes ============================================================ 10:07:38 (1693303658) [ 1535.663359] Lustre: DEBUG MARKER: SKIP: sanity test_57a ldiskfs only test [ 1535.993805] Lustre: DEBUG MARKER: == sanity test 57b: default LOV EAs are stored inside large inodes ============================================================= 10:07:39 (1693303659) [ 1536.207158] Lustre: DEBUG MARKER: SKIP: sanity test_57b ldiskfs only test [ 1536.609154] Lustre: DEBUG MARKER: == sanity test 58: verify cross-platform wire constants ======================================================================== 10:07:39 (1693303659) [ 1537.846544] Lustre: DEBUG MARKER: == sanity test 59: verify cancellation of llog records async =================================================================== 10:07:40 (1693303660) [ 1556.912458] Lustre: DEBUG MARKER: == sanity test 60a: llog_test run from kernel module and test llog_reader ========================================================== 10:07:59 (1693303679) [ 1557.213772] Lustre: DEBUG MARKER: test_60 run 17612 - from kernel mode [ 1566.507511] Lustre: 101575:0:(llog_test.c:2307:llog_test_setup()) Setup llog-test device over MGS device [ 1566.507777] Lustre: 101575:0:(llog_test.c:112:llog_test_1()) 1a: create a log with name: 198f203b [ 1566.509002] Lustre: 101575:0:(llog_test.c:129:llog_test_1()) 1b: close newly-created log [ 1566.509216] Lustre: 101575:0:(llog_test.c:160:llog_test_2()) 2a: re-open a log with name: 198f203b [ 1566.509686] Lustre: 101575:0:(llog_test.c:180:llog_test_2()) 2b: create a log without specified NAME & LOGID [ 1566.521999] Lustre: 101575:0:(llog_test.c:198:llog_test_2()) 2b: write 1 llog records, check llh_count [ 1566.522666] Lustre: 101575:0:(llog_test.c:211:llog_test_2()) 2c: re-open the log by LOGID and verify llh_count [ 1566.528920] Lustre: 101575:0:(llog_test.c:258:llog_test_2()) 2d: destroy this log [ 1566.529402] Lustre: 101575:0:(llog_test.c:418:llog_test_3()) 3a: write 1023 fixed-size llog records [ 1566.684549] Lustre: 101575:0:(llog_test.c:382:llog_test3_process()) test3: processing records from index 501 to the end [ 1566.767070] Lustre: 101575:0:(llog_test.c:392:llog_test3_process()) test3: total 525 records processed with 0 paddings [ 1566.768105] Lustre: 101575:0:(llog_test.c:474:llog_test_3()) 3b: write 566 variable size llog records [ 1567.057722] Lustre: 101575:0:(llog_test.c:546:llog_test_3()) 3c: write records with variable size until BITMAP_SIZE, return -ENOSPC [ 1596.381780] Lustre: 101575:0:(llog_test.c:569:llog_test_3()) 3c: wrote 63962 more records before end of llog is reached [ 1596.382323] Lustre: 101575:0:(llog_test.c:598:llog_test_4()) 4a: create a catalog log with name: 198f203c [ 1596.383021] Lustre: 101575:0:(llog_test.c:613:llog_test_4()) 4b: write 1 record into the catalog [ 1596.385188] Lustre: 101575:0:(llog_test.c:640:llog_test_4()) 4c: cancel 1 log record [ 1596.385705] Lustre: 101575:0:(llog_test.c:652:llog_test_4()) 4d: write 64767 more log records [ 1626.200241] Lustre: 101575:0:(llog_test.c:668:llog_test_4()) 4e: add 5 large records, one record per block [ 1626.201472] Lustre: 101575:0:(llog_test.c:688:llog_test_4()) 4f: put newly-created catalog [ 1626.201595] Lustre: 101575:0:(llog_test.c:786:llog_test_5()) 5a: re-open catalog by id [ 1626.201830] Lustre: 101575:0:(llog_test.c:799:llog_test_5()) 5b: print the catalog entries.. we expect 2 [ 1626.204029] Lustre: 101790:0:(llog_test.c:717:cat_print_cb()) seeing record at index 1 - [0x1:0xd:0x0] in log [0xa:0xc:0x0] [ 1626.204202] Lustre: 101575:0:(llog_test.c:811:llog_test_5()) 5c: Cancel 64767 records, see one log zapped [ 1634.175159] Lustre: 101575:0:(llog_test.c:819:llog_test_5()) 5c: print the catalog entries.. we expect 1 [ 1634.175871] Lustre: 101796:0:(llog_test.c:717:cat_print_cb()) seeing record at index 2 - [0x1:0xe:0x0] in log [0xa:0xc:0x0] [ 1634.175950] Lustre: 101796:0:(llog_test.c:717:cat_print_cb()) Skipped 1 previous similar message [ 1634.176089] Lustre: 101575:0:(llog_test.c:831:llog_test_5()) 5d: add 1 record to the log with many canceled empty pages [ 1634.176922] Lustre: 101575:0:(llog_test.c:839:llog_test_5()) 5e: print plain log entries.. expect 6 [ 1634.177535] Lustre: 101575:0:(llog_test.c:851:llog_test_5()) 5f: print plain log entries reversely.. expect 6 [ 1634.178811] Lustre: 101575:0:(llog_test.c:865:llog_test_5()) 5g: close re-opened catalog [ 1634.178931] Lustre: 101575:0:(llog_test.c:895:llog_test_6()) 6a: re-open log 198f203b using client API [ 1634.179206] Lustre: MGS: non-config logname received: 198f203b [ 1634.183788] Lustre: 101575:0:(llog_test.c:927:llog_test_6()) 6b: process log 198f203b using client API [ 1634.779139] Lustre: 101575:0:(llog_test.c:931:llog_test_6()) 6b: processed 63962 records [ 1634.779538] Lustre: 101575:0:(llog_test.c:938:llog_test_6()) 6c: process log 198f203b reversely using client API [ 1702.942775] Lustre: 101575:0:(llog_test.c:942:llog_test_6()) 6c: processed 63962 records [ 1702.943295] Lustre: 101575:0:(llog_test.c:1089:llog_test_7()) 7a: test llog_logid_rec [ 1726.343180] Lustre: 101575:0:(llog_test.c:1100:llog_test_7()) 7b: test llog_unlink64_rec [ 1741.287300] Lustre: 101575:0:(llog_test.c:1111:llog_test_7()) 7c: test llog_setattr64_rec [ 1755.024491] Lustre: 101575:0:(llog_test.c:1122:llog_test_7()) 7d: test llog_size_change_rec [ 1769.807022] Lustre: 101575:0:(llog_test.c:1133:llog_test_7()) 7e: test llog_changelog_rec [ 1779.430334] Lustre: 101575:0:(llog_test.c:1040:llog_test_7_sub()) 7_sub: records are not aligned, written 64071 from 64767 [ 1791.836879] Lustre: 101575:0:(llog_test.c:1145:llog_test_7()) 7f: test llog_changelog_user_rec2 [ 1800.684599] Lustre: 101575:0:(llog_test.c:1040:llog_test_7_sub()) 7_sub: records are not aligned, written 64139 from 64767 [ 1809.917628] Lustre: 101575:0:(llog_test.c:1156:llog_test_7()) 7g: test llog_gen_rec [ 1830.187568] Lustre: 101575:0:(llog_test.c:1167:llog_test_7()) 7h: test llog_setattr64_rec_v2 [ 1845.509990] Lustre: 101575:0:(llog_test.c:1040:llog_test_7_sub()) 7_sub: records are not aligned, written 64071 from 64767 [ 1863.532514] Lustre: 101575:0:(llog_test.c:1274:llog_test_8()) 8a: fill the first plain llog [ 1863.538538] Lustre: 101575:0:(llog_test.c:1303:llog_test_8()) 8b: first llog [0x1:0x1a:0x0] [ 1863.549563] Lustre: 101575:0:(llog_test.c:1321:llog_test_8()) 8b: fill the second plain llog [ 1863.604101] Lustre: 101575:0:(llog_test.c:1345:llog_test_8()) 8b: pin llog [0x1:0x1c:0x0] [ 1863.604357] Lustre: 101575:0:(llog_test.c:1348:llog_test_8()) 8b: clean first llog record in catalog [ 1863.604660] Lustre: 101575:0:(llog_test.c:1361:llog_test_8()) 8c: corrupt first chunk in the middle [ 1863.604934] Lustre: 101575:0:(llog_test.c:1364:llog_test_8()) 8c: corrupt second chunk at start [ 1863.605207] Lustre: 101575:0:(llog_test.c:1367:llog_test_8()) 8d: count survived records [ 1863.631953] LustreError: 101575:0:(llog.c:478:llog_verify_record()) MGS: [0xa:0xc:0x0] rec type=0 idx=0 len=0, magic is bad [ 1863.632602] Lustre: 101575:0:(llog_test.c:1397:llog_test_8()) 8d: close re-opened catalog [ 1863.632842] Lustre: 101575:0:(llog_test.c:1460:llog_test_9()) 9a: test llog_logid_rec [ 1863.633425] Lustre: 101575:0:(llog_test.c:1444:llog_test_9_sub()) 9_sub: record type 1064553b in log 0x1:0x1e:0x0 [ 1863.633696] Lustre: 101575:0:(llog_test.c:1471:llog_test_9()) 9b: test llog_obd_cfg_rec [ 1863.634220] Lustre: 101575:0:(llog_test.c:1482:llog_test_9()) 9c: test llog_changelog_rec [ 1863.634749] Lustre: 101575:0:(llog_test.c:1494:llog_test_9()) 9d: test llog_changelog_user_rec2 [ 1863.635313] Lustre: 101575:0:(llog_test.c:1594:llog_test_10()) 10a: create a catalog log with name: 198f203d [ 1863.723307] Lustre: 101575:0:(llog_test.c:1624:llog_test_10()) 10b: write 64767 log records [ 1898.114080] Lustre: 101575:0:(llog_test.c:1650:llog_test_10()) 10c: write 129534 more log records [ 1954.053023] Lustre: 101575:0:(llog_test.c:1682:llog_test_10()) 10c: write 64767 more log records [ 1977.730411] Lustre: 101575:0:(llog_cat.c:100:llog_cat_new_log()) MGS: there are no more free slots in catalog 198f203d [ 1977.918831] Lustre: 101575:0:(llog_test.c:1709:llog_test_10()) 10c: wrote 64011 records then 756 failed with ENOSPC [ 1977.919407] Lustre: 101575:0:(llog_test.c:1728:llog_test_10()) 10d: Cancel 64767 records, see one log zapped [ 1987.430110] Lustre: 101575:0:(llog_test.c:1742:llog_test_10()) 10d: print the catalog entries.. we expect 3 [ 1987.432223] Lustre: 102310:0:(llog_test.c:717:cat_print_cb()) seeing record at index 2 - [0x1:0x23:0x0] in log [0xa:0xd:0x0] [ 1987.500610] Lustre: 101575:0:(llog_test.c:1772:llog_test_10()) 10e: write 64767 more log records [ 2015.790334] Lustre: 101575:0:(llog_cat.c:100:llog_cat_new_log()) MGS: there are no more free slots in catalog 198f203d [ 2015.790557] Lustre: 101575:0:(llog_cat.c:100:llog_cat_new_log()) Skipped 755 previous similar messages [ 2015.839959] Lustre: 101575:0:(llog_test.c:1799:llog_test_10()) 10e: wrote 64578 records then 189 failed with ENOSPC [ 2015.840253] Lustre: 101575:0:(llog_test.c:1801:llog_test_10()) 10e: print the catalog entries.. we expect 4 [ 2015.840392] Lustre: 101575:0:(llog_cat.c:939:llog_cat_process_or_fork()) MGS: catlog [0xa:0xd:0x0] crosses index zero [ 2015.840593] Lustre: 101575:0:(llog_test.c:717:cat_print_cb()) seeing record at index 2 - [0x1:0x23:0x0] in log [0xa:0xd:0x0] [ 2015.840780] Lustre: 101575:0:(llog_test.c:717:cat_print_cb()) Skipped 2 previous similar messages [ 2015.841220] Lustre: 101575:0:(llog_test.c:1838:llog_test_10()) 10e: catalog successfully wrap around, last_idx 1, first 1 [ 2015.917762] Lustre: 101575:0:(llog_test.c:1855:llog_test_10()) 10f: Cancel 64767 records, see one log zapped [ 2027.208111] Lustre: 101575:0:(llog_test.c:1869:llog_test_10()) 10f: print the catalog entries.. we expect 3 [ 2027.208464] Lustre: 101575:0:(llog_cat.c:939:llog_cat_process_or_fork()) MGS: catlog [0xa:0xd:0x0] crosses index zero [ 2027.208718] Lustre: 101575:0:(llog_cat.c:939:llog_cat_process_or_fork()) Skipped 1 previous similar message [ 2027.208970] Lustre: 101575:0:(llog_test.c:717:cat_print_cb()) seeing record at index 3 - [0x1:0x24:0x0] in log [0xa:0xd:0x0] [ 2027.209247] Lustre: 101575:0:(llog_test.c:717:cat_print_cb()) Skipped 3 previous similar messages [ 2027.347634] Lustre: 101575:0:(llog_test.c:1900:llog_test_10()) 10f: write 64767 more log records [ 2058.107051] Lustre: 101575:0:(llog_cat.c:100:llog_cat_new_log()) MGS: there are no more free slots in catalog 198f203d [ 2058.119982] Lustre: 101575:0:(llog_cat.c:100:llog_cat_new_log()) Skipped 188 previous similar messages [ 2058.253033] Lustre: 101575:0:(llog_test.c:1927:llog_test_10()) 10f: wrote 64578 records then 189 failed with ENOSPC [ 2058.313015] Lustre: 101575:0:(llog_test.c:1974:llog_test_10()) 10g: Cancel 64767 records, see one log zapped [ 2058.313228] Lustre: 101575:0:(llog_cat.c:939:llog_cat_process_or_fork()) MGS: catlog [0xa:0xd:0x0] crosses index zero [ 2071.455500] Lustre: 101575:0:(llog_test.c:1986:llog_test_10()) 10g: print the catalog entries.. we expect 3 [ 2071.455800] Lustre: 101575:0:(llog_cat.c:939:llog_cat_process_or_fork()) MGS: catlog [0xa:0xd:0x0] crosses index zero [ 2071.456096] Lustre: 101575:0:(llog_test.c:717:cat_print_cb()) seeing record at index 4 - [0x1:0x25:0x0] in log [0xa:0xd:0x0] [ 2071.456481] Lustre: 101575:0:(llog_test.c:717:cat_print_cb()) Skipped 2 previous similar messages [ 2071.493923] Lustre: 101575:0:(llog_test.c:2016:llog_test_10()) 10g: Cancel 64767 records, see one log zapped [ 2079.962038] Lustre: 101575:0:(llog_test.c:2030:llog_test_10()) 10g: print the catalog entries.. we expect 2 [ 2080.080092] Lustre: 101575:0:(llog_test.c:2068:llog_test_10()) 10g: Cancel 64767 records, see one log zapped [ 2092.904285] Lustre: 101575:0:(llog_test.c:2082:llog_test_10()) 10g: print the catalog entries.. we expect 1 [ 2092.916682] Lustre: 101575:0:(llog_test.c:717:cat_print_cb()) seeing record at index 2 - [0x1:0x3d9:0x0] in log [0xa:0xd:0x0] [ 2092.916860] Lustre: 101575:0:(llog_test.c:717:cat_print_cb()) Skipped 4 previous similar messages [ 2092.917239] Lustre: 101575:0:(llog_test.c:2108:llog_test_10()) 10g: llh_cat_idx has also successfully wrapped! [ 2092.917710] Lustre: 102485:0:(llog_test.c:1554:cat_check_old_cb()) seeing record at index 2 - [0x1:0x3d9:0x0] in log [0xa:0xd:0x0] [ 2093.430056] Lustre: 101575:0:(llog_test.c:2132:llog_test_10()) 10h: write 64767 more log records [ 2093.439727] LustreError: 101575:0:(libcfs_fail.h:190:cfs_race()) cfs_race id 1317 sleeping [ 2093.990082] LustreError: 102485:0:(libcfs_fail.h:201:cfs_race()) cfs_fail_race id 1317 waking [ 2093.997120] LustreError: 101575:0:(libcfs_fail.h:199:cfs_race()) cfs_fail_race id 1317 awake: rc=444 [ 2095.036985] LustreError: 102485:0:(libcfs_fail.h:201:cfs_race()) cfs_fail_race id 1317 waking [ 2095.037235] Lustre: 102485:0:(llog_test.c:1554:cat_check_old_cb()) seeing record at index 3 - [0x1:0x497:0x0] in log [0xa:0xd:0x0] [ 2124.806070] LustreError: 101575:0:(libcfs_fail.h:201:cfs_race()) cfs_fail_race id 1317 waking [ 2125.951178] Lustre: 101575:0:(llog_test.c:2159:llog_test_10()) 10h: wrote 64767 records then 0 failed with ENOSPC [ 2125.952343] Lustre: 101575:0:(llog_test.c:2172:llog_test_10()) 10: put newly-created catalog [ 2128.016061] systemd[1]: mnt-lustre\x2dmds1.mount: Succeeded. [ 2128.051374] Lustre: Failing over lustre-MDT0000 [ 2128.645800] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -19 [ 2128.646036] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2129.288345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2129.438045] Lustre: server umount lustre-MDT0000 complete [ 2133.082036] systemd[1]: mnt-lustre\x2dmds1.mount: Succeeded. [ 2134.310981] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2134.311143] LustreError: Skipped 4 previous similar messages [ 2135.125660] LustreError: 166-1: MGC192.168.121.234@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2135.127482] Lustre: Evicted from MGS (at 192.168.121.234@tcp) after server handle changed from 0x8c25e6449edfecc5 to 0x8c25e6449f313e9b [ 2135.128886] Lustre: MGC192.168.121.234@tcp: Connection restored to 192.168.121.234@tcp (at 0@lo) [ 2135.295095] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2135.346972] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2136.495555] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2136.517050] Lustre: Skipped 1 previous similar message [ 2136.538714] LustreError: 3175:0:(client.c:3259:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@000000008fd6689f x1775556010115712/t8589984163(8589984163) o101->lustre-MDT0000-mdc-ffff8f75d6b79000@0@lo:12/10 lens 520/664 e 0 to 0 dl 1693304266 ref 2 fl Interpret:RPQU/204/0 rc 301/301 uid:0 gid:0 job:'lfs.0' [ 2136.798621] Lustre: DEBUG MARKER: == sanity test 60b: limit repeated messages from CERROR/CWARN ========================================================== 10:17:39 (1693304259) [ 2140.389325] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 2140.389745] Lustre: Skipped 1 previous similar message [ 2140.420502] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000bd0:3462 to 0x2c0000bd0:3521 [ 2140.436584] Lustre: lustre-OST0000: deleting orphan objects from 0x280000bd0:3660 to 0x280000bd0:3713 [ 2141.838121] Lustre: DEBUG MARKER: == sanity test 60c: unlink file when mds full ============ 10:17:44 (1693304264) [ 2197.197378] Lustre: DEBUG MARKER: == sanity test 60d: test printk console message masking == 10:18:39 (1693304319) [ 2197.313884] Lustre: DEBUG MARKER: test message ID 17456 1554 [ 2198.684166] Lustre: DEBUG MARKER: == sanity test 60e: no space while new llog is being created ========================================================== 10:18:41 (1693304321) [ 2199.279255] Lustre: *** cfs_fail_loc=15b, val=0*** [ 2199.285222] Lustre: *** cfs_fail_loc=15b, val=0*** [ 2200.506362] Lustre: DEBUG MARKER: == sanity test 60f: change debug_path works ============== 10:18:43 (1693304323) [ 2200.596219] Lustre: *** cfs_fail_loc=50e, val=0*** [ 2200.596529] LustreError: dumping log to /tmp/f60f.sanity.1693304323.96711 [ 2201.602747] Lustre: DEBUG MARKER: == sanity test 60g: transaction abort won't cause MDT hung ========================================================== 10:18:44 (1693304324) [ 2201.788647] Lustre: *** cfs_fail_loc=19a, val=0*** [ 2202.257378] LustreError: 103544:0:(llog_cat.c:737:llog_cat_cancel_arr_rec()) lustre-MDT0000-osd: fail to cancel 1 llog-records: rc = -5 [ 2202.257756] LustreError: 103544:0:(llog_cat.c:773:llog_cat_cancel_records()) lustre-MDT0000-osd: fail to cancel 1 of 1 llog-records: rc = -5 [ 2202.328030] Lustre: *** cfs_fail_loc=19a, val=0*** [ 2202.335682] Lustre: Skipped 5 previous similar messages [ 2203.332487] Lustre: *** cfs_fail_loc=19a, val=0*** [ 2203.332665] Lustre: Skipped 4 previous similar messages [ 2204.423249] LustreError: 103544:0:(llog_cat.c:737:llog_cat_cancel_arr_rec()) lustre-MDT0001-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [ 2204.423518] LustreError: 103544:0:(llog_cat.c:737:llog_cat_cancel_arr_rec()) Skipped 1 previous similar message [ 2204.423669] LustreError: 103544:0:(llog_cat.c:773:llog_cat_cancel_records()) lustre-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [ 2204.423854] LustreError: 103544:0:(llog_cat.c:773:llog_cat_cancel_records()) Skipped 1 previous similar message [ 2205.446085] Lustre: *** cfs_fail_loc=19a, val=0*** [ 2205.447759] Lustre: Skipped 14 previous similar messages [ 2205.904364] LustreError: 6341:0:(llog_cat.c:737:llog_cat_cancel_arr_rec()) lustre-MDT0001-osd: fail to cancel 1 llog-records: rc = -5 [ 2205.904492] LustreError: 6341:0:(llog_cat.c:737:llog_cat_cancel_arr_rec()) Skipped 3 previous similar messages [ 2205.904564] LustreError: 6341:0:(llog_cat.c:773:llog_cat_cancel_records()) lustre-MDT0001-osd: fail to cancel 1 of 1 llog-records: rc = -5 [ 2205.904635] LustreError: 6341:0:(llog_cat.c:773:llog_cat_cancel_records()) Skipped 3 previous similar messages [ 2209.560258] Lustre: *** cfs_fail_loc=19a, val=0*** [ 2209.560361] Lustre: Skipped 29 previous similar messages [ 2211.608869] LustreError: 103544:0:(llog_cat.c:737:llog_cat_cancel_arr_rec()) lustre-MDT0001-osp-MDT0000: fail to cancel 1 llog-records: rc = -116 [ 2211.609123] LustreError: 103544:0:(llog_cat.c:737:llog_cat_cancel_arr_rec()) Skipped 6 previous similar messages [ 2211.609266] LustreError: 103544:0:(llog_cat.c:773:llog_cat_cancel_records()) lustre-MDT0001-osp-MDT0000: fail to cancel 1 of 1 llog-records: rc = -116 [ 2211.609442] LustreError: 103544:0:(llog_cat.c:773:llog_cat_cancel_records()) Skipped 6 previous similar messages [ 2215.057918] Lustre: dir [0x200000bd2:0x2192:0x0] stripe 0 readdir failed: -2, directory is partially accessed! [ 2215.058170] Lustre: Skipped 110 previous similar messages [ 2216.732663] Lustre: DEBUG MARKER: == sanity test 60h: striped directory with missing stripes can be accessed ========================================================== 10:18:59 (1693304339) [ 2216.861217] Lustre: *** cfs_fail_loc=188, val=0*** [ 2217.024860] Lustre: dir [0x240000405:0xbcb:0x0] stripe 2 readdir failed: -2, directory is partially accessed! [ 2217.025174] Lustre: Skipped 1 previous similar message [ 2217.710495] Lustre: *** cfs_fail_loc=189, val=0*** [ 2218.358029] Lustre: dir [0x240000405:0xbd2:0x0] stripe 2 readdir failed: -2, directory is partially accessed! [ 2218.358679] Lustre: Skipped 4 previous similar messages [ 2219.184136] Lustre: DEBUG MARKER: SKIP: sanity test_60i skipping SLOW test 60i [ 2219.497855] Lustre: DEBUG MARKER: == sanity test 60j: llog_reader reports corruptions ====== 10:19:02 (1693304342) [ 2219.869357] Lustre: DEBUG MARKER: SKIP: sanity test_60j ldiskfs only test [ 2220.231247] Lustre: DEBUG MARKER: == sanity test 61a: mmap() writes don't make sync hang ========================================================================== 10:19:03 (1693304343) [ 2221.344411] Lustre: DEBUG MARKER: == sanity test 61b: mmap() of unstriped file is successful ========================================================== 10:19:04 (1693304344) [ 2222.468251] Lustre: DEBUG MARKER: == sanity test 63a: Verify oig_wait interruption does not crash ================================================================= 10:19:05 (1693304345) [ 2284.778554] Lustre: DEBUG MARKER: == sanity test 63b: async write errors should be returned to fsync ============================================================= 10:20:07 (1693304407) [ 2284.970858] Lustre: *** cfs_fail_loc=406, val=0*** [ 2284.971023] LustreError: 107412:0:(osc_request.c:2766:osc_build_rpc()) prep_req failed: -12 [ 2284.971212] LustreError: 107412:0:(osc_cache.c:2175:osc_check_rpcs()) Write request failed with -12 [ 2288.864945] Lustre: DEBUG MARKER: == sanity test 64a: verify filter grant calculations (in kernel) =============================================================== 10:20:11 (1693304411) [ 2290.438946] Lustre: DEBUG MARKER: SKIP: sanity test_64b skipping SLOW test 64b [ 2290.818681] Lustre: DEBUG MARKER: == sanity test 64c: verify grant shrink ================== 10:20:13 (1693304413) [ 2292.254863] Lustre: DEBUG MARKER: == sanity test 64d: check grant limit exceed ============= 10:20:15 (1693304415) [ 2318.727696] Lustre: DEBUG MARKER: == sanity test 64e: check grant consumption (no grant allocation) ========================================================== 10:20:41 (1693304441) [ 2319.489568] systemd[1]: mnt-lustre.mount: Succeeded. [ 2320.160422] Lustre: Unmounted lustre-client [ 2320.521053] Lustre: Mounted lustre-client [ 2321.218849] Lustre: *** cfs_fail_loc=725, val=0*** [ 2321.992837] systemd[1]: mnt-lustre.mount: Succeeded. [ 2322.073396] Lustre: Unmounted lustre-client [ 2322.321690] Lustre: Mounted lustre-client [ 2322.900754] Lustre: *** cfs_fail_loc=725, val=0*** [ 2324.266455] Lustre: DEBUG MARKER: == sanity test 64f: check grant consumption (with grant allocation) ========================================================== 10:20:47 (1693304447) [ 2324.683911] systemd[1]: mnt-lustre.mount: Succeeded. [ 2324.774005] Lustre: Unmounted lustre-client [ 2325.010479] Lustre: Mounted lustre-client [ 2325.938258] systemd[1]: mnt-lustre.mount: Succeeded. [ 2326.414523] LustreError: 16008:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 3ms [ 2326.520063] LustreError: 16008:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake [ 2326.917137] LustreError: 106150:0:(fail.c:144:__cfs_fail_timeout_set()) cfs_fail_timeout interrupted [ 2327.706751] Lustre: DEBUG MARKER: == sanity test 64g: grant shrink on MDT ================== 10:20:50 (1693304450) [ 2331.225900] Lustre: DEBUG MARKER: == sanity test 64h: grant shrink on read ================= 10:20:54 (1693304454) [ 2340.637001] Lustre: DEBUG MARKER: == sanity test 64i: shrink on reconnect ================== 10:21:03 (1693304463) [ 2341.590173] Lustre: *** cfs_fail_loc=513, val=0*** [ 2341.598724] LustreError: 40400:0:(service.c:2130:ptlrpc_server_handle_req_in()) drop incoming rpc opc 17, x1775556018415808 [ 2341.767973] systemd[1]: mnt-lustre\x2dost1.mount: Succeeded. [ 2341.796359] Lustre: Failing over lustre-OST0000 [ 2342.230261] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2342.236835] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2342.237190] Lustre: Skipped 4 previous similar messages [ 2342.244212] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2342.244377] Lustre: Skipped 4 previous similar messages [ 2343.670271] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2343.670526] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2344.180390] Lustre: server umount lustre-OST0000 complete [ 2346.390603] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2346.392810] LustreError: Skipped 9 previous similar messages [ 2348.150799] Lustre: 3177:0:(client.c:2309:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1693304464/real 1693304464] req@00000000eed51cc4 x1775556018415808/t0(0) o17->lustre-OST0000-osc-ffff8f76256ef000@0@lo:28/4 lens 456/432 e 0 to 1 dl 1693304471 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 uid:0 gid:0 job:'lctl.0' [ 2356.484782] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2356.485081] LustreError: Skipped 3 previous similar messages [ 2356.988877] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2356.990788] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2358.798225] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2359.081831] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 2359.082845] Lustre: lustre-OST0000-osc-ffff8f76256ef000: Connection restored to 192.168.121.234@tcp (at 0@lo) [ 2359.083010] Lustre: Skipped 5 previous similar messages [ 2360.036015] Lustre: DEBUG MARKER: tmp.136UuqwYDI: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2360.467854] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 2362.716606] Lustre: DEBUG MARKER: SKIP: sanity test_65a skipping excluded test 65a [ 2363.077960] Lustre: DEBUG MARKER: == sanity test 65b: directory setstripe -S stripe_size*2 -i 0 -c 1 ========================================================== 10:21:26 (1693304486) [ 2364.275819] Lustre: DEBUG MARKER: == sanity test 65c: directory setstripe -S stripe_size*4 -i 1 -c 1 ========================================================== 10:21:27 (1693304487) [ 2365.076088] Lustre: DEBUG MARKER: == sanity test 65d: directory setstripe -S stripe_size -c stripe_count ========================================================== 10:21:28 (1693304488) [ 2366.126821] Lustre: DEBUG MARKER: == sanity test 65e: directory setstripe defaults ========= 10:21:29 (1693304489) [ 2367.022651] Lustre: DEBUG MARKER: == sanity test 65f: dir setstripe permission (should return error) ============================================================= 10:21:30 (1693304490) [ 2367.864144] Lustre: DEBUG MARKER: == sanity test 65g: directory setstripe -d =============== 10:21:30 (1693304490) [ 2369.055080] Lustre: DEBUG MARKER: == sanity test 65h: directory stripe info inherit ============================================================================== 10:21:32 (1693304492) [ 2370.385671] Lustre: DEBUG MARKER: == sanity test 65i: various tests to set root directory striping ========================================================== 10:21:33 (1693304493) [ 2373.133471] Lustre: DEBUG MARKER: == sanity test 65j: set default striping on root directory (bug 6367)=========================================================== 10:21:36 (1693304496) [ 2374.698649] systemd[1]: mnt-lustre.mount: Succeeded. [ 2374.841978] Lustre: Unmounted lustre-client [ 2374.845115] Lustre: Skipped 1 previous similar message [ 2375.019856] systemd[1]: mnt-lustre\x2dmds1.mount: Succeeded. [ 2376.036662] Lustre: server umount lustre-MDT0000 complete [ 2376.513950] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 2376.514195] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2376.514424] Lustre: Skipped 2 previous similar messages [ 2376.514649] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2376.514881] LustreError: Skipped 2 previous similar messages [ 2376.913271] systemd[1]: mnt-lustre\x2dmds2.mount: Succeeded. [ 2376.923274] LustreError: 5922:0:(ldlm_lockd.c:2573:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1693304500 with bad export cookie 10098730921851174555 [ 2376.924368] LustreError: 166-1: MGC192.168.121.234@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2377.146887] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2377.255470] Lustre: server umount lustre-MDT0001 complete [ 2378.202287] systemd[1]: mnt-lustre\x2dost1.mount: Succeeded. [ 2380.330427] Lustre: server umount lustre-OST0000 complete [ 2380.840878] systemd[1]: mnt-lustre\x2dost2.mount: Succeeded. [ 2381.097673] Lustre: server umount lustre-OST0001 complete [ 2385.281058] LNet: 111801:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2385.292119] LNet: Removed LNI 192.168.121.234@tcp [ 2386.565156] systemd-udevd[773]: Specified user 'tss' unknown [ 2386.590101] systemd-udevd[773]: Specified group 'tss' unknown [ 2386.743059] systemd-udevd[112132]: Using default interface naming scheme 'rhel-8.0'. [ 2387.282273] systemd[1]: usr-sbin-mount.lustre.mount: Succeeded. [ 2387.442456] systemd-udevd[773]: Specified user 'tss' unknown [ 2387.465462] systemd-udevd[773]: Specified group 'tss' unknown [ 2387.832397] systemd-udevd[112226]: Using default interface naming scheme 'rhel-8.0'. [ 2388.523652] libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2 [ 2389.643583] Lustre: Lustre: Build Version: 2.15.57 [ 2389.809224] LNet: Added LNI 192.168.121.234@tcp [8/256/0/180] [ 2389.823965] LNet: Accept secure, port 988 [ 2390.694537] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2393.634996] systemd-udevd[773]: Specified user 'tss' unknown [ 2393.668903] systemd-udevd[773]: Specified group 'tss' unknown [ 2393.811168] systemd-udevd[113207]: Using default interface naming scheme 'rhel-8.0'. [ 2397.284319] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2398.822820] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2398.911457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2401.348783] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2403.521388] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2404.552641] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2404.561019] Lustre: lustre-OST0000: deleting orphan objects from 0x280000bd0:6232 to 0x280000bd0:6273 [ 2406.299802] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000bd0:6033 to 0x2c0000bd0:6049 [ 2406.390501] Lustre: Mounted lustre-client [ 2407.524929] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:3798 to 0x2c0000400:3841 [ 2407.561457] Lustre: lustre-OST0000: deleting orphan objects from 0x280000bd1:3164 to 0x280000bd1:3201 [ 2412.886974] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2413.175425] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 2419.190735] Lustre: DEBUG MARKER: SKIP: sanity test_65k skipping excluded test 65k [ 2419.500628] Lustre: DEBUG MARKER: == sanity test 65l: lfs find on -1 stripe dir ================================================================================== 10:22:22 (1693304542) [ 2420.682067] Lustre: DEBUG MARKER: == sanity test 65m: normal user can't set filesystem default stripe ========================================================== 10:22:23 (1693304543) [ 2421.874700] Lustre: DEBUG MARKER: == sanity test 65n: don't inherit default layout from root for new subdirectories ========================================================== 10:22:24 (1693304544) [ 2434.106723] Lustre: DEBUG MARKER: == sanity test 65o: pool inheritance for mdt component === 10:22:37 (1693304557) [ 2444.555462] Lustre: DEBUG MARKER: == sanity test 65p: setstripe with yaml file and huge number ========================================================== 10:22:47 (1693304567) [ 2445.582462] Lustre: DEBUG MARKER: == sanity test 66: update inode blocks count on client ========================================================================= 10:22:48 (1693304568) [ 2447.616183] Lustre: DEBUG MARKER: == sanity test 69: verify oa2dentry return -ENOENT doesn't LBUG ================================================================ 10:22:50 (1693304570) [ 2448.068162] Lustre: *** cfs_fail_loc=217, val=0*** [ 2449.193626] Lustre: DEBUG MARKER: SKIP: sanity test_71 skipping SLOW test 71 [ 2449.483888] Lustre: DEBUG MARKER: == sanity test 72a: Test that remove suid works properly (bug5695) ============================================================== 10:22:52 (1693304572) [ 2450.444146] Lustre: DEBUG MARKER: == sanity test 72b: Test that we keep mode setting if without file data changed (bug 24226) ========================================================== 10:22:53 (1693304573) [ 2452.246163] Lustre: DEBUG MARKER: == sanity test 73: multiple MDC requests (should not deadlock) ========================================================== 10:22:55 (1693304575) [ 2452.722299] LustreError: 113715:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 129 sleeping for 5000ms [ 2453.712434] LustreError: 113715:0:(fail.c:144:__cfs_fail_timeout_set()) cfs_fail_timeout interrupted [ 2479.746559] Lustre: DEBUG MARKER: == sanity test 74a: ldlm_enqueue freed-export error path, ls (shouldn't LBUG) ========================================================== 10:23:22 (1693304602) [ 2479.817422] Lustre: *** cfs_fail_loc=30e, val=0*** [ 2479.817586] LustreError: 113718:0:(ldlm_lockd.c:1480:ldlm_handle_enqueue()) ### lock on destroyed export 00000000b157834b ns: mdt-lustre-MDT0000_UUID lock: 000000005bec7166/0x2b3b843e1b9f5ca lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x2b3b843e1b9f5bc expref: 14 pid: 113718 timeout: 0 lvb_type: 0 initiator: MDT0 [ 2479.818106] LustreError: 11-0: lustre-MDT0000-mdc-ffff8f75c9d78000: operation ldlm_enqueue to node 0@lo failed: rc = -107 [ 2479.818282] Lustre: lustre-MDT0000-mdc-ffff8f75c9d78000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2479.818680] Lustre: lustre-MDT0000: Client 039def72-9e11-41b5-8382-d542a2089cf6 (at 0@lo) reconnecting [ 2479.818952] Lustre: lustre-MDT0000-mdc-ffff8f75c9d78000: Connection restored to 192.168.121.234@tcp (at 0@lo) [ 2480.542572] Lustre: DEBUG MARKER: == sanity test 74b: ldlm_enqueue freed-export error path, touch (shouldn't LBUG) ========================================================== 10:23:23 (1693304603) [ 2480.560922] Lustre: *** cfs_fail_loc=30e, val=0*** [ 2480.561257] LustreError: 113714:0:(ldlm_lockd.c:1480:ldlm_handle_enqueue()) ### lock on destroyed export 00000000b157834b ns: mdt-lustre-MDT0000_UUID lock: 00000000d7a49a0c/0x2b3b843e1b9f63a lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x2b3b843e1b9f62c expref: 15 pid: 113714 timeout: 0 lvb_type: 0 initiator: MDT0 [ 2480.562044] LustreError: 11-0: lustre-MDT0000-mdc-ffff8f75c9d78000: operation ldlm_enqueue to node 0@lo failed: rc = -107 [ 2480.562299] Lustre: lustre-MDT0000-mdc-ffff8f75c9d78000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2480.562769] Lustre: lustre-MDT0000: Client 039def72-9e11-41b5-8382-d542a2089cf6 (at 0@lo) reconnecting [ 2480.563036] Lustre: lustre-MDT0000-mdc-ffff8f75c9d78000: Connection restored to 192.168.121.234@tcp (at 0@lo) [ 2481.459291] Lustre: DEBUG MARKER: == sanity test 74c: ldlm_lock_create error path, (shouldn't LBUG) ========================================================== 10:23:24 (1693304604) [ 2481.497807] Lustre: *** cfs_fail_loc=319, val=0*** [ 2482.253107] Lustre: DEBUG MARKER: == sanity test 76a: confirm clients recycle inodes properly ============================================================== 10:23:25 (1693304605) [ 2492.368933] Lustre: DEBUG MARKER: == sanity test 76b: confirm clients recycle directory inodes properly ============================================================== 10:23:35 (1693304615) [ 2513.350650] Lustre: lustre-OST0000-osc-ffff8f75c9d78000: disconnect after 21s idle [ 2518.877065] Lustre: DEBUG MARKER: == sanity test 77a: normal checksum read/write operation ========================================================== 10:24:02 (1693304642) [ 2519.907488] Lustre: DEBUG MARKER: == sanity test 77b: checksum error on client write, read ========================================================== 10:24:03 (1693304643) [ 2520.005435] Lustre: *** cfs_fail_loc=409, val=0*** [ 2520.042961] LustreError: 168-f: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-0@lo inode [0x200001b72:0x31e:0x0] object 0x280000bd0:6410 extent [0-1048575]: client csum 7c5afdc2, server csum 7c5afdc1 [ 2520.091909] LustreError: 132-0: lustre-OST0000-osc-ffff8f75c9d78000: BAD WRITE CHECKSUM: changed on the client after we checksummed it - likely false positive due to mmap IO (bug 11742): from 0@lo inode [0x200001b72:0x31e:0x0] object 0x280000bd0:6410 extent [0-1048575], original client csum 7c5afdc2 (type 20), server csum 7c5afdc1 (type 20), client csum now 7c5afdc1 [ 2520.092430] LustreError: 112550:0:(osc_request.c:2409:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@00000000ce654d02 x1775558473694592/t30064771358(30064771358) o4->lustre-OST0000-osc-ffff8f75c9d78000@0@lo:6/4 lens 488/448 e 0 to 0 dl 1693304650 ref 3 fl Interpret:RQU/204/0 rc 0/0 uid:0 gid:0 job:'dd.0' [ 2521.420092] Lustre: DEBUG MARKER: set checksum type to crc32, rc = 0 [ 2521.465327] Lustre: *** cfs_fail_loc=408, val=0*** [ 2521.467053] LustreError: 133-1: lustre-OST0000-osc-ffff8f75c9d78000: BAD READ CHECKSUM: from 0@lo inode [0x200001b72:0x31e:0x0] object 0x280000bd0:6410 extent [1048576-2097151], client 815fdb08/815fdb08, server b56186d3, cksum_type 1 [ 2521.467374] LustreError: 112547:0:(osc_request.c:2409:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@000000004ee2b888 x1775558473696512/t0(0) o3->lustre-OST0000-osc-ffff8f75c9d78000@0@lo:6/4 lens 488/440 e 0 to 0 dl 1693304651 ref 2 fl Interpret:RMQU/200/0 rc 1048576/1048576 uid:0 gid:0 job:'cmp.0' [ 2522.231425] LustreError: 132-0: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 0@lo inode [0x200001b72:0x31e:0x0] object 0x280000bd0:6410 extent [1048576-2097151], client returned csum 815fdb08 (type 1), server csum b56186d3 (type 1) [ 2522.567597] Lustre: DEBUG MARKER: set checksum type to adler, rc = 0 [ 2522.634725] Lustre: *** cfs_fail_loc=408, val=0*** [ 2522.635985] LustreError: 133-1: lustre-OST0000-osc-ffff8f75c9d78000: BAD READ CHECKSUM: from 0@lo inode [0x200001b72:0x31e:0x0] object 0x280000bd0:6410 extent [1048576-2097151], client f6740c16/f6740c16, server 4b870d85, cksum_type 2 [ 2522.636274] LustreError: 112550:0:(osc_request.c:2409:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@00000000f3538bd2 x1775558473698176/t0(0) o3->lustre-OST0000-osc-ffff8f75c9d78000@0@lo:6/4 lens 488/440 e 0 to 0 dl 1693304652 ref 2 fl Interpret:RMQU/200/0 rc 1048576/1048576 uid:0 gid:0 job:'cmp.0' [ 2523.670986] LustreError: 132-0: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 0@lo inode [0x200001b72:0x31e:0x0] object 0x280000bd0:6410 extent [1048576-2097151], client returned csum f6740c16 (type 2), server csum 4b870d85 (type 2) [ 2524.038546] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 2524.118277] LustreError: 133-1: lustre-OST0000-osc-ffff8f75c9d78000: BAD READ CHECKSUM: from 0@lo inode [0x200001b72:0x31e:0x0] object 0x280000bd0:6410 extent [0-1048575], client db6a8029/db6a8029, server 131e0fe9, cksum_type 4 [ 2525.121952] LustreError: 132-0: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 0@lo inode [0x200001b72:0x31e:0x0] object 0x280000bd0:6410 extent [0-1048575], client returned csum db6a8029 (type 4), server csum 131e0fe9 (type 4) [ 2525.505502] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 2525.536516] Lustre: *** cfs_fail_loc=408, val=0*** [ 2525.539630] Lustre: Skipped 1 previous similar message [ 2525.546989] LustreError: 112548:0:(osc_request.c:2409:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@000000004e65ed7d x1775558473702272/t0(0) o3->lustre-OST0000-osc-ffff8f75c9d78000@0@lo:6/4 lens 488/440 e 0 to 0 dl 1693304655 ref 2 fl Interpret:RMQU/200/0 rc 1048576/1048576 uid:0 gid:0 job:'cmp.0' [ 2525.547721] LustreError: 112548:0:(osc_request.c:2409:osc_brw_redo_request()) Skipped 1 previous similar message [ 2526.887659] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 2526.934738] LustreError: 133-1: lustre-OST0000-osc-ffff8f75c9d78000: BAD READ CHECKSUM: from 0@lo inode [0x200001b72:0x31e:0x0] object 0x280000bd0:6410 extent [0-1048575], client a484fdd5/a484fdd5, server 7c5afdc1, cksum_type 20 [ 2526.937663] LustreError: Skipped 1 previous similar message [ 2527.990549] LustreError: 132-0: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 0@lo inode [0x200001b72:0x31e:0x0] object 0x280000bd0:6410 extent [0-1048575], client returned csum a484fdd5 (type 20), server csum 7c5afdc1 (type 20) [ 2527.995373] LustreError: Skipped 1 previous similar message [ 2528.273097] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 2529.231040] Lustre: DEBUG MARKER: == sanity test 77c: checksum error on client read with debug ========================================================== 10:24:12 (1693304652) [ 2529.866461] Lustre: *** cfs_fail_loc=408, val=0*** [ 2529.871534] Lustre: Skipped 1 previous similar message [ 2529.883168] Lustre: 112548:0:(osc_request.c:1987:dump_all_bulk_pages()) dumping checksum data to /tmp/lustre-log-checksum_dump-osc-[0x200001b72:0x31f:0x0]:[0-1048575]-a484fdd5-7c5afdc1 [ 2529.890472] LustreError: dumping log to /tmp/lustre-log.1693304653.112548 [ 2529.911041] LustreError: 112548:0:(osc_request.c:2409:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@000000005a3f8c7d x1775558473708736/t0(0) o3->lustre-OST0000-osc-ffff8f75c9d78000@0@lo:6/4 lens 488/440 e 0 to 0 dl 1693304660 ref 2 fl Interpret:RMQU/200/0 rc 1048576/1048576 uid:0 gid:0 job:'dd.0' [ 2529.911678] LustreError: 112548:0:(osc_request.c:2409:osc_brw_redo_request()) Skipped 1 previous similar message [ 2531.590586] Lustre: 116831:0:(tgt_handler.c:1942:dump_all_bulk_pages()) dumping checksum data to /tmp/lustre-log-checksum_dump-ost-[0x200001b72:0x31f:0x0]:[0-1048575]-a484fdd5-7c5afdc1 [ 2531.607865] LustreError: dumping log to /tmp/lustre-log.1693304654.116831 [ 2541.865873] Lustre: DEBUG MARKER: == sanity test 77d: checksum error on OST direct write, read ========================================================== 10:24:24 (1693304664) [ 2541.981557] Lustre: *** cfs_fail_loc=409, val=0*** [ 2542.049699] LustreError: 168-f: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-0@lo inode [0x200001b72:0x321:0x0] object 0x2c0000bd0:6185 extent [0-1048575]: client csum 30ec5402, server csum 30ec5401 [ 2542.173776] LustreError: 132-0: lustre-OST0001-osc-ffff8f75c9d78000: BAD WRITE CHECKSUM: changed on the client after we checksummed it - likely false positive due to mmap IO (bug 11742): from 0@lo inode [0x200001b72:0x321:0x0] object 0x2c0000bd0:6185 extent [0-1048575], original client csum 30ec5402 (type 20), server csum 30ec5401 (type 20), client csum now 30ec5401 [ 2542.188424] LustreError: 112548:0:(osc_request.c:2409:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@000000000ad798ef x1775558473713600/t17179869476(17179869476) o4->lustre-OST0001-osc-ffff8f75c9d78000@0@lo:6/4 lens 488/448 e 0 to 0 dl 1693304672 ref 2 fl Interpret:RMQU/200/0 rc 0/0 uid:0 gid:0 job:'directio.0' [ 2543.434790] LustreError: 133-1: lustre-OST0001-osc-ffff8f75c9d78000: BAD READ CHECKSUM: from 0@lo inode [0x200001b72:0x321:0x0] object 0x2c0000bd0:6185 extent [1048576-2097151], client 53195492/53195492, server 30ec5401, cksum_type 20 [ 2543.435136] LustreError: Skipped 1 previous similar message [ 2544.630540] LustreError: 132-0: lustre-OST0001: BAD READ CHECKSUM: should have changed on the client or in transit: from 0@lo inode [0x200001b72:0x321:0x0] object 0x2c0000bd0:6185 extent [1048576-2097151], client returned csum 53195492 (type 20), server csum 30ec5401 (type 20) [ 2544.636969] LustreError: Skipped 1 previous similar message [ 2545.446772] Lustre: DEBUG MARKER: == sanity test 77f: repeat checksum error on write (expect error) ========================================================== 10:24:28 (1693304668) [ 2545.776946] Lustre: DEBUG MARKER: set checksum type to crc32, rc = 0 [ 2545.819537] LustreError: 168-f: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-0@lo inode [0x200001b72:0x322:0x0] object 0x280000bd0:6412 extent [1048576-2097151]: client csum b2f1b12, server csum b2f1b11 [ 2545.886092] LustreError: 132-0: lustre-OST0000-osc-ffff8f75c9d78000: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 0@lo inode [0x200001b72:0x322:0x0] object 0x280000bd0:6412 extent [1048576-2097151], original client csum b2f1b12 (type 1), server csum b2f1b11 (type 1), client csum now b2f1b12 [ 2549.201467] LustreError: 168-f: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-0@lo inode [0x200001b72:0x322:0x0] object 0x280000bd0:6412 extent [2097152-3145727]: client csum b2f1b12, server csum b2f1b11 [ 2549.205710] LustreError: Skipped 15 previous similar messages [ 2549.224054] LustreError: 132-0: lustre-OST0000-osc-ffff8f75c9d78000: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 0@lo inode [0x200001b72:0x322:0x0] object 0x280000bd0:6412 extent [2097152-3145727], original client csum b2f1b12 (type 1), server csum b2f1b11 (type 1), client csum now b2f1b12 [ 2549.224538] LustreError: Skipped 15 previous similar messages [ 2555.751652] LustreError: 168-f: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-0@lo inode [0x200001b72:0x322:0x0] object 0x280000bd0:6412 extent [1048576-2097151]: client csum b2f1b12, server csum b2f1b11 [ 2555.755015] LustreError: Skipped 15 previous similar messages [ 2555.773010] LustreError: 132-0: lustre-OST0000-osc-ffff8f75c9d78000: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 0@lo inode [0x200001b72:0x322:0x0] object 0x280000bd0:6412 extent [1048576-2097151], original client csum b2f1b12 (type 1), server csum b2f1b11 (type 1), client csum now b2f1b12 [ 2555.773682] LustreError: Skipped 15 previous similar messages [ 2560.805598] Lustre: *** cfs_fail_loc=409, val=0*** [ 2560.805745] Lustre: Skipped 89 previous similar messages [ 2560.805835] LustreError: 112548:0:(osc_request.c:2409:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@00000000e67cec9b x1775558473724032/t30064771418(30064771418) o4->lustre-OST0000-osc-ffff8f75c9d78000@0@lo:6/4 lens 488/448 e 0 to 0 dl 1693304691 ref 3 fl Interpret:RQU/204/0 rc 0/0 uid:0 gid:0 job:'ptlrpcd_00_01.0' [ 2560.806223] LustreError: 112548:0:(osc_request.c:2409:osc_brw_redo_request()) Skipped 41 previous similar messages [ 2566.871446] LustreError: 168-f: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-0@lo inode [0x200001b72:0x322:0x0] object 0x280000bd0:6412 extent [1048576-2097151]: client csum b2f1b12, server csum b2f1b11 [ 2566.876888] LustreError: Skipped 15 previous similar messages [ 2566.894343] LustreError: 132-0: lustre-OST0000-osc-ffff8f75c9d78000: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 0@lo inode [0x200001b72:0x322:0x0] object 0x280000bd0:6412 extent [1048576-2097151], original client csum b2f1b12 (type 1), server csum b2f1b11 (type 1), client csum now b2f1b12 [ 2566.895067] LustreError: Skipped 15 previous similar messages [ 2568.790716] Lustre: lustre-OST0001-osc-ffff8f75c9d78000: disconnect after 24s idle [ 2568.797719] Lustre: Skipped 1 previous similar message [ 2584.951552] LustreError: 168-f: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-0@lo inode [0x200001b72:0x322:0x0] object 0x280000bd0:6412 extent [1048576-2097151]: client csum b2f1b12, server csum b2f1b11 [ 2584.954373] LustreError: Skipped 23 previous similar messages [ 2584.972385] LustreError: 132-0: lustre-OST0000-osc-ffff8f75c9d78000: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 0@lo inode [0x200001b72:0x322:0x0] object 0x280000bd0:6412 extent [1048576-2097151], original client csum b2f1b12 (type 1), server csum b2f1b11 (type 1), client csum now b2f1b12 [ 2584.972609] LustreError: Skipped 23 previous similar messages [ 2591.045439] LustreError: 112548:0:(osc_request.c:2539:brw_interpret()) lustre-OST0000-osc-ffff8f75c9d78000: too many resent retries for object: 10737421264:6412, rc = -11. [ 2591.605528] LustreError: 112550:0:(osc_request.c:2539:brw_interpret()) lustre-OST0000-osc-ffff8f75c9d78000: too many resent retries for object: 10737421264:6412, rc = -11. [ 2592.653558] LustreError: 112550:0:(osc_request.c:2539:brw_interpret()) lustre-OST0000-osc-ffff8f75c9d78000: too many resent retries for object: 10737421264:6412, rc = -11. [ 2592.653834] LustreError: 112550:0:(osc_request.c:2539:brw_interpret()) Skipped 4 previous similar messages [ 2592.993608] Lustre: DEBUG MARKER: set checksum type to adler, rc = 0 [ 2593.035028] Lustre: *** cfs_fail_loc=409, val=0*** [ 2593.035184] Lustre: Skipped 87 previous similar messages [ 2593.063819] LustreError: 112549:0:(osc_request.c:2409:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@00000000dd43e934 x1775558473737024/t30064771466(30064771466) o4->lustre-OST0000-osc-ffff8f75c9d78000@0@lo:6/4 lens 488/448 e 0 to 0 dl 1693304723 ref 3 fl Interpret:RQU/204/0 rc 0/0 uid:0 gid:0 job:'directio.0' [ 2593.079392] LustreError: 112549:0:(osc_request.c:2409:osc_brw_redo_request()) Skipped 39 previous similar messages [ 2619.991315] LustreError: 168-f: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-0@lo inode [0x200001b72:0x322:0x0] object 0x280000bd0:6412 extent [5242880-6291455]: client csum 19eeae62, server csum 19eeae61 [ 2619.995420] LustreError: Skipped 71 previous similar messages [ 2620.037819] LustreError: 132-0: lustre-OST0000-osc-ffff8f75c9d78000: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 0@lo inode [0x200001b72:0x322:0x0] object 0x280000bd0:6412 extent [5242880-6291455], original client csum 19eeae62 (type 2), server csum 19eeae61 (type 2), client csum now 19eeae62 [ 2620.040647] LustreError: Skipped 71 previous similar messages [ 2638.099040] LustreError: 112548:0:(osc_request.c:2539:brw_interpret()) lustre-OST0000-osc-ffff8f75c9d78000: too many resent retries for object: 10737421264:6412, rc = -11. [ 2638.099336] LustreError: 112548:0:(osc_request.c:2539:brw_interpret()) Skipped 1 previous similar message [ 2638.405812] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 2659.292781] Lustre: *** cfs_fail_loc=409, val=0*** [ 2659.304646] Lustre: Skipped 279 previous similar messages [ 2659.304711] LustreError: 112548:0:(osc_request.c:2409:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@00000000c4898790 x1775558473763392/t30064771602(30064771602) o4->lustre-OST0000-osc-ffff8f75c9d78000@0@lo:6/4 lens 488/448 e 0 to 0 dl 1693304789 ref 3 fl Interpret:RQU/204/0 rc 0/0 uid:0 gid:0 job:'ptlrpcd_00_00.0' [ 2659.304862] LustreError: 112548:0:(osc_request.c:2409:osc_brw_redo_request()) Skipped 127 previous similar messages [ 2683.458786] LustreError: 112547:0:(osc_request.c:2539:brw_interpret()) lustre-OST0000-osc-ffff8f75c9d78000: too many resent retries for object: 10737421264:6412, rc = -11. [ 2683.459531] LustreError: 112547:0:(osc_request.c:2539:brw_interpret()) Skipped 7 previous similar messages [ 2684.145969] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 2684.187345] LustreError: 168-f: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-0@lo inode [0x200001b72:0x322:0x0] object 0x280000bd0:6412 extent [0-1048575]: client csum 3dbc503e, server csum 3dbc503d [ 2684.200667] LustreError: Skipped 125 previous similar messages [ 2684.221937] LustreError: 132-0: lustre-OST0000-osc-ffff8f75c9d78000: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 0@lo inode [0x200001b72:0x322:0x0] object 0x280000bd0:6412 extent [0-1048575], original client csum 3dbc503e (type 10), server csum 3dbc503d (type 10), client csum now 3dbc503e [ 2684.227886] LustreError: Skipped 125 previous similar messages [ 2728.815930] LustreError: 112547:0:(osc_request.c:2539:brw_interpret()) lustre-OST0000-osc-ffff8f75c9d78000: too many resent retries for object: 10737421264:6412, rc = -11. [ 2728.816204] LustreError: 112547:0:(osc_request.c:2539:brw_interpret()) Skipped 7 previous similar messages [ 2730.521376] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 2775.213430] LustreError: 112549:0:(osc_request.c:2539:brw_interpret()) lustre-OST0000-osc-ffff8f75c9d78000: too many resent retries for object: 10737421264:6412, rc = -11. [ 2775.213718] LustreError: 112549:0:(osc_request.c:2539:brw_interpret()) Skipped 7 previous similar messages [ 2776.822140] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 2777.838331] Lustre: DEBUG MARKER: SKIP: sanity test_77g skipping excluded test 77g [ 2778.243057] Lustre: DEBUG MARKER: == sanity test 77k: enable/disable checksum correctly ==== 10:28:21 (1693304901) [ 2778.375738] Lustre: Setting parameter lustre.osc.lustre*.checksums in log params [ 2779.043099] systemd[1]: mnt-lustre.mount: Succeeded. [ 2779.163141] Lustre: Unmounted lustre-client [ 2779.375945] Lustre: Mounted lustre-client [ 2779.660624] Lustre: Modifying parameter lustre.osc.lustre*.checksums in log params [ 2780.208714] systemd[1]: mnt-lustre.mount: Succeeded. [ 2780.270127] Lustre: Unmounted lustre-client [ 2780.429426] Lustre: Mounted lustre-client [ 2781.384864] systemd[1]: mnt-lustre.mount: Succeeded. [ 2781.453626] Lustre: Unmounted lustre-client [ 2782.525779] systemd[1]: mnt-lustre.mount: Succeeded. [ 2782.877314] Lustre: Mounted lustre-client [ 2782.877567] Lustre: Skipped 1 previous similar message [ 2783.804890] systemd[1]: mnt-lustre.mount: Succeeded. [ 2783.882762] Lustre: Unmounted lustre-client [ 2783.886440] Lustre: Skipped 1 previous similar message [ 2785.203126] Lustre: Setting parameter lustre.osc.lustre*.checksums in log params [ 2785.209525] Lustre: Skipped 1 previous similar message [ 2787.882120] Lustre: DEBUG MARKER: == sanity test 77l: preferred checksum type is remembered after reconnected ========================================================== 10:28:30 (1693304910) [ 2788.382971] Lustre: DEBUG MARKER: set checksum type to invalid, rc = 22 [ 2788.754033] Lustre: DEBUG MARKER: set checksum type to crc32, rc = 0 [ 2790.316769] Lustre: DEBUG MARKER: tmp.136UuqwYDI: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8f75db91a000.ost_server_uuid 40 [ 2805.562932] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8f75db91a000.ost_server_uuid in IDLE state after 14 sec [ 2806.650548] Lustre: DEBUG MARKER: tmp.136UuqwYDI: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8f75db91a000.ost_server_uuid 40 [ 2806.938263] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8f75db91a000.ost_server_uuid in FULL state after 0 sec [ 2807.215060] Lustre: DEBUG MARKER: set checksum type to adler, rc = 0 [ 2808.456302] Lustre: DEBUG MARKER: tmp.136UuqwYDI: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8f75db91a000.ost_server_uuid 40 [ 2830.195386] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8f75db91a000.ost_server_uuid in IDLE state after 20 sec [ 2831.081543] Lustre: DEBUG MARKER: tmp.136UuqwYDI: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8f75db91a000.ost_server_uuid 40 [ 2831.256314] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8f75db91a000.ost_server_uuid in FULL state after 0 sec [ 2831.479718] Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0 [ 2832.457116] Lustre: DEBUG MARKER: tmp.136UuqwYDI: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8f75db91a000.ost_server_uuid 40 [ 2855.096650] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8f75db91a000.ost_server_uuid in IDLE state after 21 sec [ 2856.091530] Lustre: DEBUG MARKER: tmp.136UuqwYDI: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8f75db91a000.ost_server_uuid 40 [ 2856.338228] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8f75db91a000.ost_server_uuid in FULL state after 0 sec [ 2856.573774] Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0 [ 2857.477355] Lustre: DEBUG MARKER: tmp.136UuqwYDI: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8f75db91a000.ost_server_uuid 40 [ 2880.303872] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8f75db91a000.ost_server_uuid in IDLE state after 21 sec [ 2881.135676] Lustre: DEBUG MARKER: tmp.136UuqwYDI: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8f75db91a000.ost_server_uuid 40 [ 2881.330506] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8f75db91a000.ost_server_uuid in FULL state after 0 sec [ 2881.597272] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 2882.666518] Lustre: DEBUG MARKER: tmp.136UuqwYDI: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8f75db91a000.ost_server_uuid 40 [ 2906.413612] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8f75db91a000.ost_server_uuid in IDLE state after 22 sec [ 2907.685095] Lustre: DEBUG MARKER: tmp.136UuqwYDI: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8f75db91a000.ost_server_uuid 40 [ 2907.984068] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8f75db91a000.ost_server_uuid in FULL state after 0 sec [ 2908.880101] Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0 [ 2909.374785] Lustre: DEBUG MARKER: == sanity test 77m: Verify checksum_speed is correctly read ========================================================== 10:30:32 (1693305032) [ 2910.395645] Lustre: DEBUG MARKER: == sanity test 77n: Verify read from a hole inside contiguous blocks with T10PI ========================================================== 10:30:33 (1693305033) [ 2910.789459] Lustre: DEBUG MARKER: SKIP: sanity test_77n f77n.sanity blocks not contiguous around hole [ 2911.173058] Lustre: DEBUG MARKER: == sanity test 77o: Verify checksum_type for server (mdt and ofd(obdfilter)) ========================================================== 10:30:34 (1693305034) [ 2913.199855] Lustre: DEBUG MARKER: == sanity test 78: handle large O_DIRECT writes correctly ====================================================================== 10:30:36 (1693305036) [ 2913.623354] Lustre: DEBUG MARKER: SKIP: sanity test_78 local OST [ 2914.028818] Lustre: DEBUG MARKER: == sanity test 79: df report consistency check ================================================================================= 10:30:37 (1693305037) [ 2920.812389] Lustre: DEBUG MARKER: == sanity test 80: Page eviction is equally fast at high offsets too ========================================================== 10:30:43 (1693305043) [ 2923.154861] Lustre: DEBUG MARKER: == sanity test 81a: OST should retry write when get -ENOSPC ========================================================================= 10:30:46 (1693305046) [ 2923.392795] Lustre: *** cfs_fail_loc=228, val=0*** [ 2924.137662] Lustre: DEBUG MARKER: == sanity test 81b: OST should return -ENOSPC when retry still fails ================================================================= 10:30:47 (1693305047) [ 2924.326559] Lustre: *** cfs_fail_loc=228, val=0*** [ 2925.121086] Lustre: DEBUG MARKER: == sanity test 99: cvs strange file/directory operations ========================================================== 10:30:48 (1693305048) [ 2930.037172] Lustre: DEBUG MARKER: == sanity test 100: check local port using privileged port ========================================================== 10:30:53 (1693305053) [ 2930.870912] Lustre: DEBUG MARKER: SKIP: sanity test_100 useless for local single node setup [ 2931.243140] Lustre: DEBUG MARKER: == sanity test 101a: check read-ahead for random reads === 10:30:54 (1693305054) [ 2941.020517] Lustre: DEBUG MARKER: == sanity test 101b: check stride-io mode read-ahead =========================================================================== 10:31:04 (1693305064) [ 2945.216471] Lustre: DEBUG MARKER: == sanity test 101c: check stripe_size aligned read-ahead ========================================================== 10:31:08 (1693305068) [ 2950.620503] Lustre: DEBUG MARKER: == sanity test 101d: file read with and without read-ahead enabled ========================================================== 10:31:13 (1693305073) [ 2987.837096] Lustre: DEBUG MARKER: == sanity test 101e: check read-ahead for small read(1k) for small files(500k) ========================================================== 10:31:50 (1693305110) [ 3002.006976] Lustre: DEBUG MARKER: == sanity test 101f: check mmap read performance ========= 10:32:05 (1693305125) [ 3002.319565] Lustre: DEBUG MARKER: SKIP: sanity test_101f no iozone installed [ 3002.625959] Lustre: DEBUG MARKER: SKIP: sanity test_101g skipping excluded test 101g [ 3002.961052] Lustre: DEBUG MARKER: == sanity test 101h: Readahead should cover current read window ========================================================== 10:32:05 (1693305125) [ 3004.930044] Lustre: DEBUG MARKER: == sanity test 101i: allow current readahead to exceed reservation ========================================================== 10:32:07 (1693305127) [ 3006.734627] Lustre: DEBUG MARKER: == sanity test 101j: A complete read block should be submitted when no RA ========================================================== 10:32:09 (1693305129) [ 3011.951283] Lustre: DEBUG MARKER: == sanity test 101m: read ahead for small file and last stripe of the file ========================================================== 10:32:14 (1693305134) [ 3012.164125] Lustre: DEBUG MARKER: SKIP: sanity test_101m need >= 2.13.57 and ldiskfs for fallocate [ 3012.544237] Lustre: DEBUG MARKER: SKIP: sanity test_102a skipping excluded test 102a (base 102) [ 3012.811467] Lustre: DEBUG MARKER: SKIP: sanity test_102b skipping excluded test 102b (base 102) [ 3013.044653] Lustre: DEBUG MARKER: SKIP: sanity test_102c skipping excluded test 102c (base 102) [ 3013.309675] Lustre: DEBUG MARKER: SKIP: sanity test_102d skipping excluded test 102d (base 102) [ 3013.537415] Lustre: DEBUG MARKER: SKIP: sanity test_102f skipping excluded test 102f (base 102) [ 3013.749951] Lustre: DEBUG MARKER: SKIP: sanity test_102h skipping excluded test 102h (base 102) [ 3013.987509] Lustre: DEBUG MARKER: SKIP: sanity test_102ha skipping excluded test 102ha (base 102) [ 3014.218613] Lustre: DEBUG MARKER: SKIP: sanity test_102i skipping excluded test 102i (base 102) [ 3014.482685] Lustre: DEBUG MARKER: SKIP: sanity test_102j skipping excluded test 102j (base 102) [ 3014.711412] Lustre: DEBUG MARKER: SKIP: sanity test_102k skipping excluded test 102k (base 102) [ 3014.950384] Lustre: DEBUG MARKER: SKIP: sanity test_102l skipping excluded test 102l (base 102) [ 3015.182238] Lustre: DEBUG MARKER: SKIP: sanity test_102m skipping excluded test 102m (base 102) [ 3015.445965] Lustre: DEBUG MARKER: SKIP: sanity test_102n skipping excluded test 102n (base 102) [ 3015.686470] Lustre: DEBUG MARKER: SKIP: sanity test_102p skipping excluded test 102p (base 102) [ 3015.912302] Lustre: DEBUG MARKER: SKIP: sanity test_102q skipping excluded test 102q (base 102) [ 3016.132480] Lustre: DEBUG MARKER: SKIP: sanity test_102r skipping excluded test 102r (base 102) [ 3016.361371] Lustre: DEBUG MARKER: SKIP: sanity test_102s skipping excluded test 102s (base 102) [ 3016.589862] Lustre: DEBUG MARKER: SKIP: sanity test_102t skipping excluded test 102t (base 102) [ 3016.869316] Lustre: DEBUG MARKER: == sanity test 103a: acl test ============================ 10:32:19 (1693305139) [ 3123.079010] Lustre: DEBUG MARKER: == sanity test 103b: umask lfs setstripe ================= 10:34:06 (1693305246) [ 3195.026602] Lustre: DEBUG MARKER: == sanity test 103c: 'cp -rp' won't set empty acl ======== 10:35:18 (1693305318) [ 3196.667106] Lustre: DEBUG MARKER: == sanity test 103e: inheritance of big amount of default ACLs ========================================================== 10:35:19 (1693305319) [ 3366.870068] Lustre: 145997:0:(client.c:2309:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1693305444/real 1693305444] req@000000008865180a x1775558493312320/t0(0) o49->lustre-MDT0000-mdc-ffff8f75db91a000@0@lo:23/10 lens 488/45640 e 0 to 1 dl 1693305488 ref 2 fl Rpc:XQr/202/ffffffff rc 0/-1 uid:0 gid:0 job:'setfacl.0' [ 3366.877281] Lustre: lustre-MDT0000-mdc-ffff8f75db91a000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3366.884608] Lustre: lustre-MDT0000: Client 7e5da208-bba9-428b-b525-7ed27e0ff422 (at 0@lo) reconnecting [ 3366.884908] Lustre: lustre-MDT0000-mdc-ffff8f75db91a000: Connection restored to 192.168.121.234@tcp (at 0@lo) [ 3971.943138] Lustre: DEBUG MARKER: == sanity test 103f: changelog doesn't interfere with default ACLs buffers ========================================================== 10:48:15 (1693306095) [ 3972.032163] Lustre: lustre-MDD0000: changelog on [ 3972.489152] Lustre: lustre-MDD0001: changelog off [ 3972.741730] Lustre: DEBUG MARKER: == sanity test 104a: lfs df [-ih] [path] test =================================================================================== 10:48:15 (1693306095) [ 3972.814743] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3972.829722] Lustre: lustre-OST0000-osc-ffff8f75db91a000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3972.837950] Lustre: lustre-OST0000: Client 7e5da208-bba9-428b-b525-7ed27e0ff422 (at 0@lo) reconnecting [ 3972.838183] LustreError: 167-0: lustre-OST0000-osc-ffff8f75db91a000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3973.243055] Lustre: DEBUG MARKER: tmp.136UuqwYDI: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8f75db91a000.ost_server_uuid 40 [ 3973.361278] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8f75db91a000.ost_server_uuid in FULL state after 0 sec [ 3973.798923] Lustre: DEBUG MARKER: == sanity test 104b: runas -u 500 -g 500 lfs check servers test ============================================================================== 10:48:16 (1693306096) [ 3974.258118] Lustre: DEBUG MARKER: == sanity test 104c: Verify df vs lfs_df stays same after recordsize change ========================================================== 10:48:17 (1693306097) [ 3977.906422] Lustre: DEBUG MARKER: == sanity test 104d: runas -u 500 -g 500 lctl dl test ==== 10:48:21 (1693306101) [ 3978.403258] Lustre: DEBUG MARKER: == sanity test 105a: flock when mounted without -o flock test ================================================================== 10:48:21 (1693306101) [ 3978.845831] Lustre: DEBUG MARKER: == sanity test 105b: fcntl when mounted without -o flock test ================================================================== 10:48:22 (1693306102) [ 3979.322904] Lustre: DEBUG MARKER: == sanity test 105c: lockf when mounted without -o flock test ========================================================== 10:48:22 (1693306102) [ 3979.763648] Lustre: DEBUG MARKER: == sanity test 105d: flock race (should not freeze) ================================================================== 10:48:22 (1693306102) [ 3979.866176] LustreError: 151316:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 315 sleeping for 10000ms [ 3989.890088] LustreError: 151316:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 315 awake [ 3990.394172] Lustre: DEBUG MARKER: == sanity test 105e: Two conflicting flocks from same process ========================================================== 10:48:33 (1693306113) [ 3990.891749] Lustre: DEBUG MARKER: == sanity test 106: attempt exec of dir followed by chown of that dir ========================================================== 10:48:34 (1693306114) [ 3991.408293] Lustre: DEBUG MARKER: == sanity test 107: Coredump on SIG ====================== 10:48:34 (1693306114) [ 3993.084045] Lustre: DEBUG MARKER: == sanity test 110: filename length checking ============= 10:48:36 (1693306116) [ 3993.934927] Lustre: DEBUG MARKER: == sanity test 116a: stripe QOS: free space balance ============================================================================= 10:48:37 (1693306117) [ 4073.490854] Lustre: DEBUG MARKER: == sanity test 116b: QoS shouldn't LBUG if not enough OSTs found on the 2nd pass ========================================================== 10:49:56 (1693306196) [ 4073.653222] Lustre: *** cfs_fail_loc=147, val=0*** [ 4074.138201] Lustre: DEBUG MARKER: == sanity test 117: verify osd extend ==================== 10:49:57 (1693306197) [ 4074.616152] Lustre: DEBUG MARKER: == sanity test 118a: verify O_SYNC works ================= 10:49:57 (1693306197) [ 4075.125994] Lustre: DEBUG MARKER: == sanity test 118b: Reclaim dirty pages on fatal error ==================================================================== 10:49:58 (1693306198) [ 4075.375942] Lustre: *** cfs_fail_loc=217, val=0*** [ 4075.376125] Lustre: Skipped 11 previous similar messages [ 4075.929657] Lustre: DEBUG MARKER: SKIP: sanity test_118c skipping ALWAYS excluded test 118c [ 4076.064290] Lustre: DEBUG MARKER: SKIP: sanity test_118d skipping ALWAYS excluded test 118d [ 4076.211059] Lustre: DEBUG MARKER: == sanity test 118f: Simulate unrecoverable OSC side error ==================================================================== 10:49:59 (1693306199) [ 4076.266532] Lustre: *** cfs_fail_loc=40a, val=0*** [ 4076.266673] Lustre: Skipped 423 previous similar messages [ 4076.266735] LustreError: 154471:0:(osc_request.c:2766:osc_build_rpc()) prep_req failed: -22 [ 4076.266824] LustreError: 154471:0:(osc_cache.c:2175:osc_check_rpcs()) Write request failed with -22 [ 4076.829172] Lustre: DEBUG MARKER: == sanity test 118g: Don't stay in wait if we got local -ENOMEM ==================================================================== 10:49:59 (1693306199) [ 4076.890802] LustreError: 154600:0:(osc_request.c:2766:osc_build_rpc()) prep_req failed: -12 [ 4076.890957] LustreError: 154600:0:(osc_cache.c:2175:osc_check_rpcs()) Write request failed with -12 [ 4077.485853] Lustre: DEBUG MARKER: == sanity test 118h: Verify timeout in handling recoverables errors ==================================================================== 10:50:00 (1693306200) [ 4077.750012] Lustre: *** cfs_fail_loc=20e, val=0*** [ 4077.750290] LustreError: 11-0: lustre-OST0001-osc-ffff8f75db91a000: operation ost_write to node 0@lo failed: rc = -5 [ 4077.750426] LustreError: 112550:0:(osc_request.c:2409:osc_brw_redo_request()) @@@ redo for recoverable error -5 req@000000009bde8fc6 x1775558574180288/t0(0) o4->lustre-OST0001-osc-ffff8f75db91a000@0@lo:6/4 lens 4584/224 e 0 to 0 dl 1693306207 ref 2 fl Interpret:RMQU/200/0 rc -5/-5 uid:0 gid:0 job:'multiop.0' [ 4077.750685] LustreError: 112550:0:(osc_request.c:2409:osc_brw_redo_request()) Skipped 191 previous similar messages [ 4078.790308] Lustre: *** cfs_fail_loc=20e, val=0*** [ 4078.790626] LustreError: 11-0: lustre-OST0001-osc-ffff8f75db91a000: operation ost_write to node 0@lo failed: rc = -5 [ 4080.800320] Lustre: *** cfs_fail_loc=20e, val=0*** [ 4080.800647] LustreError: 11-0: lustre-OST0001-osc-ffff8f75db91a000: operation ost_write to node 0@lo failed: rc = -5 [ 4083.910209] Lustre: *** cfs_fail_loc=20e, val=0*** [ 4083.910411] LustreError: 11-0: lustre-OST0001-osc-ffff8f75db91a000: operation ost_write to node 0@lo failed: rc = -5 [ 4087.911980] LustreError: 112550:0:(osc_request.c:2539:brw_interpret()) lustre-OST0001-osc-ffff8f75db91a000: too many resent retries for object: 11811163088:8708, rc = -5. [ 4087.920222] LustreError: 112550:0:(osc_request.c:2539:brw_interpret()) Skipped 7 previous similar messages [ 4087.920702] Lustre: 112550:0:(llite_lib.c:3981:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.121.234@tcp:/lustre/fid: [0x200001b73:0xe6b:0x0]// may get corrupted (rc -5) [ 4088.565215] Lustre: DEBUG MARKER: == sanity test 118i: Fix error before timeout in recoverable error ==================================================================== 10:50:11 (1693306211) [ 4088.801405] Lustre: *** cfs_fail_loc=20e, val=0*** [ 4088.801522] Lustre: Skipped 1 previous similar message [ 4088.801630] LustreError: 11-0: lustre-OST0001-osc-ffff8f75db91a000: operation ost_write to node 0@lo failed: rc = -5 [ 4088.801717] LustreError: Skipped 1 previous similar message [ 4096.182725] Lustre: DEBUG MARKER: == sanity test 118j: Simulate unrecoverable OST side error ==================================================================== 10:50:19 (1693306219) [ 4097.056154] Lustre: DEBUG MARKER: == sanity test 118k: bio alloc -ENOMEM and IO TERM handling =================================================================== 10:50:20 (1693306220) [ 4097.340541] Lustre: *** cfs_fail_loc=20e, val=0*** [ 4097.340685] Lustre: Skipped 3 previous similar messages [ 4097.340804] LustreError: 11-0: lustre-OST0000-osc-ffff8f75db91a000: operation ost_write to node 0@lo failed: rc = -5 [ 4097.340885] LustreError: Skipped 3 previous similar messages [ 4097.340953] LustreError: 112547:0:(osc_request.c:2409:osc_brw_redo_request()) @@@ redo for recoverable error -5 req@0000000008a2c629 x1775558574195328/t0(0) o4->lustre-OST0000-osc-ffff8f75db91a000@0@lo:6/4 lens 488/224 e 0 to 0 dl 1693306227 ref 2 fl Interpret:ReMQU/200/0 rc -5/-5 uid:0 gid:0 job:'dd.0' [ 4097.341107] LustreError: 112547:0:(osc_request.c:2409:osc_brw_redo_request()) Skipped 6 previous similar messages [ 4104.455102] Lustre: DEBUG MARKER: == sanity test 118l: fsync dir =========================== 10:50:27 (1693306227) [ 4105.033320] Lustre: DEBUG MARKER: == sanity test 118m: fdatasync dir ======================= 10:50:28 (1693306228) [ 4105.607608] Lustre: DEBUG MARKER: == sanity test 118n: statfs() sends OST_STATFS requests in parallel ========================================================== 10:50:28 (1693306228) [ 4108.342167] Lustre: DEBUG MARKER: == sanity test 119a: Short directIO read must return actual read amount ========================================================== 10:50:31 (1693306231) [ 4108.838027] Lustre: DEBUG MARKER: == sanity test 119b: Sparse directIO read must return actual read amount ========================================================== 10:50:31 (1693306231) [ 4109.333389] Lustre: DEBUG MARKER: == sanity test 119c: Testing for direct read hitting hole ========================================================== 10:50:32 (1693306232) [ 4109.848299] Lustre: DEBUG MARKER: == sanity test 120a: Early Lock Cancel: mkdir test ======= 10:50:32 (1693306232) [ 4110.640896] Lustre: DEBUG MARKER: == sanity test 120b: Early Lock Cancel: create test ====== 10:50:33 (1693306233) [ 4111.404806] Lustre: DEBUG MARKER: == sanity test 120c: Early Lock Cancel: link test ======== 10:50:34 (1693306234) [ 4112.208425] Lustre: DEBUG MARKER: == sanity test 120d: Early Lock Cancel: setattr test ===== 10:50:35 (1693306235) [ 4112.931891] Lustre: DEBUG MARKER: == sanity test 120e: Early Lock Cancel: unlink test ====== 10:50:36 (1693306236) [ 4120.850614] Lustre: DEBUG MARKER: == sanity test 120f: Early Lock Cancel: rename test ====== 10:50:44 (1693306244) [ 4128.788825] Lustre: DEBUG MARKER: == sanity test 120g: Early Lock Cancel: performance test ========================================================== 10:50:51 (1693306251) [ 4168.847839] Lustre: DEBUG MARKER: == sanity test 121: read cancel race ===================== 10:51:32 (1693306292) [ 4169.023989] Lustre: *** cfs_fail_loc=310, val=0*** [ 4169.589182] Lustre: DEBUG MARKER: == sanity test 123aa: verify statahead work ============== 10:51:32 (1693306292) [ 4170.543327] Lustre: DEBUG MARKER: ls -l 100 files without statahead: 0 sec [ 4170.755765] Lustre: DEBUG MARKER: ls -l 100 files with statahead: 0 sec [ 4173.388172] Lustre: DEBUG MARKER: ls -l 1000 files without statahead: 1 sec [ 4173.983923] Lustre: DEBUG MARKER: ls -l 1000 files with statahead: 1 sec [ 4196.112258] Lustre: DEBUG MARKER: ls -l 10000 files without statahead: 8 sec [ 4204.113537] Lustre: DEBUG MARKER: ls -l 10000 files with statahead: 7 sec [ 4204.209845] Lustre: DEBUG MARKER: statahead was stopped, maybe too many locks held! [ 4204.289984] Lustre: DEBUG MARKER: ls -l done [ 4219.084413] Lustre: DEBUG MARKER: rm -r /mnt/lustre/d123aa.sanity/: 15 seconds [ 4219.186666] Lustre: DEBUG MARKER: rm done [ 4219.663974] Lustre: DEBUG MARKER: == sanity test 123ab: verify statahead work by using statx ========================================================== 10:52:22 (1693306342) [ 4221.409762] Lustre: DEBUG MARKER: statx -l 100 files without statahead: 0 sec [ 4221.657613] Lustre: DEBUG MARKER: statx -l 100 files with statahead: 0 sec [ 4224.082754] Lustre: DEBUG MARKER: statx -l 1000 files without statahead: 1 sec [ 4224.732420] Lustre: DEBUG MARKER: statx -l 1000 files with statahead: 0 sec [ 4248.447069] Lustre: DEBUG MARKER: statx -l 10000 files without statahead: 9 sec [ 4255.999714] Lustre: DEBUG MARKER: statx -l 10000 files with statahead: 7 sec [ 4256.105662] Lustre: DEBUG MARKER: statahead was stopped, maybe too many locks held! [ 4256.189605] Lustre: DEBUG MARKER: statx -l done [ 4271.941308] Lustre: DEBUG MARKER: rm -r /mnt/lustre/d123ab.sanity/: 15 seconds [ 4272.055480] Lustre: DEBUG MARKER: rm done [ 4272.440570] Lustre: DEBUG MARKER: SKIP: sanity test_123ac skipping excluded test 123ac [ 4272.541409] Lustre: DEBUG MARKER: SKIP: sanity test_123ad skipping ALWAYS excluded test 123ad [ 4272.685141] Lustre: DEBUG MARKER: == sanity test 123b: not panic with network error in statahead enqueue (bug 15027) ========================================================== 10:53:15 (1693306395) [ 4275.047035] Lustre: DEBUG MARKER: ls done [ 4276.903977] Lustre: DEBUG MARKER: == sanity test 123c: Can not initialize inode warning on DNE statahead ========================================================== 10:53:20 (1693306400) [ 4277.769193] systemd[1]: mnt-lustre.mount: Succeeded. [ 4277.851617] Lustre: Unmounted lustre-client [ 4277.962107] Lustre: Mounted lustre-client [ 4277.962256] Lustre: Skipped 1 previous similar message [ 4278.562266] Lustre: DEBUG MARKER: == sanity test 123d: Statahead on striped directories works correctly ========================================================== 10:53:21 (1693306401) [ 4280.346872] systemd[1]: mnt-lustre.mount: Succeeded. [ 4280.420234] Lustre: Unmounted lustre-client [ 4280.519795] Lustre: Mounted lustre-client [ 4282.285546] Lustre: DEBUG MARKER: == sanity test 123e: statahead with large wide striping == 10:53:25 (1693306405) [ 4296.597049] Lustre: 113724:0:(mdt_handler.c:4389:mdt_unpack_req_pack_rep()) lustre-MDT0000: cannot pack response: rc = -75 [ 4296.597849] LustreError: 112547:0:(pack_generic.c:599:lustre_unpack_msg_v2()) buffer 1 length 1024152 is not valid [ 4296.598046] LustreError: 112547:0:(sec.c:1067:do_cli_unwrap_reply()) failed unpack reply: x1775558599452224 [ 4296.598142] LustreError: 112547:0:(client.c:1467:after_reply()) @@@ unwrap reply failed: rc = -71 req@00000000f5c24232 x1775558599452224/t0(0) o63->lustre-MDT0000-mdc-ffff8f75c968c000@0@lo:24/4 lens 1032/811672 e 0 to 0 dl 1693306426 ref 1 fl Rpc:RQU/200/ffffffff rc 0/-1 uid:0 gid:0 job:'ll_sa_162518.0' [ 4310.339352] Lustre: DEBUG MARKER: == sanity test 123f: Retry mechanism with large wide striping files ========================================================== 10:53:53 (1693306433) [ 4347.969630] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000bd0 to 0x2c00013a0 [ 4348.626860] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000bd0 to 0x2800013a0 [ 4413.425273] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x2800013a0 to 0x2800013a1 [ 4416.201176] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c00013a0 to 0x2c00013a1 [ 4420.922637] LustreError: 112548:0:(pack_generic.c:599:lustre_unpack_msg_v2()) buffer 1 length 1047520 is not valid [ 4420.922871] LustreError: 112548:0:(sec.c:1067:do_cli_unwrap_reply()) failed unpack reply: x1775558621388160 [ 4420.923012] LustreError: 112548:0:(client.c:1467:after_reply()) @@@ unwrap reply failed: rc = -71 req@000000002dd8111a x1775558621388160/t0(0) o63->lustre-MDT0000-mdc-ffff8f75c968c000@0@lo:24/4 lens 288/1047968 e 0 to 0 dl 1693306588 ref 1 fl Rpc:RQU/202/ffffffff rc 0/-1 uid:0 gid:0 job:'ll_sa_165068.0' [ 4577.324363] Lustre: DEBUG MARKER: == sanity test 124a: lru resize ================================================================================================= 10:58:20 (1693306700) [ 4577.533740] Lustre: DEBUG MARKER: create 2000 files at /mnt/lustre/d124a.sanity [ 4584.738675] Lustre: DEBUG MARKER: NSDIR=ldlm.namespaces.lustre-MDT0000-mdc-ffff8f75c968c000 [ 4584.933884] Lustre: DEBUG MARKER: NS=ldlm.namespaces.lustre-MDT0000-mdc-ffff8f75c968c000 [ 4585.192443] Lustre: DEBUG MARKER: LRU=745 [ 4585.422362] Lustre: DEBUG MARKER: LIMIT=154687 [ 4585.706144] Lustre: DEBUG MARKER: LVF=24916000 [ 4585.978023] Lustre: DEBUG MARKER: OLD_LVF=100 [ 4586.142241] Lustre: DEBUG MARKER: Sleep 50 sec [ 4636.565359] Lustre: DEBUG MARKER: Dropped 745 locks in 50s [ 4636.739302] Lustre: DEBUG MARKER: unlink 2000 files at /mnt/lustre/d124a.sanity [ 4642.415930] Lustre: DEBUG MARKER: == sanity test 124b: lru resize (performance test) ================================================================================= 10:59:25 (1693306765) [ 4656.098265] Lustre: DEBUG MARKER: doing ls -la /mnt/lustre/d124b.sanity/disable_lru_resize 3 times [ 4667.964226] Lustre: DEBUG MARKER: ls -la time: 12 seconds [ 4668.140786] Lustre: DEBUG MARKER: lru_size = 200 [ 4687.886056] Lustre: DEBUG MARKER: doing ls -la /mnt/lustre/d124b.sanity/enable_lru_resize 3 times [ 4692.614700] Lustre: DEBUG MARKER: ls -la time: 4 seconds [ 4692.716043] Lustre: DEBUG MARKER: lru_size = 2053 [ 4692.828489] Lustre: DEBUG MARKER: ls -la is 66% faster with lru resize enabled [ 4700.339450] Lustre: DEBUG MARKER: == sanity test 124c: LRUR cancel very aged locks ========= 11:00:22 (1693306822) [ 4721.549503] Lustre: DEBUG MARKER: == sanity test 124d: cancel very aged locks if lru-resize diasbaled ========================================================== 11:00:44 (1693306844) [ 4743.015631] Lustre: DEBUG MARKER: == sanity test 125: don't return EPROTO when a dir has a non-default striping and ACLs ========================================================== 11:01:06 (1693306866) [ 4744.071315] Lustre: DEBUG MARKER: == sanity test 126: check that the fsgid provided by the client is taken into account ========================================================== 11:01:07 (1693306867) [ 4744.695497] Lustre: DEBUG MARKER: == sanity test 127a: verify the client stats are sane ==== 11:01:07 (1693306867) [ 4745.452546] Lustre: DEBUG MARKER: == sanity test 127b: verify the llite client stats are sane ========================================================== 11:01:08 (1693306868) [ 4745.994495] Lustre: DEBUG MARKER: == sanity test 127c: test llite extent stats with regular [ 4813.356655] Lustre: DEBUG MARKER: == sanity test 128: interactive lfs for 2 consecutive find's ========================================================== 11:02:16 (1693306936) [ 4813.938614] Lustre: DEBUG MARKER: SKIP: sanity test_129 skipping excluded test 129 [ 4814.131370] Lustre: DEBUG MARKER: == sanity test 130a: FIEMAP (1-stripe file) ============== 11:02:17 (1693306937) [ 4814.369266] Lustre: DEBUG MARKER: SKIP: sanity test_130a LU-1941: FIEMAP unimplemented on ZFS [ 4814.555190] Lustre: DEBUG MARKER: SKIP: sanity test_130b skipping ALWAYS excluded test 130b [ 4814.682145] Lustre: DEBUG MARKER: SKIP: sanity test_130c skipping ALWAYS excluded test 130c [ 4814.813414] Lustre: DEBUG MARKER: SKIP: sanity test_130d skipping ALWAYS excluded test 130d [ 4814.948332] Lustre: DEBUG MARKER: SKIP: sanity test_130e skipping ALWAYS excluded test 130e [ 4815.092603] Lustre: DEBUG MARKER: SKIP: sanity test_130f skipping ALWAYS excluded test 130f [ 4815.249666] Lustre: DEBUG MARKER: SKIP: sanity test_130g skipping ALWAYS excluded test 130g [ 4815.473044] Lustre: DEBUG MARKER: == sanity test 131a: test iov's crossing stripe boundary for writev/readv ========================================================== 11:02:18 (1693306938) [ 4816.142727] Lustre: DEBUG MARKER: == sanity test 131b: test append writev ================== 11:02:19 (1693306939) [ 4816.720614] Lustre: DEBUG MARKER: == sanity test 131c: test read/write on file w/o objects ========================================================== 11:02:19 (1693306939) [ 4817.173220] Lustre: DEBUG MARKER: == sanity test 131d: test short read ===================== 11:02:20 (1693306940) [ 4817.700663] Lustre: DEBUG MARKER: == sanity test 131e: test read hitting hole ============== 11:02:20 (1693306940) [ 4818.152888] Lustre: DEBUG MARKER: == sanity test 133a: Verifying MDT stats ================================================================================================== 11:02:21 (1693306941) [ 4819.204240] Lustre: DEBUG MARKER: == sanity test 133b: Verifying extra MDT stats ============================================================================================ 11:02:22 (1693306942) [ 4820.130182] Lustre: DEBUG MARKER: == sanity test 133c: Verifying OST stats ================================================================================================== 11:02:23 (1693306943) [ 4851.292953] Lustre: DEBUG MARKER: == sanity test 133d: Verifying rename_stats ================================================================================================== 11:02:54 (1693306974) [ 4853.809678] Lustre: DEBUG MARKER: == sanity test 133e: Verifying OST read_bytes write_bytes nid stats =========================================================================== 11:02:56 (1693306976) [ 4854.652020] Lustre: DEBUG MARKER: == sanity test 133f: Check reads/writes of client lustre proc files with bad area io ========================================================== 11:02:57 (1693306977) [ 4892.142914] Lustre: 178308:0:(libcfs_string.c:112:cfs_str2mask()) unknown mask ''. [ 4892.142914] mask usage: [+|-] ... [ 4892.150641] badarea_io: page allocation failure: order:10, mode:0x6040c0(GFP_KERNEL|__GFP_COMP), nodemask=(null) [ 4892.150860] CPU: 0 PID: 178308 Comm: badarea_io Tainted: G W O --------- - - 4.18.0 #2 [ 4892.150987] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 4892.151079] Call Trace: [ 4892.151124] dump_stack+0x5c/0x80 [ 4892.151187] warn_alloc.cold.32+0x69/0xe9 [ 4892.151256] ? _cond_resched+0x10/0x20 [ 4892.151316] __alloc_pages_nodemask+0xadf/0xc50 [ 4892.151394] ? __lock_acquire.isra.16+0x2f3/0x5b0 [ 4892.151471] cache_alloc_refill+0x32e/0x8f0 [ 4892.151540] ? proc_dobitmasks+0xc4/0x140 [libcfs] [ 4892.151619] __kmalloc_track_caller+0x18e/0x1b0 [ 4892.151705] memdup_user_nul+0x24/0x70 [ 4892.151767] proc_dobitmasks+0xc4/0x140 [libcfs] [ 4892.151848] lnet_debugfs_write+0x38/0x60 [libcfs] [ 4892.151928] full_proxy_write+0x4b/0x70 [ 4892.151988] vfs_write+0xb9/0x1c0 [ 4892.152047] ksys_write+0x3d/0xa0 [ 4892.152106] do_syscall_64+0x43/0x120 [ 4892.152165] entry_SYSCALL_64_after_hwframe+0x65/0xca [ 4892.152252] RIP: 0033:0x7f5c455b8648 [ 4892.152310] Code: 89 02 48 c7 c0 ff ff ff ff eb b3 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 55 6f 2d 00 8b 00 85 c0 75 17 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 58 c3 0f 1f 80 00 00 00 00 41 54 49 89 d4 55 [ 4892.152555] RSP: 002b:00007ffe64631f08 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [ 4892.152665] RAX: ffffffffffffffda RBX: 00007ffe64632018 RCX: 00007f5c455b8648 [ 4892.152772] RDX: 0000000000200000 RSI: 00007ffe64631f1c RDI: 0000000000000003 [ 4892.152879] RBP: 00000000004007f0 R08: 00007f5c4588bd20 R09: 00007f5c4588bd20 [ 4892.152986] R10: 0000000000000005 R11: 0000000000000246 R12: 0000000000400700 [ 4892.153093] R13: 00007ffe64632010 R14: 0000000000000000 R15: 0000000000000000 [ 4892.153257] Mem-Info: [ 4892.153306] active_anon:719771 inactive_anon:225424 isolated_anon:0 [ 4892.153306] active_file:25053 inactive_file:5334 isolated_file:0 [ 4892.153306] unevictable:0 dirty:0 writeback:0 [ 4892.153306] slab_reclaimable:34288 slab_unreclaimable:218192 [ 4892.153306] mapped:4234 shmem:938501 pagetables:447 bounce:0 [ 4892.153306] free:1874167 free_pcp:44 free_cma:0 [ 4892.153732] Node 0 active_anon:2879084kB inactive_anon:901696kB active_file:100212kB inactive_file:21336kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:16936kB dirty:0kB writeback:0kB shmem:3754004kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB all_unreclaimable? no [ 4892.154084] DMA free:13860kB min:84kB low:104kB high:124kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [ 4892.154421] lowmem_reserve[]: 0 3503 12360 12360 [ 4892.154499] DMA32 free:3358676kB min:19132kB low:23912kB high:28692kB active_anon:97980kB inactive_anon:75916kB active_file:152kB inactive_file:0kB unevictable:0kB writepending:0kB present:3653600kB managed:3588064kB mlocked:0kB kernel_stack:1040kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [ 4892.154847] lowmem_reserve[]: 0 0 8856 8856 [ 4892.154906] Normal free:4124132kB min:64744kB low:76832kB high:88920kB active_anon:2781104kB inactive_anon:825648kB active_file:100060kB inactive_file:21336kB unevictable:0kB writepending:0kB present:9355264kB managed:9068744kB mlocked:0kB kernel_stack:7552kB pagetables:1788kB bounce:0kB free_pcp:176kB local_pcp:24kB free_cma:0kB [ 4892.155276] lowmem_reserve[]: 0 0 0 0 [ 4892.155334] DMA: 1*4kB (U) 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 2*2048kB (UM) 2*4096kB (M) = 13860kB [ 4892.155522] DMA32: 10905*4kB (UE) 10696*8kB (UE) 10364*16kB (UE) 9477*32kB (UE) 8199*64kB (UE) 6179*128kB (UE) 3380*256kB (UE) 903*512kB (UE) 109*1024kB (U) 3*2048kB (UM) 0*4096kB = 3359300kB [ 4892.155762] Normal: 50332*4kB (UME) 44065*8kB (UME) 38205*16kB (UME) 28532*32kB (UME) 15856*64kB (UME) 5419*128kB (UME) 1059*256kB (UM) 122*512kB (UM) 4*1024kB (UM) 0*2048kB 0*4096kB = 4124232kB [ 4892.156003] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [ 4892.156129] 968888 total pagecache pages [ 4892.156186] 0 pages in swap cache [ 4892.156247] Swap cache stats: add 0, delete 0, find 0/0 [ 4892.156320] Free swap = 0kB [ 4892.156376] Total swap = 0kB [ 4892.156432] 3256214 pages RAM [ 4892.156487] 0 pages HighMem/MovableOnly [ 4892.156543] 88035 pages reserved [ 4892.156653] SLAB: Unable to allocate memory on node 0, gfp=0x6040c0(GFP_KERNEL|__GFP_COMP) [ 4892.156767] cache: kmalloc-4M, object size: 4194304, order: 10 [ 4892.156864] node 0: slabs: 0/0, objs: 0/0 [ 4892.283319] badarea_io: page allocation failure: order:10, mode:0x6040c0(GFP_KERNEL|__GFP_COMP), nodemask=(null) [ 4892.283450] CPU: 0 PID: 178316 Comm: badarea_io Tainted: G W O --------- - - 4.18.0 #2 [ 4892.283622] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 4892.283716] Call Trace: [ 4892.283762] dump_stack+0x5c/0x80 [ 4892.283827] warn_alloc.cold.32+0x69/0xe9 [ 4892.283890] ? _cond_resched+0x10/0x20 [ 4892.283952] __alloc_pages_nodemask+0xadf/0xc50 [ 4892.284033] ? __lock_acquire.isra.16+0x2f3/0x5b0 [ 4892.284114] cache_alloc_refill+0x32e/0x8f0 [ 4892.284184] ? cfs_trace_dump_debug_buffer_usrstr+0xa/0x50 [libcfs] [ 4892.284293] __kmalloc_track_caller+0x18e/0x1b0 [ 4892.284374] memdup_user_nul+0x24/0x70 [ 4892.284437] cfs_trace_dump_debug_buffer_usrstr+0xa/0x50 [libcfs] [ 4892.284539] lnet_debugfs_write+0x38/0x60 [libcfs] [ 4892.284622] full_proxy_write+0x4b/0x70 [ 4892.284685] vfs_write+0xb9/0x1c0 [ 4892.284746] ksys_write+0x3d/0xa0 [ 4892.284807] do_syscall_64+0x43/0x120 [ 4892.284868] entry_SYSCALL_64_after_hwframe+0x65/0xca [ 4892.284947] RIP: 0033:0x7fb5d18fe648 [ 4892.285008] Code: 89 02 48 c7 c0 ff ff ff ff eb b3 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 55 6f 2d 00 8b 00 85 c0 75 17 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 58 c3 0f 1f 80 00 00 00 00 41 54 49 89 d4 55 [ 4892.285267] RSP: 002b:00007ffc9ab28f38 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [ 4892.285380] RAX: ffffffffffffffda RBX: 00007ffc9ab29048 RCX: 00007fb5d18fe648 [ 4892.285492] RDX: 0000000000200000 RSI: 00007ffc9ab28f4c RDI: 0000000000000003 [ 4892.285603] RBP: 00000000004007f0 R08: 00007fb5d1bd1d20 R09: 00007fb5d1bd1d20 [ 4892.285715] R10: 0000000000000005 R11: 0000000000000246 R12: 0000000000400700 [ 4892.285826] R13: 00007ffc9ab29040 R14: 0000000000000000 R15: 0000000000000000 [ 4892.285940] SLAB: Unable to allocate memory on node 0, gfp=0x6040c0(GFP_KERNEL|__GFP_COMP) [ 4892.286054] cache: kmalloc-4M, object size: 4194304, order: 10 [ 4892.286149] node 0: slabs: 0/0, objs: 0/0 [ 4892.320044] badarea_io: page allocation failure: order:10, mode:0x6040c0(GFP_KERNEL|__GFP_COMP), nodemask=(null) [ 4892.320289] CPU: 0 PID: 178318 Comm: badarea_io Tainted: G W O --------- - - 4.18.0 #2 [ 4892.320418] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 4892.320510] Call Trace: [ 4892.320556] dump_stack+0x5c/0x80 [ 4892.320620] warn_alloc.cold.32+0x69/0xe9 [ 4892.320682] ? _cond_resched+0x10/0x20 [ 4892.320743] __alloc_pages_nodemask+0xadf/0xc50 [ 4892.320823] ? __lock_acquire.isra.16+0x2f3/0x5b0 [ 4892.320903] cache_alloc_refill+0x32e/0x8f0 [ 4892.320970] ? debugfs_doint+0x41/0x140 [libcfs] [ 4892.321051] __kmalloc_track_caller+0x18e/0x1b0 [ 4892.321129] memdup_user_nul+0x24/0x70 [ 4892.321194] debugfs_doint+0x41/0x140 [libcfs] [ 4892.321284] lnet_debugfs_write+0x38/0x60 [libcfs] [ 4892.321366] full_proxy_write+0x4b/0x70 [ 4892.321428] vfs_write+0xb9/0x1c0 [ 4892.321488] ksys_write+0x3d/0xa0 [ 4892.321550] do_syscall_64+0x43/0x120 [ 4892.321611] entry_SYSCALL_64_after_hwframe+0x65/0xca [ 4892.321690] RIP: 0033:0x7f0b81fab648 [ 4892.321750] Code: 89 02 48 c7 c0 ff ff ff ff eb b3 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 55 6f 2d 00 8b 00 85 c0 75 17 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 58 c3 0f 1f 80 00 00 00 00 41 54 49 89 d4 55 [ 4892.322000] RSP: 002b:00007ffe373cb868 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [ 4892.322110] RAX: ffffffffffffffda RBX: 00007ffe373cb978 RCX: 00007f0b81fab648 [ 4892.322225] RDX: 0000000000200000 RSI: 00007ffe373cb87c RDI: 0000000000000003 [ 4892.322335] RBP: 00000000004007f0 R08: 00007f0b8227ed20 R09: 00007f0b8227ed20 [ 4892.322444] R10: 0000000000000005 R11: 0000000000000246 R12: 0000000000400700 [ 4892.322553] R13: 00007ffe373cb970 R14: 0000000000000000 R15: 0000000000000000 [ 4892.322666] SLAB: Unable to allocate memory on node 0, gfp=0x6040c0(GFP_KERNEL|__GFP_COMP) [ 4892.322777] cache: kmalloc-4M, object size: 4194304, order: 10 [ 4892.322883] node 0: slabs: 0/0, objs: 0/0 [ 4892.343534] badarea_io: page allocation failure: order:10, mode:0x6040c0(GFP_KERNEL|__GFP_COMP), nodemask=(null) [ 4892.343678] CPU: 1 PID: 178320 Comm: badarea_io Tainted: G W O --------- - - 4.18.0 #2 [ 4892.343735] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 4892.343777] Call Trace: [ 4892.343804] dump_stack+0x5c/0x80 [ 4892.343838] warn_alloc.cold.32+0x69/0xe9 [ 4892.343870] ? _cond_resched+0x10/0x20 [ 4892.343901] __alloc_pages_nodemask+0xadf/0xc50 [ 4892.343939] ? __lock_acquire.isra.16+0x2f3/0x5b0 [ 4892.343976] cache_alloc_refill+0x32e/0x8f0 [ 4892.344011] ? proc_fail_loc+0x3b/0x120 [libcfs] [ 4892.344050] __kmalloc_track_caller+0x18e/0x1b0 [ 4892.344088] memdup_user_nul+0x24/0x70 [ 4892.344120] proc_fail_loc+0x3b/0x120 [libcfs] [ 4892.344159] lnet_debugfs_write+0x38/0x60 [libcfs] [ 4892.344203] full_proxy_write+0x4b/0x70 [ 4892.344235] vfs_write+0xb9/0x1c0 [ 4892.344264] ksys_write+0x3d/0xa0 [ 4892.344294] do_syscall_64+0x43/0x120 [ 4892.344324] entry_SYSCALL_64_after_hwframe+0x65/0xca [ 4892.344360] RIP: 0033:0x7fcbdce05648 [ 4892.344390] Code: 89 02 48 c7 c0 ff ff ff ff eb b3 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 55 6f 2d 00 8b 00 85 c0 75 17 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 58 c3 0f 1f 80 00 00 00 00 41 54 49 89 d4 55 [ 4892.344487] RSP: 002b:00007fff33949958 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [ 4892.344535] RAX: ffffffffffffffda RBX: 00007fff33949a68 RCX: 00007fcbdce05648 [ 4892.344581] RDX: 0000000000200000 RSI: 00007fff3394996c RDI: 0000000000000003 [ 4892.344628] RBP: 00000000004007f0 R08: 00007fcbdd0d8d20 R09: 00007fcbdd0d8d20 [ 4892.344675] R10: 0000000000000005 R11: 0000000000000246 R12: 0000000000400700 [ 4892.344721] R13: 00007fff33949a60 R14: 0000000000000000 R15: 0000000000000000 [ 4892.344773] SLAB: Unable to allocate memory on node 0, gfp=0x6040c0(GFP_KERNEL|__GFP_COMP) [ 4892.344823] cache: kmalloc-4M, object size: 4194304, order: 10 [ 4892.344867] node 0: slabs: 0/0, objs: 0/0 [ 4892.374545] badarea_io: page allocation failure: order:10, mode:0x6040c0(GFP_KERNEL|__GFP_COMP), nodemask=(null) [ 4892.374675] CPU: 1 PID: 178322 Comm: badarea_io Tainted: G W O --------- - - 4.18.0 #2 [ 4892.374737] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 4892.374778] Call Trace: [ 4892.374806] dump_stack+0x5c/0x80 [ 4892.374840] warn_alloc.cold.32+0x69/0xe9 [ 4892.374873] ? _cond_resched+0x10/0x20 [ 4892.374905] __alloc_pages_nodemask+0xadf/0xc50 [ 4892.374946] ? __lock_acquire.isra.16+0x2f3/0x5b0 [ 4892.374985] cache_alloc_refill+0x32e/0x8f0 [ 4892.375022] ? debugfs_doint+0x41/0x140 [libcfs] [ 4892.375060] __kmalloc_track_caller+0x18e/0x1b0 [ 4892.375098] memdup_user_nul+0x24/0x70 [ 4892.375130] debugfs_doint+0x41/0x140 [libcfs] [ 4892.375170] lnet_debugfs_write+0x38/0x60 [libcfs] [ 4892.375214] full_proxy_write+0x4b/0x70 [ 4892.375245] vfs_write+0xb9/0x1c0 [ 4892.375275] ksys_write+0x3d/0xa0 [ 4892.375305] do_syscall_64+0x43/0x120 [ 4892.375337] entry_SYSCALL_64_after_hwframe+0x65/0xca [ 4892.375373] RIP: 0033:0x7fef805d1648 [ 4892.375404] Code: 89 02 48 c7 c0 ff ff ff ff eb b3 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 55 6f 2d 00 8b 00 85 c0 75 17 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 58 c3 0f 1f 80 00 00 00 00 41 54 49 89 d4 55 [ 4892.375504] RSP: 002b:00007ffcab904d88 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [ 4892.375553] RAX: ffffffffffffffda RBX: 00007ffcab904e98 RCX: 00007fef805d1648 [ 4892.375601] RDX: 0000000000200000 RSI: 00007ffcab904d9c RDI: 0000000000000003 [ 4892.375649] RBP: 00000000004007f0 R08: 00007fef808a4d20 R09: 00007fef808a4d20 [ 4892.375697] R10: 0000000000000005 R11: 0000000000000246 R12: 0000000000400700 [ 4892.375745] R13: 00007ffcab904e90 R14: 0000000000000000 R15: 0000000000000000 [ 4892.375797] SLAB: Unable to allocate memory on node 0, gfp=0x6040c0(GFP_KERNEL|__GFP_COMP) [ 4892.375848] cache: kmalloc-4M, object size: 4194304, order: 10 [ 4892.375902] node 0: slabs: 0/0, objs: 0/0 [ 4892.720523] Lustre: 178408:0:(libcfs_string.c:112:cfs_str2mask()) unknown mask ''. [ 4892.720523] mask usage: [+|-] ... [ 4892.720677] Lustre: 178408:0:(libcfs_string.c:112:cfs_str2mask()) Skipped 1 previous similar message [ 4894.259046] systemd[1]: mnt-lustre.mount: Succeeded. [ 4894.303102] Lustre: Unmounted lustre-client [ 4894.388724] systemd[1]: mnt-lustre\x2dmds1.mount: Succeeded. [ 4894.445701] LustreError: 178499:0:(client.c:1257:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000c0344f2c x1775558673270720/t0(0) o1000->lustre-MDT0001-osp-MDT0000@0@lo:24/4 lens 304/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 uid:0 gid:0 job:'umount.0' [ 4894.445967] LustreError: 178499:0:(osp_object.c:637:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x20000000a:0x1:0x0]: rc = -5 [ 4894.446097] LustreError: 178499:0:(llog_cat.c:452:llog_cat_close()) lustre-MDT0001-osp-MDT0000: failure destroying log during cleanup: rc = -5 [ 4895.831018] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4895.831416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4896.688948] Lustre: server umount lustre-MDT0000 complete [ 4897.392813] systemd[1]: mnt-lustre\x2dmds2.mount: Succeeded. [ 4897.422299] LustreError: 166685:0:(ldlm_lockd.c:2573:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1693307020 with bad export cookie 194701810596431239 [ 4897.422640] LustreError: 166-1: MGC192.168.121.234@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4897.814721] Lustre: server umount lustre-MDT0001 complete [ 4898.513820] systemd[1]: mnt-lustre\x2dost1.mount: Succeeded. [ 4904.630087] Lustre: 178720:0:(client.c:2309:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1693307021/real 1693307021] req@00000000cb2e72d6 x1775558673272192/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1693307027 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 uid:0 gid:0 job:'umount.0' [ 4904.648524] Lustre: server umount lustre-OST0000 complete [ 4904.955818] systemd[1]: mnt-lustre\x2dost2.mount: Succeeded. [ 4911.110071] Lustre: 178802:0:(client.c:2309:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1693307028/real 1693307028] req@00000000cb2e72d6 x1775558673272576/t0(0) o39->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1693307034 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 uid:0 gid:0 job:'umount.0' [ 4911.283716] Lustre: server umount lustre-OST0001 complete [ 4914.500580] LNet: 179162:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4914.502501] LNet: Removed LNI 192.168.121.234@tcp [ 4915.452293] systemd-udevd[773]: Specified user 'tss' unknown [ 4915.454544] systemd-udevd[773]: Specified group 'tss' unknown [ 4915.546749] systemd-udevd[179451]: Using default interface naming scheme 'rhel-8.0'. [ 4915.783309] systemd[1]: usr-sbin-mount.lustre.mount: Succeeded. [ 4915.931819] systemd-udevd[773]: Specified user 'tss' unknown [ 4915.933363] systemd-udevd[773]: Specified group 'tss' unknown [ 4915.968223] systemd-udevd[179683]: Using default interface naming scheme 'rhel-8.0'. [ 4916.209331] libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2 [ 4917.157266] Lustre: Lustre: Build Version: 2.15.57 [ 4917.226989] LNet: Added LNI 192.168.121.234@tcp [8/256/0/180] [ 4917.227226] LNet: Accept secure, port 988 [ 4917.667149] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4918.917726] systemd-udevd[773]: Specified user 'tss' unknown [ 4918.930070] systemd-udevd[773]: Specified group 'tss' unknown [ 4918.992955] systemd-udevd[180626]: Using default interface naming scheme 'rhel-8.0'. [ 4920.551724] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4921.803872] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4921.853979] systemd-udevd[181195]: Process '/mnt/build/lustre/tests/../utils/lctl set_param 'osc.lustre*.checksums=0'' failed with exit code 2. [ 4921.871985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4923.003663] systemd-udevd[181576]: Process '/mnt/build/lustre/tests/../utils/lctl set_param 'osc.lustre*.checksums=0'' failed with exit code 2. [ 4923.078714] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4924.135637] systemd-udevd[181990]: Process '/mnt/build/lustre/tests/../utils/lctl set_param 'osc.lustre*.checksums=0'' failed with exit code 2. [ 4924.204725] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4925.247057] systemd-udevd[182380]: Process '/mnt/build/lustre/tests/../utils/lctl set_param 'osc.lustre*.checksums=0'' failed with exit code 2. [ 4927.767578] Lustre: lustre-OST0001: deleting orphan objects from 0x2c00013a1:2851 to 0x2c00013a1:20001 [ 4927.768002] Lustre: lustre-OST0000: deleting orphan objects from 0x2800013a1:6353 to 0x2800013a1:20001 [ 4927.786057] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:14161 to 0x2c0000400:14305 [ 4927.817460] Lustre: Mounted lustre-client [ 4928.799672] Lustre: lustre-OST0000: deleting orphan objects from 0x280000bd1:13512 to 0x280000bd1:13697 [ 4929.868693] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 4930.357556] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 4936.709150] Lustre: DEBUG MARKER: == sanity test 133g: Check reads/writes of server lustre proc files with bad area io ========================================================== 11:04:19 (1693307059) [ 4963.030419] Lustre: lustre-OST0000-osc-ffff8f74f9af5000: disconnect after 24s idle [ 4978.645564] Lustre: 184455:0:(libcfs_string.c:112:cfs_str2mask()) unknown mask ''. [ 4978.645564] mask usage: [+|-] ... [ 4979.256512] Lustre: 184554:0:(libcfs_string.c:112:cfs_str2mask()) unknown mask ''. [ 4979.256512] mask usage: [+|-] ... [ 4979.256770] Lustre: 184554:0:(libcfs_string.c:112:cfs_str2mask()) Skipped 1 previous similar message [ 5003.363422] Lustre: lustre-OST0000-osc-ffff8f74f9af5000: disconnect after 20s idle [ 5003.371290] Lustre: Skipped 1 previous similar message [ 5041.843956] Lustre: 186037:0:(libcfs_string.c:112:cfs_str2mask()) unknown mask ''. [ 5041.843956] mask usage: [+|-] ... [ 5044.955858] systemd[1]: mnt-lustre.mount: Succeeded. [ 5045.027733] Lustre: Unmounted lustre-client [ 5045.209562] systemd[1]: mnt-lustre\x2dmds1.mount: Succeeded. [ 5048.712814] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5048.723968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5053.594137] Lustre: server umount lustre-MDT0000 complete [ 5053.757435] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5054.127129] systemd[1]: mnt-lustre\x2dmds2.mount: Succeeded. [ 5054.154898] LustreError: 181089:0:(ldlm_lockd.c:2573:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1693307177 with bad export cookie 7751375994121571672 [ 5054.155673] LustreError: 166-1: MGC192.168.121.234@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5054.161932] LustreError: 181089:0:(ldlm_lockd.c:2573:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5054.487954] Lustre: server umount lustre-MDT0001 complete [ 5055.238002] systemd[1]: mnt-lustre\x2dost1.mount: Succeeded. [ 5061.350081] Lustre: 186423:0:(client.c:2309:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1693307178/real 1693307178] req@00000000de363ab3 x1775561122847744/t0(0) o39->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1693307184 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 uid:0 gid:0 job:'umount.0' [ 5061.392406] Lustre: server umount lustre-OST0000 complete [ 5061.960336] systemd[1]: mnt-lustre\x2dost2.mount: Succeeded. [ 5068.070093] Lustre: 186490:0:(client.c:2309:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1693307185/real 1693307185] req@000000006768c82a x1775561122848128/t0(0) o39->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1693307191 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 uid:0 gid:0 job:'umount.0' [ 5068.242780] Lustre: server umount lustre-OST0001 complete [ 5071.910700] LNet: 186826:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5071.932092] LNet: Removed LNI 192.168.121.234@tcp [ 5073.023928] systemd-udevd[773]: Specified user 'tss' unknown [ 5073.078879] systemd-udevd[773]: Specified group 'tss' unknown [ 5073.273079] systemd-udevd[187154]: Using default interface naming scheme 'rhel-8.0'. [ 5073.412749] systemd[1]: usr-sbin-mount.lustre.mount: Succeeded. [ 5073.877015] systemd-udevd[773]: Specified user 'tss' unknown [ 5074.037218] systemd-udevd[773]: Specified group 'tss' unknown [ 5074.042500] systemd-udevd[187441]: Using default interface naming scheme 'rhel-8.0'. [ 5074.107339] libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2 [ 5075.336936] Lustre: Lustre: Build Version: 2.15.57 [ 5075.478836] LNet: Added LNI 192.168.121.234@tcp [8/256/0/180] [ 5075.479206] LNet: Accept secure, port 988 [ 5076.517889] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5079.396519] systemd-udevd[773]: Specified user 'tss' unknown [ 5079.409838] systemd-udevd[773]: Specified group 'tss' unknown [ 5079.496654] systemd-udevd[188290]: Using default interface naming scheme 'rhel-8.0'. [ 5082.908561] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5084.387219] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5084.433334] systemd-udevd[188772]: Process '/mnt/build/lustre/tests/../utils/lctl set_param 'osc.lustre*.checksums=0'' failed with exit code 2. [ 5084.464148] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5086.577334] systemd-udevd[189119]: Process '/mnt/build/lustre/tests/../utils/lctl set_param 'osc.lustre*.checksums=0'' failed with exit code 2. [ 5086.876940] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5089.003106] systemd-udevd[189463]: Process '/mnt/build/lustre/tests/../utils/lctl set_param 'osc.lustre*.checksums=0'' failed with exit code 2. [ 5089.151588] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5090.154309] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5090.175413] Lustre: lustre-OST0000: deleting orphan objects from 0x2800013a1:6353 to 0x2800013a1:20033 [ 5091.191286] LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5091.201208] Lustre: lustre-OST0000: deleting orphan objects from 0x280000bd1:13512 to 0x280000bd1:13729 [ 5091.833717] systemd-udevd[189824]: Process '/mnt/build/lustre/tests/../utils/lctl set_param 'osc.lustre*.checksums=0'' failed with exit code 2. [ 5091.972643] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5092.765068] Lustre: lustre-OST0001: deleting orphan objects from 0x2c00013a1:2851 to 0x2c00013a1:20033 [ 5092.796476] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:14161 to 0x2c0000400:14337 [ 5092.797312] Lustre: Mounted lustre-client [ 5095.028462] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 5095.424357] Lustre: Modifying parameter general.lod.*.mdt_hash in log params [ 5101.355882] Lustre: DEBUG MARKER: == sanity test 133h: Proc files should end with newlines ========================================================== 11:07:04 (1693307224) [ 5117.910389] Lustre: lustre-OST0000-osc-ffff8f748849c000: disconnect after 24s idle [ 5148.160243] Lustre: lustre-OST0000-osc-ffff8f748849c000: disconnect after 20s idle [ 5148.169684] Lustre: Skipped 1 previous similar message [ 5153.190668] Lustre: lustre-OST0001-osc-ffff8f748849c000: disconnect after 24s idle [ 5183.446622] Lustre: lustre-OST0000-osc-ffff8f748849c000: disconnect after 20s idle [ 5188.490827] Lustre: lustre-OST0001-osc-ffff8f748849c000: disconnect after 23s idle [ 5228.813449] Lustre: lustre-OST0000-osc-ffff8f748849c000: disconnect after 21s idle [ 5259.041915] Lustre: lustre-OST0000-osc-ffff8f748849c000: disconnect after 20s idle [ 5259.042110] Lustre: Skipped 1 previous similar message [ 5304.402447] Lustre: lustre-OST0000-osc-ffff8f748849c000: disconnect after 23s idle [ 5304.402650] Lustre: Skipped 1 previous similar message [ 5331.475834] Lustre: DEBUG MARKER: SKIP: sanity test_134a skipping excluded test 134a [ 5331.629968] Lustre: DEBUG MARKER: == sanity test 134b: Server rejects lock request when reaching lock_limit_mb ========================================================== 11:10:54 (1693307454) [ 5331.981583] Lustre: *** cfs_fail_loc=328, val=0*** [ 5333.431888] Lustre: *** cfs_fail_loc=328, val=0*** [ 5333.439100] Lustre: Skipped 251 previous similar messages [ 5334.471903] Lustre: *** cfs_fail_loc=328, val=0*** [ 5336.791943] Lustre: *** cfs_fail_loc=328, val=0*** [ 5336.797709] Lustre: Skipped 124 previous similar messages [ 5341.032000] Lustre: *** cfs_fail_loc=328, val=0*** [ 5341.040229] Lustre: Skipped 63 previous similar messages [ 5349.431979] Lustre: *** cfs_fail_loc=328, val=0*** [ 5349.447828] Lustre: Skipped 34 previous similar messages [ 5352.062873] LustreError: 228963:0:(ldlm_resource.c:133:seq_watermark_write()) Failed to set lock_reclaim_threshold_mb, rc = -22. [ 5355.995676] Lustre: DEBUG MARKER: SKIP: sanity test_135 skipping SLOW test 135 [ 5356.232793] Lustre: DEBUG MARKER: SKIP: sanity test_136 skipping SLOW test 136 [ 5356.525405] Lustre: DEBUG MARKER: == sanity test 140: Check reasonable stack depth (shouldn't LBUG) ============================================================== 11:11:19 (1693307479) [ 5369.632630] Lustre: DEBUG MARKER: == sanity test 150a: truncate/append tests =============== 11:11:32 (1693307492) [ 5370.043793] systemd[1]: mnt-lustre.mount: Succeeded. [ 5370.142264] Lustre: Unmounted lustre-client [ 5370.287403] Lustre: Mounted lustre-client [ 5391.301966] Lustre: DEBUG MARKER: == sanity test 150b: Verify fallocate (prealloc) functionality ========================================================== 11:11:54 (1693307514) [ 5391.447880] Lustre: DEBUG MARKER: SKIP: sanity test_150b need >= 2.13.57 and ldiskfs for fallocate [ 5391.733633] Lustre: DEBUG MARKER: == sanity test 150bb: Verify fallocate modes both zero space ========================================================== 11:11:54 (1693307514) [ 5391.877657] Lustre: DEBUG MARKER: SKIP: sanity test_150bb need >= 2.13.57 and ldiskfs for fallocate [ 5392.119687] Lustre: DEBUG MARKER: == sanity test 150c: Verify fallocate Size and Blocks ==== 11:11:55 (1693307515) [ 5392.274494] Lustre: DEBUG MARKER: SKIP: sanity test_150c need >= 2.13.57 and ldiskfs for fallocate [ 5392.589942] Lustre: DEBUG MARKER: == sanity test 150d: Verify fallocate Size and Blocks - Non zero start ========================================================== 11:11:55 (1693307515) [ 5392.735906] Lustre: DEBUG MARKER: SKIP: sanity test_150d need >= 2.13.57 and ldiskfs for fallocate [ 5393.029103] Lustre: DEBUG MARKER: == sanity test 150e: Verify 60% of available OST space consumed by fallocate ========================================================== 11:11:56 (1693307516) [ 5393.234376] Lustre: DEBUG MARKER: SKIP: sanity test_150e need >= 2.13.57 and ldiskfs for fallocate [ 5393.588510] Lustre: DEBUG MARKER: == sanity test 150f: Verify fallocate punch functionality ========================================================== 11:11:56 (1693307516) [ 5393.806641] Lustre: DEBUG MARKER: SKIP: sanity test_150f LU-14160: punch mode is not implemented on OSD ZFS [ 5394.161866] Lustre: DEBUG MARKER: == sanity test 150g: Verify fallocate punch on large range ========================================================== 11:11:57 (1693307517) [ 5394.328596] Lustre: DEBUG MARKER: SKIP: sanity test_150g LU-14160: punch mode is not implemented on OSD ZFS [ 5394.585668] Lustre: DEBUG MARKER: == sanity test 150h: Verify extend fallocate updates the file size ========================================================== 11:11:57 (1693307517) [ 5394.759247] Lustre: DEBUG MARKER: SKIP: sanity test_150h need >= 2.13.57 and ldiskfs for fallocate [ 5395.007934] Lustre: DEBUG MARKER: == sanity test 151: test cache on oss and controls ========================================================================================= 11:11:58 (1693307518) [ 5395.598031] Lustre: DEBUG MARKER: SKIP: sanity test_151 not cache-capable obdfilter [ 5395.946315] Lustre: DEBUG MARKER: == sanity test 152: test read/write with enomem ====================================================================================== 11:11:59 (1693307519) [ 5396.901909] Lustre: DEBUG MARKER: == sanity test 153: test if fdatasync does not crash ================================================================================= 11:11:59 (1693307519) [ 5397.830567] Lustre: DEBUG MARKER: == sanity test 154A: lfs path2fid and fid2path basic checks ========================================================== 11:12:00 (1693307520) [ 5398.618586] Lustre: DEBUG MARKER: == sanity test 154B: verify the ll_decode_linkea tool ==== 11:12:01 (1693307521) [ 5399.800994] Lustre: DEBUG MARKER: == sanity test 154a: Open-by-FID ========================= 11:12:02 (1693307522) [ 5400.209858] LustreError: 188721:0:(fld_handler.c:264:fld_server_lookup()) srv-lustre-MDT0000: Cannot find sequence 0xf00000400: rc = -2 [ 5400.855599] Lustre: dir [0x200000bd2:0x21af:0x0] stripe 1 readdir failed: -2, directory is partially accessed! [ 5401.791971] Lustre: DEBUG MARKER: == sanity test 154b: Open-by-FID for remote directory ==== 11:12:04 (1693307524) [ 5402.258692] LustreError: 188716:0:(fld_handler.c:264:fld_server_lookup()) srv-lustre-MDT0000: Cannot find sequence 0xf00000400: rc = -2 [ 5402.259494] LustreError: 188716:0:(fld_handler.c:264:fld_server_lookup()) Skipped 3 previous similar messages [ 5403.369806] Lustre: DEBUG MARKER: == sanity test 154c: lfs path2fid and fid2path multiple arguments ========================================================== 11:12:06 (1693307526) [ 5404.291661] Lustre: DEBUG MARKER: == sanity test 154d: Verify open file fid ================ 11:12:07 (1693307527) [ 5405.477411] Lustre: DEBUG MARKER: == sanity test 154e: .lustre is not returned by readdir == 11:12:08 (1693307528) [ 5406.292383] Lustre: DEBUG MARKER: == sanity test 154f: get parent fids by reading link ea == 11:12:09 (1693307529) [ 5407.509103] Lustre: DEBUG MARKER: == sanity test 154g: various llapi FID tests ============= 11:12:10 (1693307530) [ 5505.309172] Lustre: DEBUG MARKER: == sanity test 154h: Verify interactive path2fid ========= 11:13:48 (1693307628) [ 5506.050334] Lustre: DEBUG MARKER: == sanity test 155a: Verify small file correctness: read cache:on write_cache:on ========================================================== 11:13:49 (1693307629) [ 5507.478991] Lustre: DEBUG MARKER: == sanity test 155b: Verify small file correctness: read cache:on write_cache:off ========================================================== 11:13:50 (1693307630) [ 5509.022111] Lustre: DEBUG MARKER: == sanity test 155c: Verify small file correctness: read cache:off write_cache:on ========================================================== 11:13:52 (1693307632) [ 5510.519689] Lustre: DEBUG MARKER: == sanity test 155d: Verify small file correctness: read cache:off write_cache:off ========================================================== 11:13:53 (1693307633) [ 5511.685866] Lustre: DEBUG MARKER: == sanity test 155e: Verify big file correctness: read cache:on write_cache:on ========================================================== 11:13:54 (1693307634) [ 5530.695089] Lustre: DEBUG MARKER: == sanity test 155f: Verify big file correctness: read cache:on write_cache:off ========================================================== 11:14:13 (1693307653) [ 5545.479310] Lustre: DEBUG MARKER: == sanity test 155g: Verify big file correctness: read cache:off write_cache:on ========================================================== 11:14:28 (1693307668) [ 5560.685143] Lustre: DEBUG MARKER: == sanity test 155h: Verify big file correctness: read cache:off write_cache:off ========================================================== 11:14:43 (1693307683) [ 5576.040802] Lustre: DEBUG MARKER: == sanity test 156: Verification of tunables ============= 11:14:59 (1693307699) [ 5576.276103] Lustre: DEBUG MARKER: SKIP: sanity test_156 LU-1956/LU-2261: stats not implemented on OSD ZFS [ 5576.454002] Lustre: DEBUG MARKER: == sanity test 160a: changelog sanity ==================== 11:14:59 (1693307699) [ 5576.628483] Lustre: lustre-MDD0000: changelog on [ 5578.257351] systemd[1]: mnt-lustre\x2dmds1.mount: Succeeded. [ 5578.305820] Lustre: Failing over lustre-MDT0000 [ 5580.470279] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5580.479032] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5580.485658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5581.991368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5581.998865] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5583.501075] Lustre: server umount lustre-MDT0000 complete [ 5584.499785] LustreError: 166-1: MGC192.168.121.234@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5584.500134] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5584.500636] Lustre: Evicted from MGS (at 192.168.121.234@tcp) after server handle changed from 0x9ecb860c6bacdcc4 to 0x9ecb860c6bb6f1b7 [ 5584.500959] LustreError: Skipped 1 previous similar message [ 5584.501710] Lustre: MGC192.168.121.234@tcp: Connection restored to (at 0@lo) [ 5584.684351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5584.730873] Lustre: lustre-MDD0000: changelog on [ 5584.731017] Lustre: Skipped 1 previous similar message [ 5584.736127] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5586.000439] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5586.009293] LustreError: 187570:0:(client.c:3259:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@0000000071389265 x1775561289248448/t25769805836(25769805836) o101->lustre-MDT0000-mdc-ffff8f75da24e000@0@lo:12/10 lens 648/608 e 0 to 0 dl 1693307753 ref 2 fl Interpret:RPQU/204/0 rc 301/301 uid:0 gid:0 job:'rm.0' [ 5589.672724] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.121.234@tcp (at 0@lo) [ 5589.959209] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 5589.986825] Lustre: lustre-MDD0000: changelog off [ 5589.988627] Lustre: lustre-OST0000: deleting orphan objects from 0x2800013a1:20353 to 0x2800013a1:20417 [ 5589.996697] Lustre: lustre-OST0001: deleting orphan objects from 0x2c00013a1:20348 to 0x2c00013a1:20417 [ 5590.693034] Lustre: DEBUG MARKER: == sanity test 160b: Verify that very long rename doesn't crash in changelog ========================================================== 11:15:13 (1693307713) [ 5590.867620] Lustre: lustre-MDD0000: changelog on [ 5591.647177] Lustre: lustre-MDD0001: changelog off [ 5591.647266] Lustre: Skipped 1 previous similar message [ 5592.037659] Lustre: DEBUG MARKER: == sanity test 160c: verify that changelog log catch the truncate event ========================================================== 11:15:15 (1693307715) [ 5593.489167] Lustre: lustre-MDD0001: changelog off [ 5593.501672] Lustre: Skipped 1 previous similar message [ 5593.998820] Lustre: DEBUG MARKER: == sanity test 160d: verify that changelog log catch the migrate event ========================================================== 11:15:17 (1693307717) [ 5594.380482] Lustre: lustre-MDD0000: changelog on [ 5594.380699] Lustre: Skipped 3 previous similar messages [ 5595.485667] Lustre: lustre-MDD0001: changelog off [ 5595.485773] Lustre: Skipped 1 previous similar message [ 5595.921141] Lustre: DEBUG MARKER: == sanity test 160e: changelog negative testing (should return errors) ========================================================== 11:15:19 (1693307719) [ 5597.465685] Lustre: DEBUG MARKER: == sanity test 160f: changelog garbage collect (timestamped users) ========================================================== 11:15:20 (1693307720) [ 5598.505244] Lustre: DEBUG MARKER: 1693307721: creating first files [ 5613.826906] Lustre: *** cfs_fail_loc=1313, val=0*** [ 5613.827083] Lustre: 228861:0:(mdd_dir.c:918:mdd_changelog_emrg_cleanup()) lustre-MDD0000: changelog has only 3 free catalog entries [ 5613.827258] Lustre: 228861:0:(mdd_dir.c:1001:mdd_changelog_store()) lustre-MDD0000: starting changelog garbage collection [ 5613.827816] Lustre: 242259:0:(mdd_trans.c:161:mdd_chlg_garbage_collect()) lustre-MDD0000: force deregister of changelog user cl7 idle for 16s with 4 unprocessed records [ 5615.228865] Lustre: lustre-MDD0001: changelog off [ 5615.229031] Lustre: Skipped 3 previous similar messages [ 5615.579740] Lustre: DEBUG MARKER: == sanity test 160g: changelog garbage collect on idle records ========================================================== 11:15:38 (1693307738) [ 5615.876425] Lustre: lustre-MDD0000: changelog on [ 5615.876526] Lustre: Skipped 5 previous similar messages [ 5618.176200] Lustre: 188716:0:(mdd_dir.c:1001:mdd_changelog_store()) lustre-MDD0000: starting changelog garbage collection [ 5618.176660] Lustre: 188716:0:(mdd_dir.c:1001:mdd_changelog_store()) Skipped 1 previous similar message [ 5618.188710] Lustre: 242911:0:(mdd_trans.c:161:mdd_chlg_garbage_collect()) lustre-MDD0000: force deregister of changelog user cl9 idle for 2s with 4 unprocessed records [ 5618.188965] Lustre: 242911:0:(mdd_trans.c:161:mdd_chlg_garbage_collect()) Skipped 1 previous similar message [ 5619.641665] Lustre: DEBUG MARKER: == sanity test 160h: changelog gc thread stop upon umount, orphan records delete ========================================================== 11:15:42 (1693307742) [ 5632.184718] Lustre: *** cfs_fail_loc=1316, val=0*** [ 5632.184974] Lustre: 228861:0:(mdd_dir.c:1001:mdd_changelog_store()) lustre-MDD0001: simulate starting changelog garbage collection [ 5632.185254] Lustre: 228861:0:(mdd_dir.c:1001:mdd_changelog_store()) Skipped 1 previous similar message [ 5632.204508] Lustre: 243606:0:(mdd_trans.c:161:mdd_chlg_garbage_collect()) lustre-MDD0000: force deregister of changelog user cl11 idle for 12s with 4 unprocessed records [ 5632.204724] Lustre: 243606:0:(mdd_trans.c:161:mdd_chlg_garbage_collect()) Skipped 1 previous similar message [ 5632.408210] systemd[1]: mnt-lustre\x2dmds1.mount: Succeeded. [ 5632.410029] systemd[1]: mnt-lustre\x2dmds2.mount: Succeeded. [ 5632.443219] Lustre: Failing over lustre-MDT0001 [ 5632.451316] Lustre: Failing over lustre-MDT0000 [ 5632.597782] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_disconnect to node 0@lo failed: rc = -19 [ 5632.598086] LustreError: 243663:0:(osp_dev.c:495:osp_disconnect()) lustre-MDT0000-osp-MDT0001: can't disconnect: rc = -19 [ 5632.599295] LustreError: 243663:0:(lod_dev.c:263:lod_sub_process_config()) lustre-MDT0001-mdtlov: error cleaning up LOD index 0: cmd 0xcf031 : rc = -19 [ 5633.322840] Lustre: server umount lustre-MDT0001 complete [ 5633.431623] Lustre: server umount lustre-MDT0000 complete [ 5636.463719] LustreError: 166-1: MGC192.168.121.234@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5636.464160] Lustre: Evicted from MGS (at 192.168.121.234@tcp) after server handle changed from 0x9ecb860c6bb6f1b7 to 0x9ecb860c6bb708af [ 5636.464511] Lustre: MGC192.168.121.234@tcp: Connection restored to 192.168.121.234@tcp (at 0@lo) [ 5636.464583] Lustre: Skipped 4 previous similar messages [ 5636.612686] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5636.612999] LustreError: Skipped 4 previous similar messages [ 5636.695887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5636.716219] Lustre: lustre-MDD0000: changelog on [ 5636.716983] Lustre: Skipped 3 previous similar messages [ 5636.717206] Lustre: 244197:0:(mdd_device.c:641:mdd_changelog_llog_init()) lustre-MDD0000 : orphan changelog records found, starting from index 35 to index 36, being cleared now [ 5636.725018] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5637.951597] Lustre: lustre-MDT0000-mdc-ffff8f75da24e000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5637.953345] Lustre: Skipped 3 previous similar messages [ 5637.957594] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5638.009592] LustreError: 187570:0:(client.c:3259:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@0000000071389265 x1775561289248448/t25769805836(25769805836) o101->lustre-MDT0000-mdc-ffff8f75da24e000@0@lo:12/10 lens 648/608 e 0 to 0 dl 1693307804 ref 2 fl Interpret:RPQU/204/0 rc 301/301 uid:0 gid:0 job:'rm.0' [ 5638.010058] LustreError: 187570:0:(client.c:3259:ptlrpc_replay_interpret()) Skipped 4 previous similar messages [ 5638.383728] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114 [ 5638.387748] LustreError: Skipped 1 previous similar message [ 5638.740932] Lustre: 244573:0:(mdd_device.c:641:mdd_changelog_llog_init()) lustre-MDD0001 : orphan changelog records found, starting from index 11 to index 12, being cleared now [ 5638.761217] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 5639.443758] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5643.510066] Lustre: 187574:0:(client.c:2309:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1693307759/real 1693307759] req@00000000b42580e3 x1775561292552448/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1693307766 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 uid:0 gid:0 job:'kworker/u4:0.0' [ 5643.592542] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 192.168.121.234@tcp (at 0@lo) [ 5643.603182] Lustre: Skipped 2 previous similar messages [ 5644.033266] Lustre: lustre-MDT0001: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 5644.045710] Lustre: lustre-OST0000: deleting orphan objects from 0x280000bd1:13734 to 0x280000bd1:13761 [ 5644.049301] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:14342 to 0x2c0000400:14369 [ 5644.224830] Lustre: lustre-OST0000: deleting orphan objects from 0x2800013a1:20419 to 0x2800013a1:20449 [ 5644.230108] Lustre: lustre-OST0001: deleting orphan objects from 0x2c00013a1:20348 to 0x2c00013a1:20449 [ 5644.924566] Lustre: lustre-MDD0001: changelog off [ 5644.924691] Lustre: Skipped 3 previous similar messages [ 5645.236520] Lustre: DEBUG MARKER: == sanity test 160i: changelog user register/unregister race ========================================================== 11:16:08 (1693307768) [ 5646.097877] LustreError: 245272:0:(libcfs_fail.h:190:cfs_race()) cfs_race id 1315 sleeping [ 5648.120934] LustreError: 245313:0:(libcfs_fail.h:201:cfs_race()) cfs_fail_race id 1315 waking [ 5648.122200] LustreError: 245272:0:(libcfs_fail.h:199:cfs_race()) cfs_fail_race id 1315 awake: rc=297 [ 5649.633243] Lustre: DEBUG MARKER: == sanity test 160j: client can be umounted while its chanangelog is being used ========================================================== 11:16:12 (1693307772) [ 5650.044736] Lustre: Mounted lustre-client [ 5650.810734] systemd[1]: mnt-lustre.mount: Succeeded. [ 5650.949329] Lustre: Unmounted lustre-client [ 5652.287936] systemd[1]: mnt-lustre2.mount: Succeeded. [ 5652.345306] Lustre: Unmounted lustre-client [ 5652.634249] Lustre: DEBUG MARKER: == sanity test 160k: Verify that changelog records are not lost ========================================================== 11:16:15 (1693307775) [ 5652.957391] Lustre: lustre-MDD0000: changelog on [ 5652.957606] Lustre: Skipped 7 previous similar messages [ 5653.245752] LustreError: 244227:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 15d sleeping for 3000ms [ 5656.320061] LustreError: 244227:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 15d awake [ 5661.125498] Lustre: DEBUG MARKER: == sanity test 160l: Verify that MTIME changelog records contain the parent FID ========================================================== 11:16:24 (1693307784) [ 5664.264872] Lustre: lustre-MDD0001: changelog off [ 5664.265021] Lustre: Skipped 9 previous similar messages [ 5664.781812] Lustre: DEBUG MARKER: == sanity test 160m: Changelog clear race ================ 11:16:27 (1693307787) [ 5666.044705] LustreError: 244231:0:(libcfs_fail.h:190:cfs_race()) cfs_race id 15f sleeping [ 5668.049979] LustreError: 244229:0:(libcfs_fail.h:201:cfs_race()) cfs_fail_race id 15f waking [ 5668.050260] LustreError: 244229:0:(libcfs_fail.h:201:cfs_race()) Skipped 2 previous similar messages [ 5668.076001] LustreError: 244231:0:(libcfs_fail.h:199:cfs_race()) cfs_fail_race id 15f awake: rc=299 [ 5669.366628] Lustre: DEBUG MARKER: == sanity test 160n: Changelog destroy race ============== 11:16:32 (1693307792) [ 6110.809872] LustreError: 244229:0:(libcfs_fail.h:190:cfs_race()) cfs_race id 16c sleeping [ 6112.819348] LustreError: 246134:0:(libcfs_fail.h:201:cfs_race()) cfs_fail_race id 16c waking [ 6112.826244] LustreError: 244229:0:(libcfs_fail.h:199:cfs_race()) cfs_fail_race id 16c awake: rc=298 [ 6117.997363] LustreError: 244229:0:(osd_handler.c:234:osd_trans_start()) lustre-MDT0000: can't assign tx: rc = -2 [ 6123.887855] Lustre: lustre-MDD0001: changelog off [ 6123.888082] Lustre: Skipped 3 previous similar messages [ 6124.317058] Lustre: DEBUG MARKER: == sanity test 160o: changelog user name and mask ======== 11:24:07 (1693308247) [ 6124.597352] Lustre: lustre-MDD0000: changelog on [ 6124.597557] Lustre: Skipped 7 previous similar messages [ 6124.835480] LustreError: 279216:0:(mdd_device.c:1726:mdd_changelog_name_check()) lustre-MDD0000: wrong char '#' in name 'Tt3_-#': rc = -22 [ 6124.863049] Lustre: 279222:0:(mdd_device.c:1743:mdd_changelog_name_check()) lustre-MDD0000: changelog name test_160o exists already: rc = -17 [ 6124.902582] LustreError: 279228:0:(mdd_device.c:1735:mdd_changelog_name_check()) lustre-MDD0000: name 'test_160toolongname' is over 16 symbols limit: rc = -36 [ 6128.052302] Lustre: DEBUG MARKER: == sanity test 160p: Changelog orphan cleanup with no users ========================================================== 11:24:11 (1693308251) [ 6128.435120] Lustre: DEBUG MARKER: SKIP: sanity test_160p ldiskfs only test [ 6128.704133] Lustre: DEBUG MARKER: == sanity test 160q: changelog effective mask is DEFMASK if not set ========================================================== 11:24:11 (1693308251) [ 6129.933386] Lustre: DEBUG MARKER: SKIP: sanity test_160s skipping excluded test 160s [ 6130.276742] Lustre: DEBUG MARKER: == sanity test 160t: changelog garbage collect on lack of space ========================================================== 11:24:13 (1693308253) [ 6158.213739] Lustre: *** cfs_fail_loc=18c, val=0*** [ 6158.227771] Lustre: Skipped 1 previous similar message [ 6158.227861] Lustre: 244230:0:(mdd_dir.c:888:mdd_changelog_is_space_safe()) lustre-MDD0000: changelog uses 31MB with 3MB space limit [ 6158.228030] Lustre: 244230:0:(mdd_dir.c:1001:mdd_changelog_store()) lustre-MDD0000: starting changelog garbage collection [ 6158.228179] Lustre: 244230:0:(mdd_dir.c:1001:mdd_changelog_store()) Skipped 1 previous similar message [ 6158.280201] Lustre: 280572:0:(mdd_trans.c:161:mdd_chlg_garbage_collect()) lustre-MDD0000: force deregister of changelog user cl25-user1 idle for 28s with 7506 unprocessed records [ 6158.294140] Lustre: 280572:0:(mdd_trans.c:161:mdd_chlg_garbage_collect()) Skipped 1 previous similar message [ 6159.985111] Lustre: *** cfs_fail_loc=18c, val=0*** [ 6164.506097] Lustre: DEBUG MARKER: == sanity test 161a: link ea sanity ====================== 11:24:47 (1693308287) [ 6172.316301] Lustre: DEBUG MARKER: == sanity test 161b: link ea sanity under remote directory ========================================================== 11:24:55 (1693308295) [ 6180.418798] Lustre: DEBUG MARKER: == sanity test 161c: check CL_RENME[UNLINK] changelog record flags ========================================================== 11:25:03 (1693308303) [ 6184.914481] Lustre: DEBUG MARKER: == sanity test 161d: create with concurrent .lustre/fid access ========================================================== 11:25:07 (1693308307) [ 6185.467443] LustreError: 281972:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 140c sleeping for 5000ms [ 6187.780086] LustreError: 281972:0:(fail.c:144:__cfs_fail_timeout_set()) cfs_fail_timeout interrupted [ 6188.495981] Lustre: lustre-MDD0001: changelog off [ 6188.499600] Lustre: Skipped 8 previous similar messages [ 6189.053783] Lustre: DEBUG MARKER: == sanity test 162a: path lookup sanity ================== 11:25:12 (1693308312) [ 6191.276122] Lustre: DEBUG MARKER: == sanity test 162b: striped directory path lookup sanity ========================================================== 11:25:14 (1693308314) [ 6192.575725] Lustre: DEBUG MARKER: == sanity test 162c: fid2path works with paths 100 or more directories deep ========================================================== 11:25:15 (1693308315) [ 6220.717105] Lustre: DEBUG MARKER: == sanity test 165a: ofd access log discovery ============ 11:25:43 (1693308343) [ 6225.974993] systemd[1]: mnt-lustre\x2dost1.mount: Succeeded. [ 6226.027662] Lustre: Failing over lustre-OST0000 [ 6226.231437] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6226.231601] Lustre: Skipped 5 previous similar messages [ 6226.232580] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6226.236470] Lustre: Skipped 4 previous similar messages [ 6227.748292] Lustre: server umount lustre-OST0000 complete [ 6231.291391] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6231.562271] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 6231.566909] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 6232.277491] Lustre: DEBUG MARKER: == sanity test 165b: ofd access log entries are produced and consumed ========================================================== 11:25:55 (1693308355) [ 6233.356581] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 6233.457136] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 6233.460076] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.121.234@tcp (at 0@lo) [ 6233.460266] Lustre: Skipped 1 previous similar message [ 6233.464537] Lustre: Skipped 5 previous similar messages [ 6253.920734] systemd[1]: mnt-lustre\x2dost1.mount: Succeeded. [ 6253.957274] Lustre: Failing over lustre-OST0000 [ 6256.005166] Lustre: server umount lustre-OST0000 complete [ 6256.712038] Lustre: lustre-OST0000-osc-ffff8f7564f58000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6256.712309] Lustre: Skipped 2 previous similar messages [ 6256.712657] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6256.712768] LustreError: Skipped 2 previous similar messages [ 6257.444950] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 6257.448291] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 6258.137840] Lustre: DEBUG MARKER: == sanity test 165c: full ofd access logs do not block IOs ========================================================== 11:26:21 (1693308381) [ 6259.445647] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 6259.525079] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 6259.526116] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.121.234@tcp (at 0@lo) [ 6259.526259] Lustre: Skipped 1 previous similar message [ 6267.452753] systemd[1]: mnt-lustre\x2dost1.mount: Succeeded. [ 6267.503744] Lustre: Failing over lustre-OST0000 [ 6267.558877] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6267.562864] Lustre: Skipped 2 previous similar messages [ 6269.696271] Lustre: server umount lustre-OST0000 complete [ 6271.427267] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 6271.429127] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 6272.129915] Lustre: DEBUG MARKER: == sanity test 165d: ofd_access_log mask works =========== 11:26:35 (1693308395) [ 6273.116781] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 6273.229203] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 6290.207544] systemd[1]: mnt-lustre\x2dost1.mount: Succeeded. [ 6290.249683] Lustre: Failing over lustre-OST0000 [ 6290.280329] Lustre: server umount lustre-OST0000 complete [ 6291.596045] Lustre: lustre-OST0000-osc-ffff8f7564f58000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6291.597134] Lustre: Skipped 2 previous similar messages [ 6291.597832] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6291.598076] LustreError: Skipped 2 previous similar messages [ 6292.597038] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 6292.602319] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 6293.615681] Lustre: DEBUG MARKER: == sanity test 165e: ofd_access_log MDT index filter works ========================================================== 11:26:56 (1693308416) [ 6294.566941] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 6294.655588] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 6294.659772] Lustre: lustre-OST0000-osc-ffff8f7564f58000: Connection restored to 192.168.121.234@tcp (at 0@lo) [ 6294.672367] Lustre: Skipped 5 previous similar messages [ 6302.323596] systemd[1]: mnt-lustre\x2dost1.mount: Succeeded. [ 6302.387202] Lustre: Failing over lustre-OST0000 [ 6302.420650] Lustre: server umount lustre-OST0000 complete [ 6303.967319] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 6304.776800] Lustre: DEBUG MARKER: == sanity test 165f: ofd_access_log_reader --exit-on-close works ========================================================== 11:27:07 (1693308427) [ 6309.934582] systemd[1]: mnt-lustre\x2dost1.mount: Succeeded. [ 6309.966718] Lustre: Failing over lustre-OST0000 [ 6310.005173] Lustre: server umount lustre-OST0000 complete [ 6310.794413] LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6312.474560] Lustre: lustre-OST0000: Imperative Recovery enabled, recovery window shrunk from 60-180 down to 60-180 [ 6313.205858] Lustre: DEBUG MARKER: == sanity test 169: parallel read and truncate should not deadlock ========================================================== 11:27:16 (1693308436) [ 6313.402641] Lustre: DEBUG MARKER: creating a 10 Mb file [ 6314.108486] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 6314.124381] Lustre: Skipped 1 previous similar message [ 6333.635846] Lustre: DEBUG MARKER: starting reads [ 6334.035450] Lustre: DEBUG MARKER: truncating the file [ 6334.564070] Lustre: DEBUG MARKER: killing dd [ 6334.741667] Lustre: DEBUG MARKER: removing the temporary file [ 6335.726590] Lustre: DEBUG MARKER: == sanity test 170: test lctl df to handle corrupted log =============================================================================== 11:27:38 (1693308458) [ 6335.848817] Lustre: debug daemon will attempt to start writing to /tmp/f170.sanity_log_good (512000kB max) [ 6335.912830] Lustre: shutting down debug daemon thread... [ 6335.948532] Lustre: debug daemon will attempt to start writing to /tmp/f170.sanity_log_good (512000kB max) [ 6335.980085] Lustre: shutting down debug daemon thread... [ 6338.076972] Lustre: DEBUG MARKER: == sanity test 171: test libcfs_debug_dumplog_thread stuck in do_exit() ================================================================ 11:27:41 (1693308461) [ 6338.164677] Lustre: *** cfs_fail_loc=50e, val=0*** [ 6338.184658] LustreError: dumping log to /tmp/lustre-log.1693308461.244226 [ 6338.248547] LustreError: 289047:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50e sleeping for 3000ms [ 6339.350229] Lustre: *** cfs_fail_loc=50e, val=0*** [ 6339.350469] LustreError: dumping log to /tmp/lustre-log.1693308462.189430 [ 6339.350562] Lustre: Skipped 5 previous similar messages [ 6339.920351] LustreError: dumping log to /tmp/lustre-log.1693308463.244807 [ 6341.340061] LustreError: 289047:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50e awake [ 6341.353167] LustreError: dumping log to /tmp/lustre-log.1693308464.289047 [ 6342.457299] Lustre: DEBUG MARKER: == sanity test 172: manual device removal with lctl cleanup/detach ================================================================ 11:27:45 (1693308465) [ 6342.489650] systemd[1]: mnt-lustre.mount: Succeeded. [ 6343.890351] Lustre: *** cfs_fail_loc=60e, val=0*** [ 6343.906028] Lustre: Unmounted lustre-client [ 6345.010583] Lustre: Mounted lustre-client [ 6345.021500] Lustre: Skipped 1 previous similar message [ 6345.533577] Lustre: DEBUG MARKER: == sanity test 180a: test obdecho on osc ================= 11:27:48 (1693308468) [ 6345.806902] Lustre: DEBUG MARKER: SKIP: sanity test_180a obdecho on osc is no longer supported [ 6346.220645] Lustre: DEBUG MARKER: == sanity test 180b: test obdecho directly on obdfilter == 11:27:49 (1693308469) [ 6347.383087] Lustre: DEBUG MARKER: tmp.136UuqwYDI: executing load_module obdecho/obdecho [ 6349.449033] Lustre: DEBUG MARKER: == sanity test 180c: test huge bulk I/O size on obdfilter, don't LASSERT ========================================================== 11:27:52 (1693308472) [ 6350.534363] Lustre: DEBUG MARKER: tmp.136UuqwYDI: executing load_module obdecho/obdecho [ 6350.628020] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6356.729431] Lustre: DEBUG MARKER: == sanity test 181: Test open-unlinked dir ================================================================================== 11:27:59 (1693308479) [ 6387.307087] Lustre: DEBUG MARKER: == sanity test 182a: Test parallel modify metadata operations from mdc ========================================================== 11:28:30 (1693308510) [ 6440.260665] Lustre: DEBUG MARKER: == sanity test 182b: Test parallel modify metadata operations from osp ========================================================== 11:29:23 (1693308563) [ 6665.579895] Lustre: DEBUG MARKER: == sanity test 183: No crash or request leak in case of strange dispositions ================================================================== 11:33:08 (1693308788) [ 6665.868128] Lustre: *** cfs_fail_loc=148, val=0*** [ 6666.924360] Lustre: DEBUG MARKER: == sanity test 184a: Basic layout swap =================== 11:33:09 (1693308789) [ 6669.193683] Lustre: DEBUG MARKER: == sanity test 184b: Forbidden layout swap (will generate errors) ========================================================== 11:33:12 (1693308792) [ 6671.224508] Lustre: DEBUG MARKER: == sanity test 184c: Concurrent write and layout swap ==== 11:33:14 (1693308794) [ 6676.874573] Lustre: DEBUG MARKER: == sanity test 184d: allow stripeless layouts swap ======= 11:33:19 (1693308799) [ 6679.389014] Lustre: DEBUG MARKER: == sanity test 184e: Recreate layout after stripeless layout swaps ========================================================== 11:33:22 (1693308802) [ 6681.115270] Lustre: DEBUG MARKER: == sanity test 184f: IOC_MDC_GETFILEINFO for files with long names but no striping ========================================================== 11:33:24 (1693308804) [ 6682.114421] Lustre: DEBUG MARKER: == sanity test 185: Volatile file support ================ 11:33:25 (1693308805) [ 6684.376143] Lustre: DEBUG MARKER: == sanity test 185a: Volatile file creation in .lustre/fid/ ========================================================== 11:33:27 (1693308807) [ 6689.688385] Lustre: DEBUG MARKER: == sanity test 187a: Test data version change ============ 11:33:32 (1693308812) [ 6691.056024] Lustre: DEBUG MARKER: == sanity test 187b: Test data version change on volatile file ========================================================== 11:33:34 (1693308814) [ 6692.170918] Lustre: DEBUG MARKER: == sanity test 200: OST pools ============================ 11:33:35 (1693308815) [ 6707.269607] Lustre: DEBUG MARKER: == sanity test 204a: Print default stripe attributes ===== 11:33:50 (1693308830) [ 6708.836027] Lustre: DEBUG MARKER: == sanity test 204b: Print default stripe size and offset ========================================================== 11:33:51 (1693308831) [ 6710.171822] Lustre: DEBUG MARKER: == sanity test 204c: Print default stripe count and offset ========================================================== 11:33:53 (1693308833) [ 6711.545721] Lustre: DEBUG MARKER: == sanity test 204d: Print default stripe count and size ========================================================== 11:33:54 (1693308834) [ 6713.058125] Lustre: DEBUG MARKER: == sanity test 204e: Print raw stripe attributes ========= 11:33:56 (1693308836) [ 6714.431861] Lustre: DEBUG MARKER: == sanity test 204f: Print raw stripe size and offset ==== 11:33:57 (1693308837) [ 6715.737993] Lustre: DEBUG MARKER: == sanity test 204g: Print raw stripe count and offset === 11:33:58 (1693308838) [ 6717.085120] Lustre: DEBUG MARKER: == sanity test 204h: Print raw stripe count and size ===== 11:34:00 (1693308840) [ 6718.679621] Lustre: DEBUG MARKER: == sanity test 205a: Verify job stats ==================== 11:34:01 (1693308841) [ 6721.971962] Lustre: lustre-MDD0000: changelog on [ 6721.972921] Lustre: Skipped 8 previous similar messages [ 6722.719620] Lustre: DEBUG MARKER: Test: /mnt/build/lustre/tests/../utils/lfs mkdir -i 0 -c 1 /mnt/lustre/d205a.sanity [ 6722.964248] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.lfs.25069 [ 6723.347333] Lustre: DEBUG MARKER: Test: rmdir /mnt/lustre/d205a.sanity [ 6723.597452] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.rmdir.7233 [ 6723.967788] Lustre: DEBUG MARKER: Test: lfs mkdir -i 1 /mnt/lustre/d205a.sanity.remote [ 6724.247650] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.lfs.30735 [ 6724.614263] Lustre: DEBUG MARKER: Test: mknod /mnt/lustre/f205a.sanity c 1 3 [ 6724.849652] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.mknod.14042 [ 6725.279014] Lustre: DEBUG MARKER: Test: rm -f /mnt/lustre/f205a.sanity [ 6725.527345] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.rm.11499 [ 6726.012953] Lustre: DEBUG MARKER: Test: /mnt/build/lustre/tests/../utils/lfs setstripe -i 0 -c 1 /mnt/lustre/f205a.sanity [ 6726.325283] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.lfs.3940 [ 6726.839135] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205a.sanity [ 6727.154654] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.touch.30052 [ 6727.772952] Lustre: DEBUG MARKER: Test: dd if=/dev/zero of=/mnt/lustre/f205a.sanity bs=1M count=1 oflag=sync [ 6727.981743] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.dd.4701 [ 6728.610583] Lustre: DEBUG MARKER: Test: dd if=/mnt/lustre/f205a.sanity of=/dev/null bs=1M count=1 iflag=direct [ 6728.935800] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.dd.11724 [ 6729.312793] Lustre: DEBUG MARKER: Test: /mnt/build/lustre/tests/../tests/truncate /mnt/lustre/f205a.sanity 0 [ 6729.543780] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.truncate.17543 [ 6729.947615] Lustre: DEBUG MARKER: Test: mv -f /mnt/lustre/f205a.sanity /mnt/lustre/d205a.sanity.rename [ 6730.124880] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.mv.9365 [ 6730.543032] Lustre: DEBUG MARKER: Test: /mnt/build/lustre/tests/../utils/lfs mkdir -i 0 -c 1 /mnt/lustre/d205a.sanity.expire [ 6730.889637] Lustre: DEBUG MARKER: Using JobID environment nodelocal=id.205a.lfs.27854 [ 6734.469011] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205a.sanity [ 6734.764314] Lustre: DEBUG MARKER: Using JobID environment USER=S.root.touch.0.tmp.136UuqwYDI.E [ 6735.114956] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205a.sanity [ 6735.311193] Lustre: DEBUG MARKER: Using JobID environment USER=S.root.touch.0.tmp.E [ 6735.807926] Lustre: DEBUG MARKER: Test: touch /mnt/lustre/f205a.sanity [ 6736.089681] Lustre: DEBUG MARKER: Using JobID environment session=S.root.touch.0.tmp.136UuqwYDI.E [ 6737.137641] Lustre: lustre-MDD0001: changelog off [ 6737.142462] Lustre: Skipped 1 previous similar message [ 6740.429058] Lustre: DEBUG MARKER: == sanity test 205b: Verify job stats jobid and output format ========================================================== 11:34:23 (1693308863) [ 6742.481801] Lustre: DEBUG MARKER: == sanity test 205c: Verify client stats format ========== 11:34:25 (1693308865) [ 6743.421191] Lustre: DEBUG MARKER: == sanity test 205d: verify the format of some stats files ========================================================== 11:34:26 (1693308866) [ 6746.254228] Lustre: DEBUG MARKER: == sanity test 205e: verify the output of lljobstat ====== 11:34:29 (1693308869) [ 6754.010630] Lustre: DEBUG MARKER: == sanity test 205f: verify qos_ost_weights YAML format == 11:34:37 (1693308877) [ 6755.718439] Lustre: DEBUG MARKER: == sanity test 205g: stress test for job_stats procfile == 11:34:38 (1693308878) [ 6847.615613] Lustre: DEBUG MARKER: == sanity test 205h: check jobid xattr is stored correctly ========================================================== 11:36:10 (1693308970) [ 6850.170830] Lustre: DEBUG MARKER: == sanity test 205i: check job_xattr parameter accepts and rejects values correctly ========================================================== 11:36:13 (1693308973) [ 6852.454643] Lustre: DEBUG MARKER: == sanity test 206: fail lov_init_raid0() doesn't lbug === 11:36:15 (1693308975) [ 6852.574046] Lustre: *** cfs_fail_loc=1403, val=0*** [ 6852.585532] LustreError: 317468:0:(lcommon_cl.c:196:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200003ab3:0x758e:0x0]: rc = -5 [ 6852.586615] LustreError: 317468:0:(llite_lib.c:3594:ll_prep_inode()) new_inode -fatal: rc -5 [ 6853.973660] Lustre: DEBUG MARKER: == sanity test 207a: can refresh layout at glimpse ======= 11:36:16 (1693308976) [ 6854.159041] Lustre: *** cfs_fail_loc=170, val=0*** [ 6855.547198] Lustre: DEBUG MARKER: == sanity test 207b: can refresh layout at open ========== 11:36:18 (1693308978) [ 6855.682595] Lustre: *** cfs_fail_loc=171, val=0*** [ 6855.682847] Lustre: Skipped 1 previous similar message [ 6856.453987] Lustre: DEBUG MARKER: == sanity test 208: Exclusive open ======================= 11:36:19 (1693308979) [ 6862.850445] systemd[1]: mnt-lustre\x2dmds1.mount: Succeeded. [ 6862.903801] Lustre: Failing over lustre-MDT0000 [ 6863.505928] Lustre: server umount lustre-MDT0000 complete [ 6864.311280] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6864.311733] Lustre: Skipped 8 previous similar messages [ 6864.312367] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6864.312645] LustreError: Skipped 7 previous similar messages [ 6871.350057] Lustre: 187574:0:(client.c:2309:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1693308987/real 1693308987] req@000000002ba4e1ef x1775561332794304/t0(0) o400->MGC192.168.121.234@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1693308994 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 uid:0 gid:0 job:'kworker/u4:1.0' [ 6871.351775] Lustre: 187574:0:(client.c:2309:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 6871.351985] LustreError: 166-1: MGC192.168.121.234@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6888.712214] Lustre: Evicted from MGS (at 192.168.121.234@tcp) after server handle changed from 0x9ecb860c6bb708af to 0x9ecb860c6c4ec1e1 [ 6888.721401] Lustre: MGC192.168.121.234@tcp: Connection restored to 192.168.121.234@tcp (at 0@lo) [ 6888.725153] Lustre: Skipped 8 previous similar messages [ 6889.139195] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6889.139440] Lustre: Skipped 1 previous similar message [ 6889.199712] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6889.203962] Lustre: Skipped 2 previous similar messages [ 6890.644171] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6890.647546] Lustre: Skipped 2 previous similar messages [ 6890.668286] LustreError: 187570:0:(mdc_request.c:670:mdc_replay_open()) @@@ cannot properly replay without open data req@00000000dc94adfc x1775561332784256/t34359930356(34359930356) o101->lustre-MDT0000-mdc-ffff8f7411c6b000@0@lo:12/10 lens 608/608 e 0 to 0 dl 1693309020 ref 2 fl Interpret:RPQU/204/0 rc 301/301 uid:0 gid:0 job:'touch.0' [ 6890.669542] LustreError: 187570:0:(client.c:3259:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@00000000dc94adfc x1775561332784256/t34359930356(34359930356) o101->lustre-MDT0000-mdc-ffff8f7411c6b000@0@lo:12/10 lens 608/608 e 0 to 0 dl 1693309020 ref 2 fl Interpret:RPQU/204/0 rc 301/301 uid:0 gid:0 job:'touch.0' [ 6890.669930] LustreError: 187570:0:(client.c:3259:ptlrpc_replay_interpret()) Skipped 7 previous similar messages [ 6894.092918] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 6894.130371] Lustre: lustre-OST0000: deleting orphan objects from 0x2800013a1:35553 to 0x2800013a1:47857 [ 6894.134658] Lustre: lustre-OST0001: deleting orphan objects from 0x2c00013a1:34443 to 0x2c00013a1:50033 [ 6896.838428] Lustre: DEBUG MARKER: tmp.136UuqwYDI: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6897.632592] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6901.310626] systemd[1]: mnt-lustre\x2dmds1.mount: Succeeded. [ 6901.357154] Lustre: Failing over lustre-MDT0000 [ 6901.526344] LustreError: 318234:0:(osp_precreate.c:1003:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -5 [ 6901.526621] LustreError: 318234:0:(osp_precreate.c:1003:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message [ 6902.135879] Lustre: server umount lustre-MDT0000 complete [ 6904.230627] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6904.243760] LustreError: Skipped 1 previous similar message [ 6911.830059] Lustre: 187572:0:(client.c:2309:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1693309028/real 1693309028] req@000000001d238631 x1775561332815744/t0(0) o400->MGC192.168.121.234@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1693309035 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 uid:0 gid:0 job:'kworker/u4:0.0' [ 6911.830317] LustreError: 166-1: MGC192.168.121.234@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6917.913373] Lustre: Evicted from MGS (at 192.168.121.234@tcp) after server handle changed from 0x9ecb860c6c4ec1e1 to 0x9ecb860c6c51bfab [ 6918.216533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6919.062267] LustreError: 187570:0:(mdc_request.c:670:mdc_replay_open()) @@@ cannot properly replay without open data req@00000000dc94adfc x1775561332784256/t34359930356(34359930356) o101->lustre-MDT0000-mdc-ffff8f7411c6b000@0@lo:12/10 lens 608/608 e 0 to 0 dl 1693309049 ref 2 fl Interpret:RPQU/204/0 rc 301/301 uid:0 gid:0 job:'touch.0' [ 6919.065289] LustreError: 187570:0:(client.c:3259:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@0000000062f16e03 x1775561332813184/t38654705667(38654705667) o101->lustre-MDT0000-mdc-ffff8f7411c6b000@0@lo:12/10 lens 584/608 e 0 to 0 dl 1693309049 ref 2 fl Interpret:RPQU/204/0 rc 301/301 uid:0 gid:0 job:'multiop.0' [ 6919.065700] LustreError: 187570:0:(client.c:3259:ptlrpc_replay_interpret()) Skipped 2 previous similar messages [ 6922.272567] Lustre: lustre-OST0000: deleting orphan objects from 0x2800013a1:35553 to 0x2800013a1:47857 [ 6922.272738] Lustre: lustre-OST0001: deleting orphan objects from 0x2c00013a1:34443 to 0x2c00013a1:50033 [ 6924.522001] Lustre: DEBUG MARKER: tmp.136UuqwYDI: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6925.024012] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6927.307602] Lustre: DEBUG MARKER: SKIP: sanity test_209 skipping excluded test 209 [ 6927.737899] Lustre: DEBUG MARKER: == sanity test 210: lfs getstripe does not break leases == 11:37:30 (1693309050) [ 6931.062994] Lustre: DEBUG MARKER: == sanity test 212: Sendfile test ====================================================================================================== 11:37:34 (1693309054) [ 6932.632851] Lustre: DEBUG MARKER: == sanity test 213: OSC lock completion and cancel race don't crash - bug 18829 ========================================================== 11:37:35 (1693309055) [ 6932.733456] LustreError: 187574:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 40f sleeping for 10000ms [ 6942.830042] LustreError: 187574:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 40f awake [ 6944.152025] Lustre: DEBUG MARKER: == sanity test 214: hash-indexed directory test - bug 20133 ========================================================== 11:37:47 (1693309067) [ 6955.048200] Lustre: DEBUG MARKER: == sanity test 215: lnet exists and has proper content - bugs 18102, 21079, 21517 ========================================================== 11:37:58 (1693309078) [ 6956.618612] Lustre: DEBUG MARKER: == sanity test 216: check lockless direct write updates file size and kms correctly ========================================================== 11:37:59 (1693309079) [ 6959.642057] Lustre: DEBUG MARKER: == sanity test 217: check lctl ping for hostnames with embedded hyphen ('-') ========================================================== 11:38:02 (1693309082) [ 6960.545468] Lustre: DEBUG MARKER: == sanity test 218: parallel read and truncate should not deadlock ========================================================== 11:38:03 (1693309083) [ 6960.735373] Lustre: DEBUG MARKER: creating a 10 Mb file [ 6991.257093] Lustre: DEBUG MARKER: starting reads [ 6991.595128] Lustre: DEBUG MARKER: truncating the file [ 6991.806294] Lustre: DEBUG MARKER: killing dd [ 6991.997213] Lustre: DEBUG MARKER: removing the temporary file [ 6992.785031] Lustre: DEBUG MARKER: == sanity test 219: LU-394: Write partial won't cause uncontiguous pages vec at LND ========================================================== 11:38:35 (1693309115) [ 6992.845453] Lustre: *** cfs_fail_loc=411, val=0*** [ 6993.957855] Lustre: DEBUG MARKER: == sanity test 220: preallocated MDS objects still used if ENOSPC from OST ========================================================== 11:38:36 (1693309116) [ 6995.190239] Lustre: *** cfs_fail_loc=229, val=1*** [ 6999.172870] Lustre: *** cfs_fail_loc=229, val=0*** [ 6999.173210] Lustre: Skipped 1 previous similar message [ 7004.933468] Lustre: DEBUG MARKER: == sanity test 221: make sure fault and truncate race to not cause OOM ========================================================== 11:38:47 (1693309127) [ 7006.265617] Lustre: DEBUG MARKER: == sanity test 222a: AGL for ls should not trigger CLIO lock failure ========================================================== 11:38:49 (1693309129) [ 7006.671734] LustreError: 189428:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 31a sleeping for 5000ms [ 7006.685762] LustreError: 189428:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 7 previous similar messages [ 7006.780125] LustreError: 286782:0:(fail.c:144:__cfs_fail_timeout_set()) cfs_fail_timeout interrupted [ 7007.881376] Lustre: DEBUG MARKER: == sanity test 222b: AGL for rmdir should not trigger CLIO lock failure ========================================================== 11:38:50 (1693309130) [ 7009.412256] Lustre: DEBUG MARKER: == sanity test 223: osc reenqueue if without AGL lock granted ================================================================================= 11:38:52 (1693309132) [ 7011.173906] Lustre: DEBUG MARKER: == sanity test 224a: Don't panic on bulk IO failure ====== 11:38:54 (1693309134) [ 7011.266270] Lustre: *** cfs_fail_loc=508, val=2147483648*** [ 7011.266521] LustreError: 238536:0:(events.c:210:client_bulk_callback()) event type 1, status -5, desc 00000000b0ff0269 [ 7011.266780] Lustre: 187572:0:(client.c:2309:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1693309134/real 1693309134] req@00000000c46a1b93 x1775561333193152/t0(0) o4->lustre-OST0000-osc-ffff8f7411c6b000@0@lo:6/4 lens 488/448 e 0 to 1 dl 1693309141 ref 2 fl Rpc:eXQr/200/ffffffff rc 0/-1 uid:0 gid:0 job:'dd.0' [ 7011.267429] Lustre: lustre-OST0000-osc-ffff8f7411c6b000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7011.267774] Lustre: Skipped 9 previous similar messages [ 7011.277876] Lustre: lustre-OST0000: Client 93bee11c-0681-48b6-90a9-7882efb25284 (at 0@lo) reconnecting [ 7013.815325] Lustre: DEBUG MARKER: == sanity test 224b: Don't panic on bulk IO failure ====== 11:38:56 (1693309136) [ 7014.743481] LustreError: 238536:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 515 sleeping for 3000ms [ 7014.766876] LustreError: 238536:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 1 previous similar message [ 7017.850068] LustreError: 238536:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 515 awake [ 7019.614816] Lustre: DEBUG MARKER: == sanity test 224c: Don't hang if one of md lost during large bulk RPC ========================================================== 11:39:02 (1693309142) [ 7025.815597] Lustre: *** cfs_fail_loc=520, val=57344*** [ 7025.816736] LNet: *** cfs_fail_loc=e000, val=2147483648*** [ 7025.816871] Lustre: 187573:0:(client.c:2309:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1693309149/real 1693309149] req@000000008c1cf851 x1775561333212096/t0(0) o4->lustre-OST0000-osc-ffff8f7411c6b000@0@lo:6/4 lens 488/448 e 0 to 1 dl 1693309154 ref 2 fl Rpc:eXQr/200/ffffffff rc 0/-1 uid:0 gid:0 job:'dd.0' [ 7025.818087] Lustre: lustre-OST0000: Client 93bee11c-0681-48b6-90a9-7882efb25284 (at 0@lo) reconnecting [ 7025.819964] Lustre: lustre-OST0000-osc-ffff8f7411c6b000: Connection restored to 192.168.121.234@tcp (at 0@lo) [ 7025.820655] Lustre: Skipped 11 previous similar messages [ 7025.851142] LustreError: 239309:0:(ldlm_lib.c:3539:target_bulk_io()) @@@ bulk WRITE failed: rc = -107 req@000000007ef3c531 x1775561333211840/t0(0) o4->93bee11c-0681-48b6-90a9-7882efb25284@0@lo:0/0 lens 488/448 e 0 to 0 dl 1693309154 ref 1 fl Interpret:/200/0 rc 0/0 uid:0 gid:0 job:'dd.0' [ 7025.877396] Lustre: lustre-OST0000: Bulk IO write error with 93bee11c-0681-48b6-90a9-7882efb25284 (at 0@lo), client will retry: rc = -107 [ 7030.760056] Lustre: 187572:0:(client.c:2309:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1693309149/real 1693309149] req@00000000a2fff9fc x1775561333211840/t0(0) o4->lustre-OST0000-osc-ffff8f7411c6b000@0@lo:6/4 lens 488/448 e 0 to 1 dl 1693309154 ref 2 fl Rpc:XQr/202/ffffffff rc -11/-1 uid:0 gid:0 job:'dd.0' [ 7030.760905] LustreError: 187573:0:(client.c:2182:ptlrpc_check_set()) @@@ bulk transfer failed 0/1048576/0 req@00000000eed0069b x1775561333211968/t64424516955(64424516955) o4->lustre-OST0000-osc-ffff8f7411c6b000@0@lo:6/4 lens 488/448 e 0 to 1 dl 1693309154 ref 2 fl Bulk:ReXMQU/204/0 rc 0/0 uid:0 gid:0 job:'dd.0' [ 7030.761098] Lustre: lustre-OST0000: Client 93bee11c-0681-48b6-90a9-7882efb25284 (at 0@lo) reconnecting [ 7030.761537] LustreError: 187573:0:(osc_request.c:2409:osc_brw_redo_request()) @@@ redo for recoverable error -5 req@00000000eed0069b x1775561333211968/t64424516955(64424516955) o4->lustre-OST0000-osc-ffff8f7411c6b000@0@lo:6/4 lens 488/448 e 0 to 1 dl 1693309154 ref 2 fl Interpret:ReXMQU/204/0 rc -5/0 uid:0 gid:0 job:'dd.0' [ 7036.013431] Lustre: DEBUG MARKER: == sanity test 224d: Don't corrupt data on bulk IO timeout ========================================================== 11:39:18 (1693309158) [ 7037.385231] LustreError: 238906:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 515 sleeping for 22000ms [ 7057.750088] Lustre: 187574:0:(client.c:2309:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1693309160/real 1693309160] req@000000001a24faf2 x1775561333234752/t0(0) o3->lustre-OST0000-osc-ffff8f7411c6b000@0@lo:6/4 lens 488/440 e 0 to 1 dl 1693309180 ref 2 fl Bulk:RXMQU/200/0 rc 0/0 uid:0 gid:0 job:'dd.0' [ 7057.755492] Lustre: 187574:0:(client.c:2309:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 7057.759617] LustreError: 187574:0:(client.c:2182:ptlrpc_check_set()) @@@ bulk transfer failed 0/1048576/0 req@000000001a24faf2 x1775561333234752/t0(0) o3->lustre-OST0000-osc-ffff8f7411c6b000@0@lo:6/4 lens 488/440 e 0 to 1 dl 1693309180 ref 2 fl Bulk:ReXMQU/200/0 rc 0/0 uid:0 gid:0 job:'dd.0' [ 7057.759789] Lustre: lustre-OST0000: Client 93bee11c-0681-48b6-90a9-7882efb25284 (at 0@lo) reconnecting [ 7057.761236] LustreError: 187574:0:(osc_request.c:2409:osc_brw_redo_request()) @@@ redo for recoverable error -5 req@000000001a24faf2 x1775561333234752/t0(0) o3->lustre-OST0000-osc-ffff8f7411c6b000@0@lo:6/4 lens 488/440 e 0 to 1 dl 1693309180 ref 2 fl Interpret:ReXMQU/200/0 rc -5/0 uid:0 gid:0 job:'dd.0' [ 7058.830133] LustreError: 238906:0:(fail.c:144:__cfs_fail_timeout_set()) cfs_fail_timeout interrupted [ 7058.830351] LustreError: 238906:0:(fail.c:144:__cfs_fail_timeout_set()) Skipped 8 previous similar messages [ 7058.830492] LustreError: 238906:0:(ldlm_lib.c:3539:target_bulk_io()) @@@ bulk READ failed: rc = -107 req@0000000086df173d x1775561333234752/t0(0) o3->93bee11c-0681-48b6-90a9-7882efb25284@0@lo:0/0 lens 488/440 e 0 to 0 dl 1693309180 ref 1 fl Interpret:/200/0 rc 0/0 uid:0 gid:0 job:'dd.0' [ 7058.830909] Lustre: 238906:0:(service.c:2348:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/2s); client may timeout req@0000000086df173d x1775561333234752/t0(0) o3->93bee11c-0681-48b6-90a9-7882efb25284@0@lo:0/0 lens 488/440 e 0 to 0 dl 1693309180 ref 1 fl Complete:/200/0 rc 0/0 uid:0 gid:0 job:'dd.0' [ 7060.662824] Lustre: DEBUG MARKER: SKIP: sanity test_225a skipping excluded test 225a (base 225) [ 7061.003291] Lustre: DEBUG MARKER: SKIP: sanity test_225b skipping excluded test 225b (base 225) [ 7061.494766] Lustre: DEBUG MARKER: == sanity test 226a: call path2fid and fid2path on files of all type ========================================================== 11:39:44 (1693309184) [ 7063.739328] Lustre: DEBUG MARKER: == sanity test 226b: call path2fid and fid2path on files of all type under remote dir ========================================================== 11:39:46 (1693309186) [ 7065.812346] Lustre: DEBUG MARKER: == sanity test 226c: call path2fid and fid2path under remote dir with subdir mount ========================================================== 11:39:48 (1693309188) [ 7066.097887] Lustre: Mounted lustre-client [ 7066.958747] systemd[1]: mnt-submnt.mount: Succeeded. [ 7067.025328] BUG: unable to handle kernel paging request at ffff8f75cb9be538 [ 7067.025564] PGD 224e01067 P4D 224e01067 PUD 339925067 PMD 3398c8067 PTE 800ffffd34641060 [ 7067.025724] Oops: 0000 [#1] SMP DEBUG_PAGEALLOC [ 7067.025830] CPU: 0 PID: 187571 Comm: ptlrpcd_00_00 Tainted: G W O --------- - - 4.18.0 #2 [ 7067.026018] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 7067.026151] RIP: 0010:do_raw_spin_trylock+0x0/0x40 [ 7067.026257] Code: 12 1e bb 86 48 89 df e8 be fe ff ff eb 8a 89 c6 48 89 df e8 12 e3 ff ff 66 90 eb 8b 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 <8b> 07 85 c0 75 0d ba 01 00 00 00 f0 0f b1 17 85 c0 74 03 31 c0 c3 [ 7067.026648] RSP: 0018:ffff8f75f5ebbb88 EFLAGS: 00010046 [ 7067.026752] RAX: 0000000000000001 RBX: ffff8f75cb9be550 RCX: 0000000000000001 [ 7067.026909] RDX: 0000000000000001 RSI: 00000000f6de3d3b RDI: ffff8f75cb9be538 [ 7067.027069] RBP: ffff8f75cb9be538 R08: ffff8f762ad05200 R09: ffff8f762ad051c8 [ 7067.030259] R10: 0000000000000000 R11: 00000000ffffffff R12: 0000000000000296 [ 7067.030415] R13: 0000000000000003 R14: 0000000000000000 R15: 0000000000000000 [ 7067.030570] FS: 0000000000000000(0000) GS:ffff8f762ca00000(0000) knlGS:0000000000000000 [ 7067.030724] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7067.030853] CR2: ffff8f75cb9be538 CR3: 000000031a054000 CR4: 00000000000006b0 [ 7067.031017] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7067.031170] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 7067.031323] Call Trace: [ 7067.031382] _raw_spin_lock_irqsave+0x46/0x80 [ 7067.031491] ? __wake_up_common_lock+0x4e/0x90 [ 7067.031596] __wake_up_common_lock+0x4e/0x90 [ 7067.031716] osc_extent_put+0x7a/0x200 [osc] [ 7067.031832] osc_extent_finish+0x450/0xa90 [osc] [ 7067.031952] ? brw_interpret+0xc91/0x2c30 [osc] [ 7067.032066] brw_interpret+0x506/0x2c30 [osc] [ 7067.032232] ptlrpc_check_set+0x52d/0x3180 [ptlrpc] [ 7067.032378] ptlrpcd+0x818/0xa00 [ptlrpc] [ 7067.032465] ? do_wait_intr_irq+0x80/0x80 [ 7067.032578] ? ptlrpc_disconnect_import+0x940/0x940 [ptlrpc] [ 7067.032715] kthread+0x129/0x140 [ 7067.032797] ? kthread_flush_work_fn+0x10/0x10 [ 7067.032903] ret_from_fork+0x1f/0x30 [ 7067.032990] Modules linked in: lustre(O) ofd(O) osp(O) lod(O) ost(O) mdt(O) mdd(O) mgs(O) osd_zfs(O) lquota(O) lfsck(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) zfs(O) zunicode(O) zzstd(O) zlua(O) zcommon(O) znvpair(O) zavl(O) icp(O) spl(O) [last unloaded: obdecho] [ 7067.033536] CR2: ffff8f75cb9be538 [ 7067.033617] ---[ end trace 2e0a432d1825ff78 ]--- [ 7067.034020] RIP: 0010:do_raw_spin_trylock+0x0/0x40 [ 7067.034546] Code: 12 1e bb 86 48 89 df e8 be fe ff ff eb 8a 89 c6 48 89 df e8 12 e3 ff ff 66 90 eb 8b 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 <8b> 07 85 c0 75 0d ba 01 00 00 00 f0 0f b1 17 85 c0 74 03 31 c0 c3 [ 7067.034901] RSP: 0018:ffff8f75f5ebbb88 EFLAGS: 00010046 [ 7067.035014] RAX: 0000000000000001 RBX: ffff8f75cb9be550 RCX: 0000000000000001 [ 7067.035167] RDX: 0000000000000001 RSI: 00000000f6de3d3b RDI: ffff8f75cb9be538 [ 7067.035319] RBP: ffff8f75cb9be538 R08: ffff8f762ad05200 R09: ffff8f762ad051c8 [ 7067.035472] R10: 0000000000000000 R11: 00000000ffffffff R12: 0000000000000296 [ 7067.035625] R13: 0000000000000003 R14: 0000000000000000 R15: 0000000000000000 [ 7067.035778] FS: 0000000000000000(0000) GS:ffff8f762ca00000(0000) knlGS:0000000000000000 [ 7067.035935] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7067.036078] CR2: ffff8f75cb9be538 CR3: 000000031a054000 CR4: 00000000000006b0 [ 7067.036240] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 7067.036395] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 7067.036550] Kernel panic - not syncing: Fatal exception [ 7067.056549] Kernel Offset: 0x5000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff) [ 7067.056757] Rebooting in 60 seconds..