Formatting '/tmp/tmp.2tiruzJKuI/root.img', fmt=qcow2 cluster_size=4096 extended_l2=off compression_type=zlib size=785833984 backing_file=/var/images/centos8.img backing_fmt=raw lazy_refcounts=off refcount_bits=16 RAM: 12346, ENV: VM ready in 4 Reading test skip list from /tmp/ltest.config EXCEPT="$EXCEPT " tmp.2tiruzJKuI: executing check_logdir /tmp/ltest-logs tmp.2tiruzJKuI: Reading test skip list from /tmp/ltest.config tmp.2tiruzJKuI: EXCEPT="$EXCEPT " Logging to shared log directory: /tmp/ltest-logs tmp.2tiruzJKuI: executing yml_node Writer error: failed to resolve Netlink family id IOC_LIBCFS_GET_NI error 22: Invalid argument Client: 2.15.54 MDS: 2.15.54 OSS: 2.15.54 excepting tests: 6 201 44c Stopping clients: tmp.2tiruzJKuI /mnt/lustre (opts:-f) Stopping clients: tmp.2tiruzJKuI /mnt/lustre2 (opts:-f) tmp.2tiruzJKuI: executing set_hostid tmp.2tiruzJKuI: Reading test skip list from /tmp/ltest.config tmp.2tiruzJKuI: EXCEPT="$EXCEPT " Loading modules from /mnt/build/lustre/tests/.. detected 2 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' gss/krb5 is not supported quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /tmp/lustre-mdt1 Format ost1: /tmp/lustre-ost1 Format ost2: /tmp/lustre-ost2 Format ost3: /tmp/lustre-ost3 Format ost4: /tmp/lustre-ost4 Checking servers environments Checking clients tmp.2tiruzJKuI environments Loading modules from /mnt/build/lustre/tests/.. detected 2 online CPUs by sysfs Force libcfs to create 2 CPU partitions gss/krb5 is not supported Setup mgs, mdt, osts Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 Commit the device label on /tmp/lustre-mdt1 Started lustre-MDT0000 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 Commit the device label on /tmp/lustre-ost1 Started lustre-OST0000 Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 Commit the device label on /tmp/lustre-ost2 Started lustre-OST0001 Starting ost3: -o localrecov /dev/mapper/ost3_flakey /mnt/lustre-ost3 Commit the device label on /tmp/lustre-ost3 Started lustre-OST0002 Starting ost4: -o localrecov /dev/mapper/ost4_flakey /mnt/lustre-ost4 Commit the device label on /tmp/lustre-ost4 Started lustre-OST0003 Starting client: tmp.2tiruzJKuI: -o user_xattr,flock tmp.2tiruzJKuI@tcp:/lustre /mnt/lustre Starting client tmp.2tiruzJKuI: -o user_xattr,flock tmp.2tiruzJKuI@tcp:/lustre /mnt/lustre Started clients tmp.2tiruzJKuI: 192.168.125.181@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff99c51b438000.idle_timeout=debug osc.lustre-OST0001-osc-ffff99c51b438000.idle_timeout=debug osc.lustre-OST0002-osc-ffff99c51b438000.idle_timeout=debug osc.lustre-OST0003-osc-ffff99c51b438000.idle_timeout=debug setting jobstats to procname_uid Setting lustre.sys.jobid_var from disable to procname_uid Waiting 90s for 'procname_uid' Updated after 8s: want 'procname_uid' got 'procname_uid' disable quota as required osd-ldiskfs.track_declares_assert=1 running as uid/gid/euid/egid 500/500/500/500, groups: [true] running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d0_runas_test/f1523] == sanity-flr test 42: lfs mirror verify ================= 14:49:47 (1679582987) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.183182 s, 57.2 MB/s 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.146245 s, 71.7 MB/s 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.123803 s, 84.7 MB/s 3 lfs mirror verify: '/mnt/lustre/d42.sanity-flr/f42.sanity-flr-1' invalid specified mirror id: 65534. lfs mirror verify: '/mnt/lustre/d42.sanity-flr/f42.sanity-flr-1' invalid specified mirror id: 65535. 0+1 records in 0+1 records out 2 bytes copied, 0.000765987 s, 2.6 kB/s 0+1 records in 0+1 records out 2 bytes copied, 0.00720929 s, 0.3 kB/s 0+1 records in 0+1 records out 2 bytes copied, 0.0489536 s, 0.0 kB/s 0+1 records in 0+1 records out 2 bytes copied, 0.00781313 s, 0.3 kB/s Verify /mnt/lustre/d42.sanity-flr/f42.sanity-flr-1 without -v option: lfs mirror verify: chunk [0x400000, 0x600000) has different checksum value on mirror 1 and mirror 2. lfs mirror verify: chunk [0x400000, 0x600000) has different checksum value on mirror 1 and mirror 4. PASS Verify /mnt/lustre/d42.sanity-flr/f42.sanity-flr-1 with -v option: CRC-32 checksum value for chunk [0, 0x400000): Mirror 1: 0x207b02f1 Mirror 2: 0x207b02f1 Mirror 3: 0x207b02f1 Mirror 4: 0x207b02f1 CRC-32 checksum value for chunk [0x200000, 0x600000): Mirror 1: 0x207b02f1 Mirror 2: 0x207b02f1 Mirror 3: 0x207b02f1 Mirror 4: 0x207b02f1 CRC-32 checksum value for chunk [0x400000, 0x800000): Mirror 1: 0x207b02f1 Mirror 2: 0x42571b66 Mirror 3: 0x207b02f1 Mirror 4: 0xabdaf92 lfs mirror verify: chunk [0x400000, 0x600000) has different checksum value on mirror 1 and mirror 2. lfs mirror verify: chunk [0x400000, 0x600000) has different checksum value on mirror 1 and mirror 4. CRC-32 checksum value for chunk [0x600000, 0xa00000): Mirror 1: 0x207b02f1 Mirror 2: 0x1f8ad0d8 Mirror 3: 0x207b02f1 Mirror 4: 0x18975bf9 lfs mirror verify: chunk [0x600000, 0x800000) has different checksum value on mirror 1 and mirror 2. lfs mirror verify: chunk [0x600000, 0x800000) has different checksum value on mirror 1 and mirror 4. CRC-32 checksum value for chunk [0x800000, 0xc00000): Mirror 1: 0xf1573de5 Mirror 2: 0x4649ddde Mirror 3: 0xf1573de5 Mirror 4: 0x6d648e1d lfs mirror verify: chunk [0x800000, 0xa00000) has different checksum value on mirror 1 and mirror 2. lfs mirror verify: chunk [0x800000, 0xa00000) has different checksum value on mirror 1 and mirror 4. CRC-32 checksum value for chunk [0xa00000, 0xe00000): Mirror 1: 0x41d912ff Mirror 2: 0xf6c7f2c4 Mirror 3: 0x41d912ff Mirror 4: 0xddeaa107 lfs mirror verify: chunk [0xa00000, 0x1000000) has different checksum value on mirror 1 and mirror 2. lfs mirror verify: chunk [0xa00000, 0x1000000) has different checksum value on mirror 1 and mirror 4. lfs mirror verify: '/mnt/lustre/d42.sanity-flr/f42.sanity-flr-1' chunk [0x1000000, 0xffffffffffffffff) exceeds file size 0xa00002: skipped 4 Verify /mnt/lustre/d42.sanity-flr/f42.sanity-flr-1 with --only option: CRC-32 checksum value for chunk [0, 0x400000): Mirror 2: 0x207b02f1 Mirror 4: 0x207b02f1 CRC-32 checksum value for chunk [0x200000, 0x600000): Mirror 2: 0x207b02f1 Mirror 4: 0x207b02f1 CRC-32 checksum value for chunk [0x400000, 0x800000): Mirror 2: 0x42571b66 Mirror 4: 0xabdaf92 lfs mirror verify: chunk [0x400000, 0x600000) has different checksum value on mirror 2 and mirror 4. CRC-32 checksum value for chunk [0x600000, 0xa00000): Mirror 2: 0x1f8ad0d8 Mirror 4: 0x18975bf9 lfs mirror verify: chunk [0x600000, 0x800000) has different checksum value on mirror 2 and mirror 4. CRC-32 checksum value for chunk [0x800000, 0xc00000): Mirror 2: 0x4649ddde Mirror 4: 0x6d648e1d lfs mirror verify: chunk [0x800000, 0xa00000) has different checksum value on mirror 2 and mirror 4. CRC-32 checksum value for chunk [0xa00000, 0xe00000): Mirror 2: 0xf6c7f2c4 Mirror 4: 0xddeaa107 lfs mirror verify: chunk [0xa00000, 0x1000000) has different checksum value on mirror 2 and mirror 4. lfs mirror verify: '/mnt/lustre/d42.sanity-flr/f42.sanity-flr-1' chunk [0x1000000, 0xffffffffffffffff) exceeds file size 0xa00002: skipped lfs mirror verify: chunk [0x400000, 0x600000) has different checksum value on mirror 1 and mirror 2. sanity-flr test_42: @@@@@@ FAIL: verify /mnt/lustre/d42.sanity-flr/f42.sanity-flr-1 with mirror 1 and 2 should succeed Trace dump: = ./../tests/test-framework.sh:6575:error() = sanity-flr.sh:2023:test_42() = ./../tests/test-framework.sh:6913:run_one() = ./../tests/test-framework.sh:6963:run_one_logged() = ./../tests/test-framework.sh:6784:run_test() = sanity-flr.sh:2042:main() Dumping lctl log to /tmp/ltest-logs/sanity-flr.test_42.*.1679582991.log Dumping logs only on local client. FAIL 42 (17s) LTEST: ssh exited 1