[LU-12022] sanity-flr: test_200 'checksum error for mirror 3'

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.13.0, Lustre 2.12.1, Lustre 2.14.0, Lustre 2.15.0
• Fix Version/s: Lustre 2.15.0
• Type: Bug
• Priority: Minor
• Reporter: Maloo
• Assignee: James Nunez (Inactive)
• Resolution: Fixed
• Severity: 3

Description

This issue was created by maloo for paf <[e-mail]>

This issue relates to the following test suite run: https://testing.whamcloud.com/test_sets/39847bcc-3985-11e9-8f69-52540065bddc

Error given is checksum error, but mirror resync just failed entirely. Test should probably be updated to catch the failure there rather than report a checksum error later:

lock to resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e resync_start' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..failed
lock to resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..failed
lock to resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..failed
lock to resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..failed
lock to resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e resync_start' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..failed
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..failed
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed
lock to resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e resync_start' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..failed
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e resync_start' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..failed
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e resync_start' ..failed
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e resync_start' ..failed
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 7585 7586 7587 7589 7590
failed
10.9.4.240@tcp:/lustre /mnt/lustre2 lustre rw,flock,user_xattr,lazystatfs 0 0
CMD: trevis-20vm1.trevis.whamcloud.com grep -c /mnt/lustre2' ' /proc/mounts
Stopping client trevis-20vm1.trevis.whamcloud.com /mnt/lustre2 (opts
CMD: trevis-20vm1.trevis.whamcloud.com lsof -t /mnt/lustre2
CMD: trevis-20vm1.trevis.whamcloud.com umount /mnt/lustre2 2>&1
10.9.4.240@tcp:/lustre /mnt/lustre3 lustre rw,flock,user_xattr,lazystatfs 0 0
CMD: trevis-20vm1.trevis.whamcloud.com grep -c /mnt/lustre3' ' /proc/mounts
Stopping client trevis-20vm1.trevis.whamcloud.com /mnt/lustre3 (opts
CMD: trevis-20vm1.trevis.whamcloud.com lsof -t /mnt/lustre3
CMD: trevis-20vm1.trevis.whamcloud.com umount /mnt/lustre3 2>&1
mirror_io: 524: llapi_mirror_copy_many
/mnt/lustre/f200.sanity-flr: found 10 stale components
/mnt/lustre/f200.sanity-flr: resyncing mirror: 1, components: 65537 65538 65539 65540 65541
3
sanity-flr test_200: @@@@@@ FAIL: checksum error for mirror 3
Trace dump:
= /usr/lib64/lustre/tests/test-framework.sh:5838:error()
= /usr/lib64/lustre/tests/sanity-flr.sh:2189:test_200()
= /usr/lib64/lustre/tests/test-framework.sh:6119:run_one()
= /usr/lib64/lustre/tests/test-framework.sh:6158:run_one_logged()
= /usr/lib64/lustre/tests/test-framework.sh:6005:run_test()
= /usr/lib64/lustre/tests/sanity-flr.sh:2194:main()
Dumping lctl log to /autotest/trevis/2019-02-26/lustre-reviews-el7_6-x86_64-review-zfs-1_17_1_62058__69de2681-ac9c-46f6-a357-cca06225620a/sanity-flr.test_200.*.1551150323.log
CMD: trevis-20vm1.trevis.whamcloud.com,trevis-20vm2,trevis-20vm3,trevis-20vm4 /usr/sbin/lctl dk > /autotest/trevis/2019-02-26/lustre-reviews-el7_6-x86_64-review-zfs-1_17_1_62058__69de2681-ac9c-46f6-a357-cca06225620a/sanity-flr.test_200.debug_log.\$(hostname -s).1551150323.log;
dmesg > /autotest/trevis/2019-02-26/lustre-reviews-el7_6-x86_64-review-zfs-1_17_1_62058__69de2681-ac9c-46f6-a357-cca06225620a/sanity-flr.test_200.dmesg.\$(hostname -s).1551150323.log
Resetting fail_loc on all nodes...CMD: trevis-20vm1.trevis.whamcloud.com,trevis-20vm2,trevis-20vm3,trevis-20vm4 lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null
done.

Issue Links

• is related to: LU-11226 sanity-flr test 200 fails with 'check...

Comments

[Minh Diep]

+1 on b2_12: https://testing.whamcloud.com/test_sets/1b46d18c-552d-11e9-9720-52540065bddc

[James Nunez (Inactive)]

Unfortunately, resync can fail and the test is still marked as PASS. I'll create a patch to fail and exit when we encounter a failed resync.

See https://testing.whamcloud.com/sub_tests/2a53cf5e-b65d-11e9-b753-52540065bddc for sanity-flr test 200 marked as PASS, but all resync tests print "failed"

== sanity-flr test 200: stress test ================================================================== 07:24:23 (1564842263)
Starting client: trevis-19vm1:  -o user_xattr,flock trevis-19vm4@tcp:/lustre /mnt/lustre2
CMD: trevis-19vm1 mkdir -p /mnt/lustre2
CMD: trevis-19vm1 mount -t lustre -o user_xattr,flock trevis-19vm4@tcp:/lustre /mnt/lustre2
Starting client: trevis-19vm1:  -o user_xattr,flock trevis-19vm4@tcp:/lustre /mnt/lustre3
CMD: trevis-19vm1 mkdir -p /mnt/lustre3
CMD: trevis-19vm1 mount -t lustre -o user_xattr,flock trevis-19vm4@tcp:/lustre /mnt/lustre3
fail_loc=0x1A03
CMD: trevis-19vm4 /usr/sbin/lctl set_param fail_loc=0x1A03
fail_loc=0x1A03
lock to resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 4434464 ..
Extending file size to 8121536 ..
failed
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed
Extending file size to 8432928 ..
resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..Extending file size to 8678144 ..
failed
resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..failed
Extending file size to 8785152 ..
resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..failed
lock to resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..failed
lock to resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..failed
lock to resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..failed
lock to resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..done
lock to resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..done
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..failed
lock to resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..failed
lock to resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..failed
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e resync_start' ..failed
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e resync_start' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..failed
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e resync_start' ..failed
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e resync_start' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..failed
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..done
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..failed
lock to resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..failed
lock to resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..failed
lock to resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..done
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e resync_start' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..failed
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e resync_start' ..failed
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e resync_start' ..failed
lock to resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..done
lock to resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..failed
Extending file size to 9138848 ..
lock to resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..failed
lock to resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e resync_start' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..failed
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed
lock to resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..failed
lock to resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e resync_start' ..failed
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed
lock to resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed
lock to resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..done
resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..failed
lock to resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..done
resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..Extending file size to 8838944 ..
done
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e resync_start' ..failed
lock to resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..failed
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..failed
lock to resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..done
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed
lock to resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e resync_start' ..failed
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..done
lock to resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..failed
lock to resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e resync_start' ..failed
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed
lock to resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..failed
lock to resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..failed
lock to resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed
lock to resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed
lock to resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..done
resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..failed
Extending file size to 8901856 ..
resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..failed
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e resync_start' ..failed
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e resync_start' ..failed
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e resync_start' ..failed
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e resync_start' ..failed
Extending file size to 9033472 ..
lock to resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..failed
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e resync_start' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..failed
lock to resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..failed
lock to resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..done
lock to resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..Extending file size to 9341920 ..
failed
lock to resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..done
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e resync_start' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..failed
lock to resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e resync_start' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..failed
lock to resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..failed
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e resync_start' ..failed
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e resync_start' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..done
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e resync_start' ..failed
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..done
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e resync_start' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..failed
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e resync_start' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..failed
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e resync_start' ..failed
lock to resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..done
resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..failed
resync file /mnt/lustre3/f200.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..failed
resync file /mnt/lustre3/f200.sanity-flr with '/usr/bin/lfs mirror resync' ..done

[Gerrit Updater]

James Nunez ([e-mail]) uploaded a new patch: https://review.whamcloud.com/35754
Subject: LU-12022 tests: error on resync failure sanity-flr
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 6d13e6423c25fd1ca87b716b8d83c58a7844a39a

[James Nunez (Inactive)]

Note: The patch https://review.whamcloud.com/35754 does not fix the issue as to why sanity-flr is failing. The patch is to make it very clear when the test fails.

[Andreas Dilger]

+1 for master https://testing.whamcloud.com/test_sets/2d0931a5-bdef-4565-8ba6-ed24fd0392e9

[Andreas Dilger]

James, I don't think that the resync errors should actually be considered test failures. If the file changes while the resync is happening, then the resync would be aborted and need to be done again. That's just how FLR currently is implemented.

However, the resync at the end of the test (after the write threads have been stopped) should properly resync the stale mirrors. It isn't clear why this test is still using "mirror_io resync" instead of "lfs mirror resync", since the latter is the tool that is used in production and is the tool we care is working properly. The mirror_io tool was a temporary FLR development tool, and its use and code should probably be removed (I am not aware of any functionality it has that is not available via "lfs mirror", but if there is we should consider moving it over.

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/35754/
Subject: LU-12022 tests: error on resync failure sanity-flr
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 9e1e29406c3886bce0db6623be40d3edf6834117

[Peter Jones]

Landed for 2.15

[Sebastien Buisson]

Seen again here:
https://testing.whamcloud.com/test_sets/2b8f4210-cc82-42f0-9e74-7eee7a09796d

[Patrick Farrell]

+1
https://testing.whamcloud.com/test_sets/6c5a6023-bd9d-4e05-9631-075b8dc6db58