[LU-12851] sanity-scrub test 1a fails with '(9) Expect 'inconsistent', but got '''

• Status: Open
• Project: Lustre
• Affects Version/s: Lustre 2.13.0, Lustre 2.12.4
• Type: Bug
• Priority: Minor
• Reporter: James Nunez (Inactive)
• Assignee: WC Triage
• Resolution: Unresolved
• Severity: 3

Description

sanity-scrub test 1a fails with '(9) Expect 'inconsistent', but got '''

Looking at the failure at https://testing.whamcloud.com/test_sets/66b01e60-e6c0-11e9-be86-52540065bddc, the client test_log looks the same as for successful run of this test except for the error message

CMD: trevis-20vm5 /usr/sbin/lctl get_param -n osd-*.lustre-MDT0000.oi_scrub
 sanity-scrub test_1a: @@@@@@ FAIL: (9) Expect 'inconsistent', but got '' 
  Trace dump:
  = /usr/lib64/lustre/tests/test-framework.sh:6106:error()
  = /usr/lib64/lustre/tests/sanity-scrub.sh:382:test_1a()

On console log for the MDS (vm5), we see the errors

[ 9398.918290] Lustre: DEBUG MARKER: umount /mnt/lustre-mds1
[ 9398.938866] LustreError: 9707:0:(qsd_reint.c:56:qsd_reint_completion()) lustre-MDT0000: failed to enqueue global quota lock, glb fid:[0x200000006:0x1010000:0x0], rc:-5
[ 9398.938868] LustreError: 9706:0:(qsd_reint.c:56:qsd_reint_completion()) lustre-MDT0000: failed to enqueue global quota lock, glb fid:[0x200000006:0x10000:0x0], rc:-5
[ 9398.938870] LustreError: 9707:0:(qsd_reint.c:56:qsd_reint_completion()) Skipped 3 previous similar messages
[ 9405.175755] Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&

I don’t see these errors on console logs for MDSs that pass this test.

This test has failed with this error ten times this year. Seven of those failures were for patch https://review.whamcloud.com/#/c/33916/ for LU-11803 which does touch osd_scrub.c. One was for patch https://review.whamcloud.com/#/c/34689/ for LU-11838 and also touches the osd_scrub.c code.

Comments

[Andreas Dilger]

It looks like this hasn't been hit since 2020-01-28, but started to be seen again on 2021-01-14, and has been seen 10 times in the past week.