[LU-13017] sanity test 56ra fails with ' '/usr/bin/lfs find -size 0 -type f /mnt/lustre/d56ra.sanity' should send 12 glimpse RPCs to OST'

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.14.0
• Affects Version/s: Lustre 2.14.0, Lustre 2.12.4
• Severity: 3
• Assignee: Andreas Dilger
• Reporter: James Nunez (Inactive)

Description

sanity test_56ra fails with ' '/usr/bin/lfs find -size 0 -type f /mnt/lustre/d56ra.sanity' should send 12 glimpse RPCs to OST'. We’ve seen this error twice for 2.12.3.31; https://testing.whamcloud.com/test_sets/93cddba8-0d76-11ea-98f1-52540065bddc and https://testing.whamcloud.com/test_sets/3d7c7eea-0d93-11ea-8e77-52540065bddc.

== sanity test 56ra: check lfs find -size -lazy works for data on OSTs =============================== 10:36:52 (1574332612)
Before: 6056 After: 6069 3
osc.lustre-OST0000-osc-ffff8c52a0a6c000.stats=
snapshot_time             1574332613.130677340 secs.nsecs
req_waittime              3629 samples [usec] 405 4908409 125708950 217095123014804
req_active                3630 samples [reqs] 1 16 10917 64837
ldlm_glimpse_enqueue      453 samples [reqs] 1 1 453 453
ldlm_extent_enqueue       518 samples [reqs] 1 1 518 518
read_bytes                10 samples [bytes] 0 1048576 4198400 4380984082432
write_bytes               729 samples [bytes] 4 1048576 465385120 468655951772936
ost_setattr               30 samples [usec] 493 2458360 5470309 7460871612619
ost_read                  10 samples [usec] 770 112506 334235 27947789183
ost_write                 729 samples [usec] 814 4301470 93802565 175458110296977
ost_connect               29 samples [usec] 655 12195 57083 265453679
ost_disconnect            17 samples [usec] 1029 3311 31912 67557322
ost_punch                 511 samples [usec] 503 954851 1266293 911962877985
ost_statfs                24 samples [usec] 405 1646 17727 14510469
ost_sync                  4 samples [usec] 471514 1920497 3598304 4653850793070
ost_quotactl              16 samples [usec] 414 838 8192 4340976
ldlm_cancel               952 samples [usec] 533 39037 11498758 185050546552
obd_ping                  336 samples [usec] 760 4908409 8456427 28320522807435
osc.lustre-OST0001-osc-ffff8c52a0a6c000.stats=
…
obd_ping                  364 samples [usec] 735 1157343 2386059 1579049113871
osc.lustre-OST0006-osc-ffff8c52a0a6c000.stats=
snapshot_time             1574332613.130931617 secs.nsecs
req_waittime              2363 samples [usec] 368 10241895 31650509 113679647081289
req_active                2364 samples [reqs] 1 15 6902 40110
ldlm_glimpse_enqueue      928 samples [reqs] 1 1 928 928
ldlm_extent_enqueue       14 samples [reqs] 1 1 14 14
read_bytes                2 samples [bytes] 0 0 0 0
write_bytes               19 samples [bytes] 1 1048576 12837273 13249571414835
ost_setattr               31 samples [usec] 459 10241895 14509301 109946640240861
ost_read                  2 samples [usec] 1823 2294 4117 8585765
ost_write                 19 samples [usec] 814 543145 1723935 429765970407
ost_connect               27 samples [usec] 450 12275 51577 247601705
ost_disconnect            14 samples [usec] 999 41957 63004 1800765226
ost_punch                 9 samples [usec] 622 73011 79728 5336508958
ost_statfs                24 samples [usec] 368 2161 14597 12673415
ost_quotactl              16 samples [usec] 388 462 6778 2878618
ldlm_cancel               927 samples [usec] 411 38371 11153645 173370727119
obd_ping                  352 samples [usec] 759 1625436 2970385 3116552824601
 sanity test_56ra: @@@@@@ FAIL: '/usr/bin/lfs find -size 0 -type f /mnt/lustre/d56ra.sanity' should send 12 glimpse RPCs to OST 

We’ve seen an error message that may be related for Ubuntu clients ' '/usr/bin/lfs find -size 5 -type f /mnt/lustre/d56ra.sanity' should send 14 glimpse RPCs to OST'; https://testing.whamcloud.com/test_sets/138a655c-0d5c-11ea-9487-52540065bddc for patch https://review.whamcloud.com/36405/. From the suite_log

== sanity test 56ra: check lfs find -size -lazy works for data on OSTs =============================== 17:50:11 (1574445011)
Before: 4861 After: 4873 3
osc.lustre-OST0000-osc-ffff9a29368bb000.stats=
snapshot_time             1574445012.373146631 secs.nsecs
req_waittime              2983 samples [usec] 355 2937458 64237903 59715964412813
req_active                2983 samples [reqs] 1 13 11657 81779
ldlm_glimpse_enqueue      277 samples [reqs] 1 1 277 277
ldlm_extent_enqueue       515 samples [reqs] 1 1 515 515
read_bytes                2 samples [bytes] 4096 4096 8192 33554432
write_bytes               569 samples [bytes] 3 4194304 498256074 1247211946247316
ost_setattr               33 samples [usec] 363 608888 739594 375188097832
ost_read                  2 samples [usec] 455 466 921 424181
ost_write                 569 samples [usec] 554 2937458 49108763 59029078413847
ost_connect               13 samples [usec] 464 6183 24328 96647710
ost_disconnect            9 samples [usec] 755 1623 11362 14999198
ost_punch                 511 samples [usec] 355 43985 346442 4026093074
ost_statfs                12 samples [usec] 357 1074 5859 3259503
ldlm_cancel               774 samples [usec] 390 63347 12515513 274346845595
obd_ping                  267 samples [usec] 502 117510 544749 18808100599
osc.lustre-OST0001-osc-ffff9a29368bb000.stats=
…
obd_ping                  265 samples [usec] 472 4485131 4896837 20121333347081
mdc.lustre-MDT0000-mdc-ffff9a29368bb000.stats=
snapshot_time             1574445012.413685221 secs.nsecs
req_waittime              938787 samples [usec] 305 1810374 538301391 13512208660843
req_active                938787 samples [reqs] 1 9 961715 1042401
ldlm_glimpse_enqueue      120 samples [reqs] 1 1 120 120
ldlm_ibits_enqueue        429407 samples [reqs] 1 1 429407 429407
mds_getattr               124 samples [usec] 337 1111 47863 19161519
mds_getattr_lock          5898 samples [usec] 470 7696 3084370 1885670342
mds_close                 116757 samples [usec] 305 892872 46303939 1088509049017
mds_readpage              352 samples [usec] 734 33119 1049972 8758729676
mds_statfs                14 samples [usec] 355 866 6123 2898905
mds_sync                  6 samples [usec] 1153 40361 46512 1636587052
mds_getxattr              28 samples [usec] 339 2024 26071 30757607
mds_hsm_state_set         8 samples [usec] 1150 1358 10021 12585899
ldlm_cancel               4370 samples [usec] 331 105790 2428541 42683139111
obd_ping                  204 samples [usec] 446 3724 188084 212073474
seq_query                 1 samples [usec] 472 472 472 222784
Before: 4873 After: 4885 3
Before: 4885 After: 4900 3
 sanity test_56ra: @@@@@@ FAIL: '/usr/bin/lfs find -size 5 -type f /mnt/lustre/d56ra.sanity' should send 14 glimpse RPCs to OST 

We’ve also seen this test fail in interop testing.

There is nothing interesting in the node console logs.

Issue Links

• is duplicated by: LU-13378 sanity test_56ra: should send 14 glimpse RPCs to OST
• is related to: LU-12378 sanity-quota test 1 fails with 'project quota isn't released after deletion'
• is related to: LU-13148 Interop: sanity test 56ra fails with ''/usr/bin/lfs find -size 0 -type f -lazy /mnt/lustre/d56ra.sanity' should not send glimpse RPCs to OST'