[LU-12715] sanity-pcc tests 1, 2, 3a, 5 fail with “pcc attach /mnt/lustre/d1a.sanity-pcc/* failed “

Tue, 20 Oct 2020 14:47:51 +0000

sanity-pcc tests 1a, 1b, 1e, 1g, 2b, 2c, 3a, and 5 fail then test 6 hangs for PPC clients. These tests fail consistently for PPC clients.

Looking at the failures at https://testing.whamcloud.com/test_sets/c2ff9d1e-ca0d-11e9-a2b6-52540065bddc , we see the following errors in the suite_log

== sanity-pcc test 1a: Test manual lfs pcc attach with manual HSM restore ============================ 02:08:02 (1567044482)
…
CMD: trevis-77vm8 /usr/bin/lfs pcc attach -i 2 /mnt/lustre/d1a.sanity-pcc/f1a.sanity-pcc
trevis-77vm8: lfs pcc pcc: cannot get WRITE lease, ext 0: Device or resource busy (16)
trevis-77vm8: lfs pcc pcc: cannot get lease: Device or resource busy (16)
trevis-77vm8: attach: cannot attach '/mnt/lustre/d1a.sanity-pcc/f1a.sanity-pcc' to PCC with archive ID '2': Device or resource busy
 sanity-pcc test_1a: @@@@@@ FAIL: pcc attach /mnt/lustre/d1a.sanity-pcc/f1a.sanity-pcc failed 
…
== sanity-pcc test 1b: Test manual lfs pcc attach with restore on remote access ====================== 02:08:07 (1567044487)
…
CMD: trevis-77vm8 /usr/bin/lfs pcc attach -i 2 /mnt/lustre/d1b.sanity-pcc/f1b.sanity-pcc
trevis-77vm8: lfs pcc pcc: cannot get WRITE lease, ext 0: Device or resource busy (16)
trevis-77vm8: lfs pcc pcc: cannot get lease: Device or resource busy (16)
trevis-77vm8: attach: cannot attach '/mnt/lustre/d1b.sanity-pcc/f1b.sanity-pcc' to PCC with archive ID '2': Device or resource busy
 sanity-pcc test_1b: @@@@@@ FAIL: pcc attach /mnt/lustre/d1b.sanity-pcc/f1b.sanity-pcc failed 
…
== sanity-pcc test 1e: Test RW-PCC with non-root user ================================================ 02:11:44 (1567044704)
…
trevis-77vm8:  [/usr/bin/lfs] [pcc] [attach] [-i] [2] [/mnt/lustre/d1e.sanity-pcc/f1e.sanity-pcc]
trevis-77vm8: lfs pcc pcc: cannot get WRITE lease, ext 0: Device or resource busy (16)
trevis-77vm8: lfs pcc pcc: cannot get lease: Device or resource busy (16)
trevis-77vm8: attach: cannot attach '/mnt/lustre/d1e.sanity-pcc/f1e.sanity-pcc' to PCC with archive ID '2': Device or resource busy
 sanity-pcc test_1e: @@@@@@ FAIL: failed to attach file /mnt/lustre/d1e.sanity-pcc/f1e.sanity-pcc 
…
== sanity-pcc test 1f: Test auto RW-PCC cache with non-root user ===================================== 02:11:48 (1567044708)
…
== sanity-pcc test 1g: General permission test for RW-PCC ============================================ 02:13:26 (1567044806)
…
CMD: trevis-77vm8 /usr/bin/lfs pcc attach -i 2 /mnt/lustre/f1g.sanity-pcc
trevis-77vm8: lfs pcc pcc: cannot get WRITE lease, ext 0: Device or resource busy (16)
trevis-77vm8: lfs pcc pcc: cannot get lease: Device or resource busy (16)
trevis-77vm8: attach: cannot attach '/mnt/lustre/f1g.sanity-pcc' to PCC with archive ID '2': Device or resource busy
 sanity-pcc test_1g: @@@@@@ FAIL: failed to attach file /mnt/lustre/f1g.sanity-pcc 
…
== sanity-pcc test 2b: Test multi remote open when creating ========================================== 02:16:15 (1567044975)
…
CMD: trevis-77vm8 lfs pcc attach -i 2 /mnt/lustre/d2b.sanity-pcc/multiop
trevis-77vm8: lfs pcc pcc: cannot get WRITE lease, ext 0: Device or resource busy (16)
trevis-77vm8: lfs pcc pcc: cannot get lease: Device or resource busy (16)
trevis-77vm8: attach: cannot attach '/mnt/lustre/d2b.sanity-pcc/multiop' to PCC with archive ID '2': Device or resource busy
 sanity-pcc test_2b: @@@@@@ FAIL: PCC attach /mnt/lustre/d2b.sanity-pcc/multiop failed 
…
== sanity-pcc test 2c: Test multi open on different mount points when creating ======================= 02:17:57 (1567045077)
…
CMD: trevis-77vm8 lfs pcc attach -i 2 /mnt/lustre/d2c.sanity-pcc/f2c.sanity-pcc
trevis-77vm8: lfs pcc pcc: cannot get WRITE lease, ext 0: Device or resource busy (16)
trevis-77vm8: lfs pcc pcc: cannot get lease: Device or resource busy (16)
trevis-77vm8: attach: cannot attach '/mnt/lustre/d2c.sanity-pcc/f2c.sanity-pcc' to PCC with archive ID '2': Device or resource busy
 sanity-pcc test_2c: @@@@@@ FAIL: PCC attach /mnt/lustre/d2c.sanity-pcc/f2c.sanity-pcc failed 
…
== sanity-pcc test 3a: Repeat attach/detach operations =============================================== 02:19:35 (1567045175)
…
CMD: trevis-77vm8 /usr/bin/lfs pcc attach -i 2 /mnt/lustre/d3a.sanity-pcc/f3a.sanity-pcc
trevis-77vm8: lfs pcc pcc: cannot get WRITE lease, ext 0: Device or resource busy (16)
trevis-77vm8: lfs pcc pcc: cannot get lease: Device or resource busy (16)
trevis-77vm8: attach: cannot attach '/mnt/lustre/d3a.sanity-pcc/f3a.sanity-pcc' to PCC with archive ID '2': Device or resource busy
 sanity-pcc test_3a: @@@@@@ FAIL: failed to attach file /mnt/lustre/d3a.sanity-pcc/f3a.sanity-pcc 
…
== sanity-pcc test 5: Mmap & cat a RW-PCC cached file ================================================ 02:21:58 (1567045318)
…
CMD: trevis-77vm8 /usr/bin/lfs pcc attach -i 2 /mnt/lustre/f5.sanity-pcc
trevis-77vm8: lfs pcc pcc: cannot get WRITE lease, ext 0: Device or resource busy (16)
trevis-77vm8: lfs pcc pcc: cannot get lease: Device or resource busy (16)
trevis-77vm8: attach: cannot attach '/mnt/lustre/f5.sanity-pcc' to PCC with archive ID '2': Device or resource busy
 sanity-pcc test_5: @@@@@@ FAIL: failed to attach file /mnt/lustre/f5.sanity-pcc 
…

Here are links for a few recent failures
https://testing.whamcloud.com/test_sets/c2ff9d1e-ca0d-11e9-a2b6-52540065bddc
https://testing.whamcloud.com/test_sets/1be69952-c73c-11e9-97d5-52540065bddc

Whamcloud Community JIRA

[LU-12715] sanity-pcc tests 1*, 2*, 3a, 5 fail with “pcc attach /mnt/lustre/d1a.sanity-pcc/* failed “

[LU-12715] sanity-pcc tests 1, 2, 3a, 5 fail with “pcc attach /mnt/lustre/d1a.sanity-pcc/* failed “