[LU-14358] interop: sanity-pcc and sanity-flr tests fail with ‘cannot open volatile file’on the MDS

Fri, 5 Feb 2021 13:30:44 +0000

A variety of sanity-pcc tests fail with different error messages, but all have the following in the MDS console log

[67028.845779] LustreError: 3916:0:(mdt_open.c:1613:mdt_reint_open()) lustre-MDT0000: cannot open volatile file [0x2000766a8:0x4:0x0], orphan file will be left in PENDING directory until next reboot, rc = -2

and the action on that file does not complete.

This issue has only been seen in interop testing for 2.13.0 servers and master clients with Lustre version>= 2.13.55.16. It looks like this error message and failures started on 08 AUG 2020 with failures in sanity-pcc and sanity-flr in the test session https://testing.whamcloud.com/test_sessions/7175020e-210c-436a-a2a6-be91c6c12aad

A few examples of this failure are:
2021-01-16 Lustre server 2.13.0 and Lustre client 2.13.57.53 - https://testing.whamcloud.com/test_sets/c1c698e2-e40e-4035-a3bb-de8acfacc5b5
sanity-pcc test_1a fails with 'request on 0x2000766a8:0x2:0x0 is not SUCCEED on mds1'
sanity-pcc test_1e fails with ‘failed to attach file /mnt/lustre/d1e.sanity-pcc/f1e.sanity-pcc’
sanity-pcc test_3a fails with ‘failed to attach file \/mnt\/lustre\/d3a.sanity-pcc\/f3a.sanity-pcc ‘

2021-01-07 Lustre server 2.13.0 and Lustre client 2.13.57.44 - https://testing.whamcloud.com/test_sets/c445c494-4d76-4185-9554-f5eb131d5b03
2021-12-25 Lustre server 2.13.0 and Lustre client 2.13.57.36 - https://testing.whamcloud.com/test_sets/e26503ed-449e-4070-84e3-8af9d129fad9
2021-12-10 Lustre server 2.13.0 and Lustre client 2.13.57 - https://testing.whamcloud.com/test_sets/9e2b1a2a-1d6a-4f56-b375-cd177c058eb0
sanity-pfl test 6 fails with 'Migrate(v1 -> composite) /mnt/lustre/d6.sanity-pfl/f6.sanity-pfl failed'

2021-12-25 Lustre server 2.13.0 and Lustre client 2.13.57.12 - https://testing.whamcloud.com/test_sets/3f774e46-eb0f-4b4b-90e5-a70a48ae9c56
sanity-pcc test_1c fails with 'request on 0x200076e71:0x2:0x0 is not SUCCEED on mds1'
sanity-pcc test 7b fails with 'multiop mmap write failed'
sanity-pcc test 12 fails with 'request on 0x200079d51:0x14:0x0 is not SUCCEED on mds1'
sanity-pcc test 19 fails with 'Failed to attach /mnt/lustre/f19.sanity-pcc'

We do see this error message occasionally in ost-pools test 28 and the test passes:
2021-12-25 Lustre server 2.13.0 and Lustre client 2.13.57.36 - https://testing.whamcloud.com/test_sets/fe713439-9aa7-4a71-beb0-b1b715fca596
2021-12-10 Lustre server 2.13.0 and Lustre client 2.13.57 - https://testing.whamcloud.com/test_sets/f48d3beb-c6d3-4451-8dc1-dcbe7dfe0b14

Whamcloud Community JIRA

[LU-14358] interop: sanity-pcc and sanity-flr tests fail with ‘cannot open volatile file’on the MDS