[LU-14440] interop: pjdfstest test utimensat_06 and test_utimensat_07 fail with '/usr/share/pjdfstest/utimensat/0*.t against lustre failed'

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.14.0
• Fix Version/s: Lustre 2.16.0
• Type: Bug
• Priority: Minor
• Reporter: James Nunez (Inactive)
• Assignee: Alena Nikitenko
• Resolution: Won't Do
• Labels: interop
• Severity: 3

Description

pjdfstest test_utimensat_06 and test_utimensat_07 started failing on 04 FEB 2021 for Interop testing between Lustre master, 2.14.0 RC1 and RC2, servers and 2.12.6 clients. There have been no changes to the pjdfstest suite for several months on these branches.

Looking at the suite_log from the failure at https://testing.whamcloud.com/test_sets/6de33360-e585-4818-a69b-0f6ccd14ab9a, we see

== run_pjdfstest test utimensat_06: utimensat with UTIME_NOW will work if the caller has write permission ====================================================================================================== 02:59:08 (1612925948)
Run /usr/share/pjdfstest/utimensat/06.t against ext4 filesystem
prove -f /usr/share/pjdfstest/utimensat/06.t &> /tmp/pjdfstest-ext4
Run /usr/share/pjdfstest/utimensat/06.t against lustre filesystem
prove -f /usr/share/pjdfstest/utimensat/06.t &> /tmp/pjdfstest-lustre
ext4 report
/usr/share/pjdfstest/utimensat/06.t .. ok
All tests successful.
Files=1, Tests=13,  0 wallclock secs ( 0.01 usr  0.01 sys +  0.04 cusr  0.05 csys =  0.11 CPU)
Result: PASS
lustre report
/usr/share/pjdfstest/utimensat/06.t .. 
not ok 7 - tried '-u 99 open . O_RDONLY : utimensat 0 pjdfstest_c35447b5c0515f1b855bb4071b0993c8 0 UTIME_NOW 0 UTIME_NOW 0', expected 0, got EACCES
not ok 11 - tried '-u 99 open . O_RDONLY : utimensat 0 pjdfstest_c35447b5c0515f1b855bb4071b0993c8 0 UTIME_NOW 0 UTIME_NOW 0', expected 0, got EACCES
Failed 2/13 subtests 

Test Summary Report
-------------------
/usr/share/pjdfstest/utimensat/06.t (Wstat: 0 Tests: 13 Failed: 2)
  Failed tests:  7, 11
Files=1, Tests=13,  0 wallclock secs ( 0.01 usr  0.01 sys +  0.04 cusr  0.06 csys =  0.12 CPU)
Result: FAIL
 run_pjdfstest test_utimensat_06: @@@@@@ FAIL: /usr/share/pjdfstest/utimensat/06.t against lustre failed 
  Trace dump:
  = /usr/lib64/lustre/tests/test-framework.sh:5907:error()
  = /tmp/run_pjdfstest.sh:1285:test_utimensat_06()

and

== run_pjdfstest test utimensat_07: utimensat will work if the caller is the owner or root =========== 02:59:14 (1612925954)
Run /usr/share/pjdfstest/utimensat/07.t against ext4 filesystem
prove -f /usr/share/pjdfstest/utimensat/07.t &> /tmp/pjdfstest-ext4
Run /usr/share/pjdfstest/utimensat/07.t against lustre filesystem
prove -f /usr/share/pjdfstest/utimensat/07.t &> /tmp/pjdfstest-lustre
ext4 report
/usr/share/pjdfstest/utimensat/07.t .. ok
All tests successful.
Files=1, Tests=17,  0 wallclock secs ( 0.02 usr  0.00 sys +  0.05 cusr  0.06 csys =  0.13 CPU)
Result: PASS
lustre report
/usr/share/pjdfstest/utimensat/07.t .. 
not ok 5 - tried '-u 99 open . O_RDONLY : utimensat 0 pjdfstest_5d3e9a2a2376970598f14df570b0a92d 0 UTIME_OMIT 1950000000 0 0', expected EPERM, got EACCES
not ok 6 - tried '-u 99 open . O_RDONLY : utimensat 0 pjdfstest_5d3e9a2a2376970598f14df570b0a92d 1900000000 0 0 UTIME_OMIT 0', expected EPERM, got EACCES
not ok 7 - tried '-u 99 open . O_RDONLY : utimensat 0 pjdfstest_5d3e9a2a2376970598f14df570b0a92d 1900000000 0 1950000000 0 0', expected EPERM, got EACCES
not ok 9 - tried '-u 99 open . O_RDONLY : utimensat 0 pjdfstest_5d3e9a2a2376970598f14df570b0a92d 0 UTIME_OMIT 1950000000 0 0', expected EPERM, got EACCES
not ok 10 - tried '-u 99 open . O_RDONLY : utimensat 0 pjdfstest_5d3e9a2a2376970598f14df570b0a92d 1900000000 0 0 UTIME_OMIT 0', expected EPERM, got EACCES
not ok 11 - tried '-u 99 open . O_RDONLY : utimensat 0 pjdfstest_5d3e9a2a2376970598f14df570b0a92d 1900000000 0 1950000000 0 0', expected EPERM, got EACCES
not ok 14 - tried '-u 99 open . O_RDONLY : utimensat 0 pjdfstest_5d3e9a2a2376970598f14df570b0a92d 1900000000 0 1950000000 0', expected 0, got EACCES
Failed 7/17 subtests 

Test Summary Report
-------------------
/usr/share/pjdfstest/utimensat/07.t (Wstat: 0 Tests: 17 Failed: 7)
  Failed tests:  5-7, 9-11, 14
Files=1, Tests=17,  1 wallclock secs ( 0.01 usr  0.01 sys +  0.05 cusr  0.07 csys =  0.14 CPU)
Result: FAIL
 run_pjdfstest test_utimensat_07: @@@@@@ FAIL: /usr/share/pjdfstest/utimensat/07.t against lustre failed 
  Trace dump:
  = /usr/lib64/lustre/tests/test-framework.sh:5907:error()
  = /tmp/run_pjdfstest.sh:1290:test_utimensat_07()

pjdfstest utimensat_06 can be found at https://github.com/pjd/pjdfstest/blob/master/tests/utimensat/06.t
pjdfstest utimensat_07 can be found at https://github.com/pjd/pjdfstest/blob/master/tests/utimensat/07.t

Logs for other failures are at
https://testing.whamcloud.com/test_sets/c66e9756-d5d6-42b4-9e9e-bd0f81b779f0
https://testing.whamcloud.com/test_sets/a0c843f3-01b9-479d-9e2d-91a834e878d5
https://testing.whamcloud.com/test_sets/6de33360-e585-4818-a69b-0f6ccd14ab9a

Comments

[Andreas Dilger]

According to my search, this combination of master server+b2_12 client has never passed testing, and the first (of four) runs of this combination was 2021-02-05:

https://testing.whamcloud.com/search?client_branch_type_id=98c8f076-14c4-11e9-b7d4-52540065bddc&server_branch_type_id=24a6947e-04a9-11e1-bb5f-52540025f9af&test_set_script_id=72be5818-d36d-11e9-97d5-52540065bddc&start_date=2020-10-01&end_date=2021-02-17&source=test_sets#redirect

Not saying that this isn't a bug, but the initial description implies a regression on master around 2021-02-05, but there were only a few landings to master on 2021-02-01 and 2021-02-08.

It would be useful to run a few intermediate tests (eg. 2.13.0 servers) to bisect this down to when the regression was added.

[Peter Jones]

Alena

Could you please try and narrow this issue down?

Thanks

Peter