[LU-13244] sanity test 184a fails with 'swap of file layout failed'

• Type: Bug
• Resolution: Unresolved
• Priority: Minor
• Affects Version/s: Lustre 2.14.0
• Labels: ppc
• Environment: PPC clients
• Severity: 3
• Assignee: WC Triage
• Reporter: James Nunez (Inactive)

Description

sanity test 184a fails with 'swap of file layout failed' starting on 27 SEPT 2019 and fails 100% of the time for PPC client testing.

Looking at a recent failure at https://testing.whamcloud.com/test_sets/d3720002-4a27-11ea-b69a-52540065bddc, the suite_log shows that the real problem is with ‘lfs setstripe’

== sanity test 184a: Basic layout swap =============================================================== 01:26:08 (1581125168)
lfs setstripe: '/mnt/lustre/d184a.sanity/184a' not on a mounted Lustre filesystem
lfs setstripe: '/mnt/lustre/d184a.sanity/184a/f1' is not on a Lustre filesystem: No such device (19)
lfs setstripe: '/mnt/lustre/d184a.sanity/184a' not on a mounted Lustre filesystem
lfs setstripe: '/mnt/lustre/d184a.sanity/184a/f2' is not on a Lustre filesystem: No such device (19)
lfs getstripe: cannot get lov name: Inappropriate ioctl for device (25)
error: getstripe failed for /mnt/lustre/d184a.sanity/184a/f1.
lfs getstripe: cannot get lov name: Inappropriate ioctl for device (25)
error: getstripe failed for /mnt/lustre/d184a.sanity/184a/f2.
lfs swap_layouts: error: cannot swap layout between '/mnt/lustre/d184a.sanity/184a/f1' and '/mnt/lustre/d184a.sanity/184a/f2': Inappropriate ioctl for device (25)
 sanity test_184a: @@@@@@ FAIL: swap of file layout failed 
  Trace dump:
  = /usr/lib64/lustre/tests/test-framework.sh:6121:error()
  = /usr/lib64/lustre/tests/sanity.sh:15043:test_184a()
 

There are no errors or any indication of why this test fails in any of the console logs.

There are several other sanity tests that ‘lfs setstripe’ fails with “not on a mounted Lustre filesystem” or “not a Lustre filesystem” including 200, 204a, 204b, 204c, 204d, 204e, 204f, 204g, 204h, 231a, 232a, 232b, 236, 243, 250, 253, 255a, 255b, 255c, 272e, 272f, 300m, 311, 313, 314, 399b, 402, 405, 802b, 812a, 812b and 816.

In addition, there are other tests where ‘lfs setstripe’ fails with different errors. Tests 270a, 270b, 270c, 270d, 270e, 271c, 271d, 271f, 272a, 272b, 272c, 272d and 809 all fail with

== sanity test 270a: DoM: basic functionality tests ================================================== 02:34:55 (1581129295)
lfs setstripe: cannot create composite file '/mnt/lustre/d270a.sanity/dom_file': Inappropriate ioctl for device
 sanity test_270a: @@@@@@ FAIL: Can't create DoM layout 
  Trace dump:
  = /usr/lib64/lustre/tests/test-framework.sh:6121:error()
  = /usr/lib64/lustre/tests/sanity.sh:18523:test_270a()

sanity test 270f call to ‘lfs mkdir’ also fails with ‘not on a mounted Lustre filesystem’

== sanity test 270f: DoM: maximum DoM stripe size checks ============================================= 02:35:06 (1581129306)
CMD: trevis-10vm12 /usr/sbin/lctl get_param -n lod.lustre-MDT0000-mdtlov.dom_stripesize
CMD: trevis-10vm12 /usr/sbin/lctl set_param -n lod.lustre-MDT0000-mdtlov.dom_stripesize=131072
CMD: trevis-10vm12 /usr/sbin/lctl get_param -n lod.lustre-MDT0000-mdtlov.dom_stripesize
lfs mkdir: '/mnt/lustre' not on a mounted Lustre filesystem
lfs mkdir: '/mnt/lustre/d270f.sanity' is not on a Lustre filesystem: No such device (19)
lfs setdirstripe: cannot create dir '/mnt/lustre/d270f.sanity': No such device
lfs setstripe: '/mnt/lustre' not on a mounted Lustre filesystem
lfs setstripe: '/mnt/lustre/d270f.sanity' is not on a Lustre filesystem: No such device (19)
lfs setstripe: cannot create composite file '/mnt/lustre/d270f.sanity': Inappropriate ioctl for device
 sanity test_270f: @@@@@@ FAIL: Can't set directory default striping 
  Trace dump:
  = /usr/lib64/lustre/tests/test-framework.sh:6121:error()
  = /usr/lib64/lustre/tests/sanity.sh:18757:test_270f()

Similarly, sanity test 184c, 184d, 184e, 187a, 208, 244a, 244b and 813 all fail with an inappropriate ioctl error for various lfs calls

== sanity test 184c: Concurrent write and layout swap ================================================ 01:26:13 (1581125173)
61+0 records in
61+0 records out
63963136 bytes (64 MB) copied, 0.245054 s, 261 MB/s
42+0 records in
42+0 records out
44040192 bytes (44 MB) copied, 0.16921 s, 260 MB/s
ref file size: ref1(63963136), ref2(44040192)
3904+0 records in
3904+0 records out
63963136 bytes (64 MB) copied, 0.0380003 s, 1.7 GB/s
lfs swap_layouts: error: cannot swap layout between '/mnt/lustre/d184c.sanity/184c/file1' and '/mnt/lustre/d184c.sanity/184c/file2': Inappropriate ioctl for device (25)
 sanity test_184c: @@@@@@ FAIL: swap of /mnt/lustre/d184c.sanity/184c/file1 and /mnt/lustre/d184c.sanity/184c/file2 failed 
  Trace dump:
  = /usr/lib64/lustre/tests/test-framework.sh:6121:error()
  = /usr/lib64/lustre/tests/sanity.sh:15117:test_184c()

== sanity test 184d: allow stripeless layouts swap =================================================== 01:26:15 (1581125175)
Succeed in opening file "/mnt/lustre/d184d.sanity/f184d.sanity-2"(flags=O_CREAT)
Succeed in opening file "/mnt/lustre/d184d.sanity/f184d.sanity-3"(flags=O_CREAT)
lfs getstripe: cannot get lov name: Inappropriate ioctl for device (25)
error: getstripe failed for /mnt/lustre/d184d.sanity/f184d.sanity-1.
lfs swap_layouts: error: cannot swap layout between '/mnt/lustre/d184d.sanity/f184d.sanity-2' and '/mnt/lustre/d184d.sanity/f184d.sanity-3': Inappropriate ioctl for device (25)
 sanity test_184d: @@@@@@ FAIL: swap /mnt/lustre/d184d.sanity/f184d.sanity-2 /mnt/lustre/d184d.sanity/f184d.sanity-3 layouts failed 
  Trace dump:
  = /usr/lib64/lustre/tests/test-framework.sh:6121:error()
  = /usr/lib64/lustre/tests/sanity.sh:15157:test_184d()

== sanity test 244a: sendfile with group lock tests ================================================== 02:32:09 (1581129129)
35+0 records in
35+0 records out
36700160 bytes (37 MB) copied, 0.00913071 s, 4.0 GB/s
Starting test test10 at 1581129130
Finishing test test10 at 1581129130
Starting test test11 at 1581129130
sendfile_grouplock: cannot get group lock: Inappropriate ioctl for device (25)
sendfile_grouplock: sendfile_grouplock.c:229: sendfile_copy: assertion 'rc == 0' failed: cannot set group lock 85543 for '/mnt/lustre/d244a.sanity/f244a.sanity': Inappropriate ioctl for device
 sanity test_244a: @@@@@@ FAIL: sendfile+grouplock failed 
  Trace dump:
  = /usr/lib64/lustre/tests/test-framework.sh:6121:error()
  = /usr/lib64/lustre/tests/sanity.sh:17525:test_244a()

multiop fails with similar ‘not on a mounted Lustre filesystem’ or ‘inapporpriate ioctl’ errors for sanity tests 208, 229

== sanity test 208: Exclusive open =================================================================== 02:28:13 (1581128893)
==== test 1: verify get lease work
multiop: cannot get READ lease, ext 0: Inappropriate ioctl for device (25)
multiop: apply/unlock lease error: Inappropriate ioctl for device
 sanity test_208: @@@@@@ FAIL: get lease error 
  Trace dump:
  = /usr/lib64/lustre/tests/test-framework.sh:6121:error()
  = /usr/lib64/lustre/tests/sanity.sh:15732:test_208()

== sanity test 229: getstripe/stat/rm/attr changes work on released files ============================ 02:30:54 (1581129054)
multiop: '/mnt/lustre' not on a mounted Lustre filesystem
multiop: '/mnt/lustre/f229.sanity' is not on a Lustre filesystem: No such device (19)
close: Bad file descriptor
 sanity test_229: @@@@@@ FAIL: failed to create file with released layout 
  Trace dump:
  = /usr/lib64/lustre/tests/test-framework.sh:6121:error()
  = /usr/lib64/lustre/tests/sanity.sh:16524:test_229()