[LU-17289] el9.3 sanity test_906: fio seqwrite /mnt/lustre/f906.sanity failed

• Type: Bug
• Resolution: Unresolved
• Priority: Major
• Fix Version/s: Lustre 2.17.0
• Affects Version/s: Lustre 2.16.0, Lustre 2.15.4
• Labels: always_except
• Severity: 3
• Assignee: Qian Yingjin
• Reporter: Maloo

Description

This issue was created by maloo for jianyu <[e-mail]>

This issue relates to the following test suite run: https://testing.whamcloud.com/test_sets/d8edb03f-19f7-40df-96cf-3f713ddab30d

test_906 failed with the following error:

== sanity test 906: Simple test for io_uring I/O engine via fio ========================================================== 23:00:18 (1700002818)
/usr/bin/fio
seqwrite: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=64
...
fio-3.35
Starting 2 processes
seqwrite: Laying out IO file (1 file / 50MiB)
fio: pid=1302842, err=1/file:engines/io_uring.c:1047, func=io_queue_init, error=Operation not permitted
fio: pid=1302841, err=1/file:engines/io_uring.c:1047, func=io_queue_init, error=Operation not permitted


Run status group 0 (all jobs):
 sanity test_906: @@@@@@ FAIL: fio seqwrite /mnt/lustre/f906.sanity failed 

Test session details:
clients: https://build.whamcloud.com/job/lustre-reviews/100333 - 5.14.0-362.8.1.el9_3.x86_64
servers: https://build.whamcloud.com/job/lustre-reviews/100333 - 4.18.0-477.27.1.el8_lustre.x86_64

<<Please provide additional information about the failure here>>

VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV
sanity test_906 - fio seqwrite /mnt/lustre/f906.sanity failed

Issue Links

• is related to: LU-17927 sanity test_119e: FAIL: trivial unaligned dio failed

Comments

[Andreas Dilger]

This test has started timing out in the past few weeks on Ubuntu 22.04 and 24.04 clients running kernel 5.15.0-94-generic and 6.8.0-31-generic respectively:
https://testing.whamcloud.com/search?horizon=2332800&status%5B%5D=FAIL&status%5B%5D=TIMEOUT&test_set_script_id=f9516376-32bc-11e0-aaee-52540025f9ae&sub_test_script_id=cba8ed3d-b362-4dec-9a1a-b7590caca031&source=sub_tests#redirect

I think this showing that there is a real problem with io_uring on these newer kernels.

[Andreas Dilger]

According to testing that yujian has been doing for el9.4 kernel updates, sanity test_906 is still failing 100% on el9.4 if it is not skipped:

https://testing.whamcloud.com/search?client_distribution_type_id=5776ec54-99a6-4b55-9135-2ba06d391e6b&horizon=2332800&test_set_script_id=f9516376-32bc-11e0-aaee-52540025f9ae&sub_test_script_id=cba8ed3d-b362-4dec-9a1a-b7590caca031&source=sub_tests#redirect

[Peter Jones]

So does el9.4 work with fio? If not, let's push this ticket to 2.17....

[Andreas Dilger]

The landed patch is only skipping the test, it is not fixed.

Keep this open until we get an el9 kernel that works with fio, then the always_except can get a maximum version number.

[Peter Jones]

Merged for 2.16

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/53362/
Subject: LU-17289 test: disable sanity/test_906 temporarily
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 1ecd9a05bcc6b979e20f7ea38c1d6a40f7de39b9

[Gerrit Updater]

"Qian Yingjin <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/53362
Subject: LU-17289 test: disable sanity/test_906 temporarily
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 0eef4b0818e7a1a42a54333fa713ef660c7e9404

[Jian Yu]

The failure occurred on RHEL 9.3 client.
I've tried the latest fio-3.36 and the local xfs filesystem but hit the same failure.
I also did a bit googling but didn't find a related issue reported. Not sure if RHEL 9.3 changes something in libaio that breaks the fio tool.