[LU-16019] sanity test_101j: FAIL: expected 4096 got 8192

• Type: Bug
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.16.0, Lustre 2.15.5
• Affects Version/s: Lustre 2.15.1, Lustre 2.15.3
• Environment: RHEL 9.0 client
• Severity: 3
• Assignee: Qian Yingjin
• Reporter: Maloo

Description

This issue was created by maloo for jianyu <[e-mail]>

This issue relates to the following test suite run: https://testing.whamcloud.com/test_sets/3a8eb542-e2e6-4223-8a43-3f37b87a4636

test_101j failed with the following error:

== sanity test 101j: A complete read block should be submitted when no RA ========================================================== 19:37:12 (1657913832)
Disable read-ahead
16+0 records in
16+0 records out
16777216 bytes (17 MB, 16 MiB) copied, 0.0565881 s, 296 MB/s
Reset readahead stats
4096+0 records in
4096+0 records out
16777216 bytes (17 MB, 16 MiB) copied, 1.08708 s, 15.4 MB/s
snapshot_time             3868.949651534 secs.nsecs
start_time                0.000000000 secs.nsecs
elapsed_time              3868.949651534 secs.nsecs
failed_to_fast_read       8192 samples [pages]
 sanity test_101j: @@@@@@ FAIL: expected 4096 got 8192

VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV
sanity test_101j - expected 4096 got 8192

Comments

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/48219/
Subject: LU-16019 llite: fully disable readahead in kernel I/O path
Project: fs/lustre-release
Branch: b2_15
Current Patch Set:
Commit: 508d74f0a5b5c58cbfe51a6e9a10327892f7f0f3

[Xing Huang]

+2 on b2_15 again
https://testing.whamcloud.com/test_sets/a03722d0-998d-4f4f-9d05-ce39fc7af092
https://testing.whamcloud.com/test_sets/c198d652-bf78-4096-927e-89a79722a4f7

[Xing Huang]

+1 on b2_15 again
https://testing.whamcloud.com/test_sessions/66a00fbf-6bc8-4f72-a0c3-b1a5479c0ca9

[Etienne Aujames]

This test seems to fail systematically on b2_15 with SLES 15.5:

Maloo request:
https://testing.whamcloud.com/search?end_date=2024-01-22&source=sub_tests&start_date=2024-01-16&status=FAIL&sub_test_script_id=4e0877da-9d97-11e9-9e3d-52540065bddc&test_set_script_id=f9516376-32bc-11e0-aaee-52540025f9ae#redirect

List of failed tests:

• https://testing.whamcloud.com/sub_tests/39d71f45-7d51-4f5c-ba4e-f80baaa78c54
• https://testing.whamcloud.com/sub_tests/cfe363db-6808-46eb-9230-c193311948df
• https://testing.whamcloud.com/sub_tests/de311c98-e614-4688-9309-1e94970f291a
• https://testing.whamcloud.com/sub_tests/95fa785f-6a63-4c2e-98e5-e03fb88247b4
• https://testing.whamcloud.com/sub_tests/92c92458-2a70-4d2e-aa4a-102d07a655d4
• https://testing.whamcloud.com/sub_tests/b7419692-b1ad-4b4b-98ca-eb6f7d2f72f2
• https://testing.whamcloud.com/sub_tests/7a1a16a8-d371-44ed-a0ea-b330833008c6
• https://testing.whamcloud.com/sub_tests/28d556b7-a0e4-4cc5-98f6-529cbaa51559
• https://testing.whamcloud.com/sub_tests/f120a210-072b-4b87-b989-f37ed45fd375
• https://testing.whamcloud.com/sub_tests/80fd17a2-2c1b-4d3a-ab7b-fc82f550b083
• https://testing.whamcloud.com/sub_tests/58972dee-d534-4501-b46f-3b7535249c92
• https://testing.whamcloud.com/sub_tests/002e9b15-8ce3-49f2-91c8-afc4581d4171
• https://testing.whamcloud.com/sub_tests/4fc5b38e-d128-4973-8073-d08509653557
• https://testing.whamcloud.com/sub_tests/6c6099d7-bc83-4bdc-adb0-1779eadc270b
• https://testing.whamcloud.com/sub_tests/d357820d-c1a5-4be1-8988-8facf91887b1
• https://testing.whamcloud.com/sub_tests/3898f2b3-0b8a-43a7-82a1-2978d7d0aadf
• https://testing.whamcloud.com/sub_tests/8ee204c2-5e57-47cd-a120-65a97ee093b3
• https://testing.whamcloud.com/sub_tests/bb5d3295-23c3-4177-8ffc-75598752e98e
• https://testing.whamcloud.com/sub_tests/18d788d8-bed7-43af-ab02-c00f357a6b1f
• https://testing.whamcloud.com/sub_tests/a920cbe9-da8b-4433-b48c-df880cc61131
• https://testing.whamcloud.com/sub_tests/54738b79-bea0-489d-a912-cd3609a9de92
• https://testing.whamcloud.com/sub_tests/8c38b7a0-35e4-4626-9571-8f990597b896
• https://testing.whamcloud.com/sub_tests/45939143-f996-49a3-b0f2-331a7734df15
• https://testing.whamcloud.com/sub_tests/1b741892-63f6-416a-8ad1-984c50a6924f
• https://testing.whamcloud.com/sub_tests/ed197a61-8a47-4457-ab49-bbfec4e0ea3a
• https://testing.whamcloud.com/sub_tests/8bc9bd6e-30e8-4ce0-9cdd-55f4201cf356
• https://testing.whamcloud.com/sub_tests/969a3de9-c7ea-4c95-85b7-a8b92c8ba416

[Xing Huang]

+1 on b2_15
https://testing.whamcloud.com/test_sets/a24fd753-7784-4fee-a205-80c2f69c3b9c

[Gerrit Updater]

"Jian Yu <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/48219
Subject: LU-16019 llite: fully disable readahead in kernel I/O path
Project: fs/lustre-release
Branch: b2_15
Current Patch Set: 1
Commit: a0309239b7bf3d2ea1edd2b3c6899e62310f2121

[Peter Jones]

Landed for 2.16

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/47993/
Subject: LU-16019 llite: fully disable readahead in kernel I/O path
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: f0cf7fd3cccb2313fa94a307cf862afba256b8d8

[Gerrit Updater]

"Yingjin Qian <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/47993
Subject: LU-16019 llite: fully disable readahead in kernel I/O path
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: b3cafb06ad0dd347e487df3e4c7b0086f8c23378

[Qian Yingjin]

In the new kernel 5.14 (rhel9), the redahead may be out of Lustre control:

 

void page_cache_sync_ra(struct readahead_control *ractl,
		unsigned long req_count)
{
	bool do_forced_ra = ractl->file && (ractl->file->f_mode & FMODE_RANDOM);

	/*
	 * Even if read-ahead is disabled, issue this request as read-ahead
	 * as we'll need it to satisfy the requested range. The forced
	 * read-ahead will do the right thing and limit the read to just the
	 * requested range, which we'll set to 1 page for this case.
	 */
	if (!ractl->ra->ra_pages || blk_cgroup_congested()) {
		if (!ractl->file)
			return;
		req_count = 1;
		do_forced_ra = true;
	}

	/* be dumb */
	if (do_forced_ra) {
		force_page_cache_ra(ractl, req_count);
		return;
	}

	/* do read-ahead */
	ondemand_readahead(ractl, false, req_count);
}

generic_file_read_iter()
  ->filemap_read()
    ->filemap_get_pages()
      ->page_cache_sync_readahead()
        ->page_cache_sync_ra()

Only setting @ra_pages with 0 can not totally avoid the readahead in the kernel I/O path.

I will try to set @bdi->io_pages to see whether it can totally avoid the readahead in the kernel.