[LU-14361] Add support for statahead pattern with regularized file name

• Type: Improvement
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.17.0
• Labels: mtg_7 statahead
• Assignee: Qian Yingjin
• Reporter: Qian Yingjin

Description

Some applications do stat() calls under a directory within which all the children files have regularized file name:

• mdtest benchmark: mdtest.$i
• du more than 10000 entries: file.$i
• ML/AI with ingested data that have typically a rule of filename in the directory.

These applications call stat() on the files in the Alphabetic sorting order of the file name.

This kind of metadata operations can not be optimized by current statahead mechanism.
The current statahead mechanism works as follows:
1. Open the directory via opendir() call. It will authorize the statahead.
2. readdir() to get the name and inode number for the dentries.
3. do stat() on the dentries one by one.
4. Close the directory which will deauthorize the statahead.

In current statahead mechanism, the stat() calls in the order of populate the dentries via readdir(). For the ldiskfs backend, it is ordering by the hash of the file name, not a kinds fo sorting oder.

However, we can improve the statahead to support statahead pattern with regularized file name to optimize the metadata performance for the above applications

Issue Links

• is related to: LU-18510 sanity-sec test_63: crash in ll_prep_md_op_data()
• is related to: LU-17450 sanity: interop test failures with master+2.15
• is related to: LU-18536 statahead: decrease the inuse entry count when entry state waiting timeout
• is related to: LU-14139 batched statahead processing
• is related to: LU-17399 sanity: test_123i Error: 'File name pattern statahead not trigger'
• is related to: LU-14139 batched statahead processing
• is related to: LU-14380 Make statahead better support Breadth First Search (BFS) or Depth First Search (DFS)
• is related to: LU-18523 tests sanity/test_65j: wait for ll_sa stat-ahead thread to quit during cleanup

Comments

[Peter Jones]

Merged for 2.17

[Peter Jones]

Am I right in thinking that the above-mentioned fix was tracked under LU-18536, which is now merged, so this ticket can be marked as resolved again?

[Qian Yingjin]

This is not a bug for stat-ahead when the fname pattern stat-ahead is enabled by default.

The reason is that fname pattern stat-ahead is not same as dir listing pattern stat-ahead.
In dir listing dir, the stat-ahead thread (ll_sa_xx) will quit when the closedir() is called.
However, in fname pattern stat-ahead, we wait for @ll_sa_timeout (30s by default) if it is no longer using the directory (traverse and stat) for this long time and then quit.

A simple patch should fix this error:

diff --git a/lustre/tests/sanity.sh b/lustre/tests/sanity.sh
index 8f34ffa5a8..d26256d97a 100755
--- a/lustre/tests/sanity.sh
+++ b/lustre/tests/sanity.sh
@@ -92,6 +92,8 @@ proc_regexp="/{proc,sys}/{fs,sys,kernel/debug}/{lustre,lnet}/"
 
 cleanup() {
        echo -n "cln.."
+       wait_update_facet client "pgrep ll_sa" "" 35 ||
+               error "ll_sa thread is still running"
        pgrep ll_sa > /dev/null && { echo "There are ll_sa thread not exit!"; exit 20; }
        cleanupall ${FORCE} $* || { echo "FAILed to clean up"; exit 20; }
 }

== sanity test 65j: set default striping on root directory (bug 6367)=========================================================== 09:34:58 (1733016898)
cln..Waiting 35s for ''
Waiting 25s for ''
Stopping clients: qcn /mnt/lustre (opts:)
Stopping client qcn /mnt/lustre opts:

I will make a patch for it later.

[Alex Zhuravlev]

# ONLY=63-65  bash sanity.sh
...
== sanity test 65j: set default striping on root directory (bug 6367)=========================================================== 10:11:30 (1733393490)
[  134.944066] Lustre: DEBUG MARKER: == sanity test 65j: set default striping on root directory (bug 6367)=========================================================== 10:11:30 (1733393490)
cln..There are ll_sa thread not exit!
test_65j returned 20
FAIL 65j (2s)

[Qian Yingjin]

Hi, Alex,

How do you reproduce the failure?

I can not reproduce the bug by using the following command:

 ONLY="65j" MDSCOUNT=2 REFORMAT=yes ./sanity.sh 

Could you please advise?

[Alex Zhuravlev]

the patch introduced a problem in sanity:

 == sanity test 65j: set default striping on root directory (bug 6367)=========================================================== 03:57:43 (1733284663)
cln..There are ll_sa thread not exit!
test_65j returned 20
FAIL 65j (1s)

the bisection is very definitive:

COMMIT		TESTED	PASSED	FAILED		COMMIT DESCRIPTION
241cf3c6d0      1       0       1       BAD     LU-16235 hsm: get a valid cookie for RAoLU request
f4bf36d8f4      3       0       3       BAD     LU-14361 statahead: add tunable for fname pattern detection
5803284ac3      3       3       0       GOOD    LU-11850 obd: use netlink to get lustre stats
8b5254fbab      3       3       0       GOOD    LU-11073 tests: enable DNE in recovery-mds-scale
371c0c1aef      3       3       0       GOOD    LU-11610 target: Fix correct return value in out_handler.c
1ccc52da74      3       3       0       GOOD    LU-18471 ldiskfs: large directory causes htree corruption

[Peter Jones]

Merged for 2.17

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/51592/
Subject: LU-14361 statahead: add tunable for fname pattern detection
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: f4bf36d8f4957b62e4edab58b971829dccedd0c0

[Andreas Dilger]

Two patches still to be landed for this ticket to be completed:

• https://review.whamcloud.com/51592 "LU-14361 statahead: add tunable for fname pattern detection"
• https://review.whamcloud.com/55850 "LU-14361 statahead: wait all inuse statahead done during umount"

but the second fix patch should be merged into the first patch to avoid crashes during testing that have so far prevented the patch from landing.

[Gerrit Updater]

"Qian Yingjin <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/55850
Subject: LU-14361 statahead: wait all inuse statahead done during umount
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 4ea4a1a754536678dda492c7b56257d0641011bc