[LU-10948] client cache open lock after N opens

• Type: Improvement
• Resolution: Unresolved
• Priority: Minor
• Affects Version/s: Lustre 2.9.0
• Environment: cent server/sles client

Epic/Theme

Performance

• Assignee: Oleg Drokin
• Reporter: Bob Ciotti (Inactive)

Description

listed as minor but when a user does this, we start to get phone calls form other users and then page POC to identify code/user. Workaround is to terminate user job(s).

Oleg has said that Lustre has an existing feature for a client to acquire open lock but off by default. This to mimic NFS behavior.

Ideal change would be that we can specify a number of times that a file is opened on single client at which time lock is acquired. (e.g. 10th time)

Use case is naive user who loop in this way on like 5000+ threads in java:

do until till the sun turns black()

{ fd = open(*my_thread_ID, O_APPEND) calculate_something_small_but_useful() write(fd, *fortytwo, 42) close(fd }

Users often don't have complete control over the code they run and as a result may not be able to quickly make even simple changes.

Issue Links

• is duplicated by: LU-5426 Add more controls for open file cache
• is related to: LU-12262 Improve sbi_flags checking
• is related to: LU-14694 racer timeouts with LU-10948
• is related to: LU-14743 Interop: sanity test 429 fails with 'multiop failed'
• is related to: LU-16463 llapi_open_by_fid open should not be cached systematically
• is related to: LU-10957 Return more lock bits from MDS for open-create
• is related to: LU-11623 Allow caching of open-created dentries
• is related to: LU-7915 investigate heuristics for SPARK client getting MDS openlock
• is related to: LU-4398 mdt_object_open_lock() may not flush conflicting handles
• is related to: LU-10269 Fixes for selective trybits
• is related to: LU-11199 mdsrate open() performance degradation
• is related to: LU-8585 All Lustre test suites should pass with subdirectory mount
• is related to: LU-10955 Cancel open lock as part of close request
• is related to: LU-7915 investigate heuristics for SPARK client getting MDS openlock

Comments

[Gerrit Updater]

Mike Pershin ([e-mail]) uploaded a new patch: https://review.whamcloud.com/41091
Subject: LU-10948 llite: Revalidate dentries in ll_intent_file_open
Project: fs/lustre-release
Branch: b2_12
Current Patch Set: 1
Commit: d1242a54f369712b5c39cc51e3118e9f83c5895e

[Andreas Dilger]

This issue has not been resolved, as the main functional patches have not been landed yet. The few patches that have been landed so far are just cleaning up issues in the code.

There are two patches from this ticket outstanding for this ticket that add a simple client-side per-inode counter+threshold to decide whether the client should fetch the openlock:

• https://review.whamcloud.com/32157 "LU-10948 llite: Revalidate dentries in ll_intent_file_open"
• https://review.whamcloud.com/32158 "LU-10948 llite: WIP! Introduce inode open counter"

There is also an older patch on LU-7915 that just provides a whole-client tunable to enable opencache:

• https://review.whamcloud.com/19664 "LU-7915 llite: add a tunable to enable opencache"

Getting the patches for this ticket updated to the latest master and reviewing/investigating the test failures would be useful for getting it landed sooner.

The LU-7915 patch might be useful for testing how much opencache would help a specific workload, but always-on opencache had shown to be a performance hit in some cases in the past, which is why we didn't enable opencache permanently. Still, landing patch 19664 would at least give an available tunable to turn this on/off on a per-client basis if it shows a net improvement for a site's workload.

[Alex Kulyavtsev]

Is this resolved?

In what lustre release this patch is available?

 

[James A Simmons]

Is this resolved?

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/35039/
Subject: LU-10948 mdt: Remove openlock compat code with 2.1
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: f4e39f710f9069208594870b5cdd37879b46a404

[Gerrit Updater]

Oleg Drokin ([e-mail]) uploaded a new patch: https://review.whamcloud.com/35039
Subject: LU-10948 mdt: Remove openlock compat code with 2.1
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 1f0f1746f2b2f368e76b8b188f85efe96a7f69e3

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/32157/
Subject: LU-10948 llite: Revalidate dentries in ll_intent_file_open
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 14ca3157b21d8bd22be29c9578819b72fd39a1e5

[Patrick Farrell (Inactive)]

Ah, sorry, that was ambiguous.  Reduction in the open rate, so, bad news.

On a real system, this mdsrate open benchmark drops from 35K opens/second to around 11K.
This is 10K opens per process, 64 processes, opens are random files from among 300000 existing files (created by mdsrate earlier): 
aprun -n 64 /usr/lib64/lustre/tests/mdsrate -d /mnt/lustre/mdsrate --open --iters 10000 --nfile=300000

On a much smaller VM, I see a drop from 8K opens/second to 4K with this benchmark.
This is 8K opens per process, 4 processes, opens randomly selected from among 30000 existing files:
mpirun -n 4 /usr/lib64/lustre/tests/mdsrate -d /mnt/lustre/mdsrate --open --iters 8000 --nfile=30000

As discussed elsewhere, I'll open an LU for this in a few minutes.

[Andreas Dilger]

Patrick, when you say "reduction in the mdsrate open() benchmark", does that mean "reduction in the time taken" (== good), or "reduction in the open rate" (== bad)?

[Patrick Farrell (Inactive)]

For what it's worth, we've observed a 70-80% reduction in the mdsrate open() benchmark from the change that https://review.whamcloud.com/#/c/32156/ reverses.

Just something to consider - The impact of that change alone is enormous.

[Mahmoud Hanafi]

Here is comparing the C code vs FORTRAN. Including strace of the FORTRAN.

 C 100 iterations
----------------  
/proc/fs/lustre/mdc/nbptest2-MDT0000-mdc-ffff88203d2dd800/md_stats
close               9
intent_lock         11
/proc/fs/lustre/mdc/nbptest2-MDT0000-mdc-ffff88203d2dd800/stats
req_waittime        29
req_active          29
mds_close           9
ldlm_cancel         9
/proc/fs/lustre/llite/nbptest2-ffff88203d2dd800/stats
write_bytes         100
open                100
close               100
getxattr            10
getxattr_hits       9
inode_permission    200

/proc/fs/lustre/mdt/nbptest2-MDT0000/exports/10.151.31.132@o2ib/ldlm_stats
ldlm_enqueue        11
ldlm_cancel         9
/proc/fs/lustre/mdt/nbptest2-MDT0000/exports/10.151.31.132@o2ib/stats
open                10
close               9
getxattr            1

Fortran 100 iterations
---------------------
/proc/fs/lustre/mdc/nbptest2-MDT0000-mdc-ffff88203d2dd800/md_stats
close               99
intent_lock         302 
setattr             100
/proc/fs/lustre/mdc/nbptest2-MDT0000-mdc-ffff88203d2dd800/stats
req_waittime        497
req_active          497
mds_close           99
ldlm_cancel         99
obd_ping            1
/proc/fs/lustre/llite/nbptest2-ffff88203d2dd800/stats
write_bytes         100
ioctl               200
open                100
close               100
seek                200
truncate            100
getattr             101
getxattr            100
getxattr_hits       100 
inode_permission    406

/proc/fs/lustre/mdt/nbptest2-MDT0000/exports/10.151.31.132@o2ib/ldlm_stats
ldlm_enqueue        198
ldlm_cancel         99
/proc/fs/lustre/mdt/nbptest2-MDT0000/exports/10.151.31.132@o2ib/stats
open                99
close               99
getattr             100
setattr             100

 

fortran.100iter.strace