[LU-9221] Create pid-based hash to enhance Jobstats performance

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.9.0
• Fix Version/s: Lustre 2.11.0
• Type: Improvement
• Priority: Minor
• Reporter: Ben Evans (Inactive)
• Assignee: Ben Evans (Inactive)
• Resolution: Fixed
• Epic/Theme: Performance

Description

Currently jobstats gets information by either dedicating an entire node to a single job (LU-7195) or reading from the userspace environment for each call. The first is very fast, but can be inaccurate, the second introduces a significant performance penalty on the client. This patch creates a cache of JobID names mapped to PIDs. It has several fallback mechanisms in order to present the most accurate info available.

Issue Links

• is related to: LUDOC-381 Improve documentation for jobstats

Comments

[Cory Spitz]

Patch up for review is at https://review.whamcloud.com/#/c/25208/.

[Cory Spitz]

@hdoreau, the OpenSFS LWG is looking to you to be the reviewer of record for this enhancement. Can you accept? We're getting close to code cut-off for v2.10 and we were hoping that you would have time for reviews. Thanks!

[Peter Jones]

Could still make 2.10 if reviews complete in time but otherwise will be in 2.10.1

[Henri Doreau (Inactive)]

@cory: I'd be glad to help but it will not have much time to review Ben's patch before LUG...

As seen in first revisions of the patch, introducing such a new cache comes with a risk of subtle race conditions and memory leaks. Therefore reviewing and testing it seriously cannot be done in a rush.

[Ben Evans (Inactive)]

JobID Cache Proposal.docx
Updated doc

[Ben Evans (Inactive)]

https://jira.hpdd.intel.com/browse/LUDOC-381

[Andrew Perepechko]

Another interesting tool which might be useful to implement this feature is the linux key management system (see man keyctl).

One can create a key with some data attached and link it to one of the available keyrings (thread, process, session, default session, etc). The key will be stored on a temporal or permanent basis. The cache can be easily searched from the kernel using request_key().

It's also possible to create a key type so that requested keys are created using a userspace upcall (see /sbin/request-key, /etc/request-key.conf).

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/25208/
Subject: LU-9221 jobstats: Create a pid-based hash for jobid values
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 08479b74ec3599ee91e14f3f646389bb0aca4575

[Peter Jones]

Landed for 2.11