[LU-16087] show distribution information of jobstats Created: 11/Aug/22  Updated: 11/May/23  Resolved: 06/Dec/22

Status: Resolved
Project: Lustre
Component/s: None
Affects Version/s: None
Fix Version/s: Lustre 2.16.0

Type: Improvement Priority: Minor
Reporter: Feng Lei Assignee: Feng Lei
Resolution: Fixed Votes: 0
Labels: None

Issue Links:
Related
is related to LU-12872 Adding more stats into JOBSTATS Resolved
Rank (Obsolete): 9223372036854775807

 Description   

JOBSTSTS has been very useful to understand what type of IO is coming from application per JOBID/UID and GID, but it also would be nice to have more stats (e.g. like RPC size, discontiguous pages, etc that is covered by "brw_stats" today) into JOBSTATS to understand detail IO workload/size per JOBID.
The other hand, the hybrid system with SSD and HDD (with/without same namespace) is naturally coming and those new stats allow let administrator or users know more guideline/information e.g. which OST devices are preferred by each application.

LU-12872 is used to track the requirement of adding more stats.

LU-16087 is used to track the requirement of show distribution information of jobstats.

 Comments   
Comment by Feng Lei [ 11/Aug/22 ]

Copy from LU-12872:

I'm going to add:

enum {
    LPROCFS_CNTR_EXTERNALLOCK = 0x0001,
    LPROCFS_CNTR_AVGMINMAX    = 0x0002,
    LPROCFS_CNTR_STDDEV       = 0x0004,
    LPROCFS_CNTR_HISTOGRAM    = 0x0008,

    ...
    LPROCFS_TYPE_BYTES_FULL_HISTOGRAM = LPROCFS_TYPE_BYTES_FULL | LPROCFS_CNTR_HISTOGRAM,

The output may be like:

  read_pages: { samples: 0, unit: pages, min: 0, max: 0, sum: 0, sumsq: 0, histgram: {1: xxx, 2: yyy, 4: zzz, ...} }
Comment by Gerrit Updater [ 22/Aug/22 ]

"Feng, Lei <flei@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/48278
Subject: LU-16087 lprocfs: add histogram to stats counter
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: b2af413b9a03a96105a6e9a12e47acf98cc17b64

Comment by Gerrit Updater [ 06/Dec/22 ]

"Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/48278/
Subject: LU-16087 lprocfs: add histogram to stats counter
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: fde40ce32c91c804cb85be085f2aaf06170047b6

Comment by Peter Jones [ 06/Dec/22 ]

Landed for 2.16

Generated at Sat Feb 10 03:23:53 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.