[LU-16087] show distribution information of jobstats

• Type: Improvement
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.16.0
• Assignee: Feng Lei
• Reporter: Feng Lei

Description

JOBSTSTS has been very useful to understand what type of IO is coming from application per JOBID/UID and GID, but it also would be nice to have more stats (e.g. like RPC size, discontiguous pages, etc that is covered by "brw_stats" today) into JOBSTATS to understand detail IO workload/size per JOBID.
The other hand, the hybrid system with SSD and HDD (with/without same namespace) is naturally coming and those new stats allow let administrator or users know more guideline/information e.g. which OST devices are preferred by each application.

LU-12872 is used to track the requirement of adding more stats.

LU-16087 is used to track the requirement of show distribution information of jobstats.

Issue Links

• is related to: LU-12872 Adding more stats into JOBSTATS

Comments

[Peter Jones]

Landed for 2.16

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/48278/
Subject: LU-16087 lprocfs: add histogram to stats counter
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: fde40ce32c91c804cb85be085f2aaf06170047b6

[Gerrit Updater]

"Feng, Lei <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/48278
Subject: LU-16087 lprocfs: add histogram to stats counter
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: b2af413b9a03a96105a6e9a12e47acf98cc17b64

[Feng Lei]

Copy from LU-12872:

I'm going to add:

enum {
    LPROCFS_CNTR_EXTERNALLOCK = 0x0001,
    LPROCFS_CNTR_AVGMINMAX    = 0x0002,
    LPROCFS_CNTR_STDDEV       = 0x0004,
    LPROCFS_CNTR_HISTOGRAM    = 0x0008,

    ...
    LPROCFS_TYPE_BYTES_FULL_HISTOGRAM = LPROCFS_TYPE_BYTES_FULL | LPROCFS_CNTR_HISTOGRAM,

The output may be like:

  read_pages: { samples: 0, unit: pages, min: 0, max: 0, sum: 0, sumsq: 0, histgram: {1: xxx, 2: yyy, 4: zzz, ...} }