-
New Feature
-
Resolution: Unresolved
-
Medium
-
None
-
None
-
3
-
9223372036854775807
Problem
CSP-managed Lustre services wall off backend access. Users, admins and
cluster operators cannot read the server-side `mdt.*.job_stats` and
`obdfilter.*.job_stats` counters. Jobstats is an essential tool for cluster
operators to monitor and assess filesystem use. It helps them see which
jobs use the resources and check that sharing is fair and works as
intended. We need jobstats on the clients because operators cannot access
it on the servers.
Large Lustre users increasingly run both on-premises Lustre and
CSP-managed Lustre services. Operators need the same per-job visibility in
both environments. Client-side jobstats provides a uniform observability
surface for monitoring use and checking resource sharing across both.
Proposed behavior
Keep the feature client-only. It must work with unmodified servers and
require no server upgrade, server configuration change, wire-format change
or disk-format change. Operators must be able to deploy it on clients
without changes by the managed service provider.
Follow server jobstats semantics and defaults wherever they apply. Reuse
job ID selection, corresponding counter names and units, YAML records,
clear and per-job removal, and cleanup rules. Keep existing `jobid_var`
and `jobid_name` behavior and defaults, including `jobid_var=disable`.
Keep the default `job_cleanup_interval` at 600 seconds. Cleanup is
periodic; the interval is not an exact record-expiry deadline.
Use one feature LU for three interfaces:
- `osc.*.job_stats`: per-job bytes and RPC activity for each OST import,
with client RPC round-trip and queue times. - `mdc.*.job_stats`: per-job metadata operations and Data-on-MDT bytes
for each MDT import, with client RPC timings. - `llite.*.job_stats`: a later delivery for application operations and
bytes per mount, including reads served from the page cache.
The first delivery implements OSC and MDC. Client counting is disabled
by default. Each device has `enable_job_stats=0` and
`job_cleanup_interval=600`. Client `job_stats` files use debugfs mode
`0600`. Their rejected-update counters use mode `0400`. The interface
location and control names need upstream design review.
OSC and MDC use the job ID already carried on each RPC. Preserve existing
shared-file, writeback, background-worker and retry attribution limits.
Do not add a new job ID or change requests to improve attribution.
Disabling a job ID mode does not rewrite IDs already carried by requests
in flight.
Count successful RPC payload bytes after completion, using the transferred
size for bulk I/O and the validated reply payload size for inline reads.
Server page padding at EOF is included when it is transferred. These
counts can exceed bytes returned to the application. Application-byte
accounting belongs to the later llite interface.
Count metadata operations from successful replies observed by the client. For
metadata batches, count each validated successful positive lookup in the
completed prefix. Do not count an unreturned or retried suffix as done.
The existing batch interpreter owns reply decoding.
Client RPC time is not server service time. Data RPC timing can include
an attempt that received an error reply; successful-byte accounting cannot.
Queue time applies to requests sent once, excluding deliberate retry
backoff. A client sees only its own activity. These counters do not promise
exact equality with server counters across errors, retries, cache hits,
server versions or unsupported batch operations.
Bounded memory and visible loss
Use a node-wide job-record memory budget across all client devices:
- `client_jobstats_max_memory`: reserved allocation limit in bytes;
default 64 MiB, or 67,108,864 bytes. - `client_jobstats_memory`: current reserved bytes, including allocations
in progress and records waiting for final release. - ` {osc,mdc}.*.job_stats_dropped`: rejected counter updates since device
initialization. This counts updates, not bytes, jobs or RPCs. One RPC
can attempt more than one update.
Reserve space before allocating a new record. Use shared counters on
clients and allocate their storage at admission. Do not add per-CPU
counter storage later without charging it. Server counter allocation and
defaults remain unchanged.
A zero budget stops new records; existing records continue to collect.
Lowering the budget below current reservations retains existing records
and prevents new ones until enough memory is released. Clear and cleanup
return reservations after readers and deferred frees finish. Clearing
records does not reset `job_stats_dropped`.
The budget covers reserved job-record allocations. It does not cap all
Lustre memory, fixed device state, reader buffers or allocator metadata.
Allocation or admission failure must be visible without failing the
application's I/O. If optional jobstats setup fails, the mount can still
succeed and enabling the unavailable interface returns an error.
Existing engine prerequisites
The shared engine has existing fixes under review. Track them as explicit
prerequisites, with their own LUs and reviews:
| Review | Existing defect | Revision selected for the prerequisite baseline |
| [62167 / LU-19555](https://review.whamcloud.com/c/fs/lustre-release/+/62167) | seq_file reference handling and teardown | `4b69d9fbacea8074d3e84ac6e881d168122196ef` |
| [68602 / LU-17000](https://review.whamcloud.com/c/fs/lustre-release/+/68602) | position-tree corruption after insertion collision | `b95be1553298813870e0b975e5886a46423b5285` |
| [69763 / LU-20984](https://review.whamcloud.com/c/fs/lustre-release/+/69763) | missing cleanup wakeups | `53a75006617a030a1e9b255305bf98aaf2ef52fe` |
Refresh these reviews before submission. Record any overlap when building
the prerequisite baseline. Do not copy partial fixes into the feature or
present them as new client work. An unmerged prerequisite must be stated
as such. No server deployment is required by these client build
dependencies. Preserve server behavior in the feature changes relative
to that baseline and test the shared engine on servers separately.
Delivery and review
Deliver OSC/MDC first with tests, man pages and focused `lljobstat` support.
The new selectors are `O`/`osc` and `-M`/`-mdc`. Focused changes to the
existing Python tool are allowed; the exception does not extend to new
Python software. Keep llite under the same feature LU for later review of
completed-byte accounting, attribution, memory and higher event rates.
Keep independent tiny-write accounting, queued-AIO accounting, job ID
cache invalidation and BRW retry changes outside this candidate. They are
not prerequisites for observing OSC/MDC requests. Do not expand this
feature to fix unrelated behavior.
Review interfaces, default behavior, memory accounting, visible loss,
operation mapping and attribution limits. Coordinate output with
LU-19768(https://jira.whamcloud.com/browse/LU-19768). Coordinate the
client enable control with existing server control work in
[review 70052](https://review.whamcloud.com/c/fs/lustre-release/+/70052).
Neither coordination item changes the client-only deployment requirement.
The test plan must cover disabled and enabled collection, old and current
servers, inline and bulk EOF reads, Data-on-MDT, partial batch completion,
large YAML output, actual `lljobstat` parsing, root-only access, memory
limits, allocation failures and concurrent logging/read/clear/unmount.
Test server behavior relative to the prerequisite baseline. Measure memory
and contention on representative large clients before making performance
claims. Report unsupported and unavailable cases directly.
- is related to
-
LU-16781 account jobids to original process
-
- Open
-
- is related to
-
LU-20984 Retroactive scanning of past patches for problems.
-
- Open
-
-
LU-19555 jobstats fails to cleanup timely on shutdown.
-
- Resolved
-
-
LU-21045 Add server-side control to disable jobstats collection independently of clients
-
- Open
-
-
LU-16781 account jobids to original process
-
- Open
-
-
LU-17000 Coverity and smatch static analysis issues
-
- Open
-
-
LU-19768 add jobstats, brw_stats, others to netlink/YAML output
-
- In Progress
-