[LU-9727] Lustre Audit with Changelogs

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.10.0
• Fix Version/s: Lustre 2.11.0, Lustre 2.12.0
• Type: New Feature
• Priority: Minor
• Reporter: Sebastien Buisson (Inactive)
• Assignee: Sebastien Buisson (Inactive)
• Resolution: Fixed
• Labels: feature, patch, sec

Description

Hi,

As Lustre Changelogs are a centralized mechanism reporting activity on the file system, we would like to use it as a basis for an audit facility for Lustre. The aim is to be able to track all accesses to files residing on Lustre, so that they can be recorded and looked up later for auditing purposes.

Changelogs cannot be used as-is to achieve auditing, because of the following limitations we have identified so far:
(a) uid/gid information is not recorded;
(b) OPEN and GEXATTR operations are not recorded;
(c) CLOSE operations are not recorded if the file is opened in READ_ONLY mode;
(d) Changelogs only record successful operations, not attempts.

Further comments on limitations:
(a) LU-1996 (https://review.whamcloud.com/4060) added support for jobid in Changelogs. If jobid is set to procname_uid, Changelogs will contain procname.uid information. So this could be used to know which user is doing the access. But jobid can be used for another purpose than audit, so we cannot always rely on it. We should create a new changelog extension similar to changelog_ext_jobid, that would hold uid/gid information.

(b) and (c) We do understand that it would have a performance cost to record OPEN and GEXATTR operations, as it would mean generating a write in the Changelogs for a read operation. Similarly for a CLOSE when a file is opened read-only. We will have to exclude OPEN and GETXATTR from the default Changelogs mask, and potentially create a dedicated changelogs entry type for the 'close on read-only' case, excluded by default. Moreover, we will evaluate the performance cost when these operations are recorded.

(d) Having all access attempts recorded will definitely increase MDS/MDT load, so we should examine carefully the performance impact of doing this. We would warn users about how much they would suffer by recording all access attempts.

I will feed this ticket by pushing patches to address the various limitations identified here (and possibly others to come).

Sebastien.

Issue Links

• is related to: LU-15373 changelog improvements tracking
• is related to: LU-11050 mdd: modifying changelog_deniednext d...
• is related to: LU-10450 NULL pointer deref in mdd_changelog_d...
• is related to: LU-10738 mdd: LBUG() from changelog_store_data...
• is related to: LU-10750 mdd_close() should check if changelog...
• is related to: LUDOC-391 document changelog audit feature
• is related to: LU-13308 update changelog_ext_nid to handle IP...
• is related to: LU-15372 add projid to changelog
• is related to: LU-10483 Replace FMODE_READ and FMODE_WRITE in...

Comments

[Andreas Dilger]

Sebastien, thanks for filing the ticket. Some comments on the various points:

• I don't think using JobID for audit is sufficient, since this information is extracted from the user process environment, and it would be trivial for the user to change this. This should use the process UID/GID sent in the RPC to the MDS.
• is it enough to record read-only OPEN operations for audit, or do you also need to record CLOSE operations in that case? We might consider checking at open time whether the operation is being audited, and then set a flag in the MDS open handle and then record the CLOSE only if the open was also recorded.
• it should optionally be possible to record the client NID along with the process UID/GID. That won't increase the ChangeLog overhead significantly, but can be useful for analysis.
• would it be enough to record the OPEN only once in the ChangeLog per JobID or UID/GID if there is an MPI job opening the same file thousands of times from different threads? That would reduce the ChangeLog load significantly, without significantly affecting the audit information.
• we could potentially enable/disable audit on a per-nodemap basis, so that it would be possible to have some nodes (e.g. backup, HSM agent nodes) that do not flood the audit logs.

[Sebastien Buisson (Inactive)]

Hi Andreas,

Thank you very much for your insight and your advice, this is very useful. I am in a preliminary stage, so I will be able to take your remarks into account for sure.
I might push the first patches in the next days/weeks.

Thanks,
Sebastien.

[Gerrit Updater]

Sebastien Buisson ([e-mail]) uploaded a new patch: https://review.whamcloud.com/27927
Subject: LU-9727 lustre: implement CL_OPEN for Changelogs
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: c651f0de79cc2f315fbf1afa1e5d274c28a6ea4c

[Gerrit Updater]

Sebastien Buisson ([e-mail]) uploaded a new patch: https://review.whamcloud.com/27929
Subject: LU-9727 lustre: record CLOSE if OPEN was recorded
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: d877e5ef8e6884e547752cfd7de0db09228b6340

[Gerrit Updater]

Matthew S ([e-mail]) uploaded a new patch: https://review.whamcloud.com/28045
Subject: LU-9727 lustre: Add an additional set of 64 changelog flags.
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 9aefdc58761a4b43178ca56c44b1176acfa60175

[Gerrit Updater]

Sebastien Buisson ([e-mail]) uploaded a new patch: https://review.whamcloud.com/28114
Subject: LU-9727 lustre: add uid/gid to Changelogs entries
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 0a0ee971ebe1cb4fe99d40810e10954fbe19c4e8

[Sebastien Buisson (Inactive)]

Hi Andreas,

> it should optionally be possible to record the client NID along with the process UID/GID. That won't increase the
> ChangeLog overhead significantly, but can be useful for analysis.

I have been looking into this, but I could not figure out how to retrieve the client's NID at the MDD layer level.
Did you have something in mind?

Or should I just extend the 'struct lu_ucred' to add the client's NID information? Unless it is already in the environment (struct lu_env), but I was not able to find it.

Thanks,
Sebastien.

[Andreas Dilger]

See for example https://review.whamcloud.com/27908 which is checking NIDs in the OST callpath to determine if they are local or remote.

[Gerrit Updater]

Sebastien Buisson ([e-mail]) uploaded a new patch: https://review.whamcloud.com/28213
Subject: LU-9727 lustre: add client NID to Changelogs entries
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 83674afcbcace9413024f2d7e3c1f0fb149d74a4

[Gerrit Updater]

Sebastien Buisson ([e-mail]) uploaded a new patch: https://review.whamcloud.com/28214
Subject: LU-9727 lustre: implement CL_OPEN for Changelogs
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 7f7414b752393b2a2b2b5d6ee1d77da7f9ffed4c

[Gerrit Updater]

Sebastien Buisson ([e-mail]) uploaded a new patch: https://review.whamcloud.com/28251
Subject: LU-9727 lustre: add CL_GETXATTR for Changelogs
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: b365e113c5f1a82c5cae9e82edced1e5dc4a88fc

[Gerrit Updater]

Sebastien Buisson ([e-mail]) uploaded a new patch: https://review.whamcloud.com/28299
Subject: LU-9727 lustre: limit OPEN and CLOSE rates in Changelogs
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 396a9869dc2d77c75cc76fa67b590e939ccffa75

[Gerrit Updater]

Sebastien Buisson ([e-mail]) uploaded a new patch: https://review.whamcloud.com/28313
Subject: LU-9727 nodemap: add audit_mode flag to nodemap
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 0b8bba030e42cb1929530eeb73f35dc03a78a685

[Gerrit Updater]

Sebastien Buisson ([e-mail]) uploaded a new patch: https://review.whamcloud.com/28314
Subject: LU-9727 lustre: record if enable_audit is set on nodemap
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: fa5d26a9f87d2c1e35c3734778e3e76f52cba6a8

[Peter Jones]

Sebastien

Are you able to share details about how you have validated the correct operation of this functionality?

Peter

[Gerrit Updater]

Sebastien Buisson ([e-mail]) uploaded a new patch: https://review.whamcloud.com/30315
Subject: LU-9727 utils: make llog_reader decode changelog fields
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 373f0e1cc4e6dc6a9d4bd290ba4245830c13373d

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/28045/
Subject: LU-9727 lustre: Add an additional set of 64 changelog flags.
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 08ffb6f1428fb0500e7befce5d50959658e768c6

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/28114/
Subject: LU-9727 lustre: add uid/gid to Changelogs entries
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 361edea4707254f4752ffd8c2db6c77a3ab9539c

[Gerrit Updater]

Sebastien Buisson ([e-mail]) uploaded a new patch: https://review.whamcloud.com/30943
Subject: LU-9727 tests: exercise new changelog fields and records
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: f43fab9c0e6b9e89f5d595072eb8e95929e85509

[Gerrit Updater]

Sebastien Buisson ([e-mail]) uploaded a new patch: https://review.whamcloud.com/30970
Subject: LU-9727 doc: update llog_reader man page for Changelogs
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: ce391d8c8a550826f84f23e1357ba27a3f493047

[Gerrit Updater]

Sebastien Buisson ([e-mail]) uploaded a new patch: https://review.whamcloud.com/30985
Subject: LU-9727 tests: adjust module load ordering
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: d0bdc1be41e554c99dd95f9cd0c46ce059d3f3a0

[Olaf Weber]

I just notices that the cr_prev field in changelog_rec is not used. Not used as in never set in the kernel.

In principle this means we could drop the changelog_ext_extra_flags extension and use that field (renamed) instead. Same as now the CLF_EXTRA_FLAGS flag would indicate that the contents of the field are valid and can be interpreted. It does appear we cannot assume the field is zeroed.

(Added) If we want to make this change it should be before a Lustre version with CLF_EXTRA_FLAGS in it ships.

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/28213/
Subject: LU-9727 lustre: add client NID to Changelogs entries
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: d2629cfcabfa3a22ddf4a6c474364d0012d80390

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/28214/
Subject: LU-9727 lustre: implement CL_OPEN for Changelogs
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 21fb4d93a94ef3876051fed31c5ef0c33f484f9d

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/27929/
Subject: LU-9727 lustre: record CLOSE if OPEN was recorded
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: afef52b9f2b5cb3af735d698883951fdd129af20

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/28313/
Subject: LU-9727 nodemap: add audit_mode flag to nodemap
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 942a9853f7b4c6fe22729468f1802ab782087e4e

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/28251/
Subject: LU-9727 lustre: add CL_GETXATTR for Changelogs
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: c722371c18809aaa1de36e5cb61a54de947611b4

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/28299/
Subject: LU-9727 lustre: limit OPEN and CLOSE rates in Changelogs
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: b45f8364a307d1b13ebaf5dc59da33bddde92769

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/28812/
Subject: LU-9727 lustre: record denied OPEN in Changelogs
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: ccb6fe4b5994c0b8e8890265acfa78e865592431

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/30315/
Subject: LU-9727 utils: make llog_reader decode changelog fields
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 8811869b1e88175d2ea6ead64f7c584b97db98bd

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/30970/
Subject: LU-9727 doc: update llog_reader man page for Changelogs
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: a13e325130d60b2bec46f67517fa46892e368337

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/28314/
Subject: LU-9727 lustre: record if enable_audit is set on nodemap
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 9dffcdd2fa07520aab89edd15f627518d3f6cff2

[Quentin Bouget]

Hi Sebastien,

Could you have a look at LU-10738? I think one of the patches for LU-9727 might have caused it.

Quentin

[Sebastien Buisson (Inactive)]

Hi Quentin,

I will try to have a look, thanks for letting me know.

Cheers.

[Gerrit Updater]

Sebastien Buisson ([e-mail]) uploaded a new patch: https://review.whamcloud.com/31456
Subject: LU-9727 mdd: properly call recording_changelog()
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 806a23eeaa1992ee343a382a402cd02d30a9b51e

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/31456/
Subject: LU-9727 mdd: properly call recording_changelog()
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: dfa318a29b8fe708468989d67ac6928a42bec72d

[Peter Jones]

Functionality has landed for 2.11. Let's track the landing of the tests under a new ticket. This can still land after code freeze and before GA if it is ready in time

[Gerrit Updater]

Quentin Bouget ([e-mail]) uploaded a new patch: https://review.whamcloud.com/32335
Subject: LU-9727 tests: exercise new changelog fields and records
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 5aeb9778b5943d05e27a5993d4faa7284ed829ff

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/32335/
Subject: LU-9727 tests: exercise new changelog fields and records
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 773804640a5d7bb9d106714096dab30cd873501c