[LU-7340] ChangeLogs catalog full condition should be handled more gracefully

• Type: Improvement
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.11.0, Lustre 2.10.3
• Affects Version/s: Lustre 2.8.0, Lustre 2.5.4
• Assignee: Bruno Faccini (Inactive)
• Reporter: Bruno Faccini (Inactive)

Description

Presently when a LLOG Catalog wraps and its latest assigned index collides with its oldest and still in use index, ENOSPC is returned and the caller just ignore the fact that LLOG record could not be written.

For ChangeLogs specific usage, some actions could be attempted to recover space/records, some ideas have already been detailed in LU-6556, but it seems better to address them in this separate ticket.

Input from Andreas :
I think the other thing that is needed here is to automatically unregister ChangeLog watcher(s) if the changelog is full or the MDS runs out of space (by default), or block all MDS operations until the ChangeLog can be written (if /proc tunable is set to make ChangeLog updates mandatory). It should unregister starting with the oldest watcher on the assumption that the older watcher was forgotten and newer ones are still running, and that this will release the most space. The unregistration should cancel records up to the next watcher, or all remaining records if no other watchers are left.

Input from Robert :
I suggest going a step further and proactively remove stale watchers after a configurable period or when hitting a max watermark to try o avoid running out of space. Also, being unregistered is a reasonable notification to the application that they've lost their changelog feed and need to resync.

Issue Links

• is duplicated by: LU-1586 no free catalog slots for log
• is related to: LU-8856 ZFS-MDT 100% full. Cannot delete files.
• is related to: LU-10527 LustreError: 7830:0:(llog_cat.c:313:llog_cat_current_log()) ASSERTION( llh )
• is related to: LU-10680 MDT becoming unresponsive in 2.10.3
• is related to: LU-12871 enable changelog garbage collection by default
• is related to: LU-9055 MDS crash due to changelog being full