[LU-6556] changelog catalog corruption if all possible records is define

• Type: Bug
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.8.0
• Affects Version/s: Lustre 2.5.3

Environment

redhat kernel 2.6.32_504.8.1.el6
lustre 2;5.3.90 + LU-5740 LU-4582 LU-5678 LU-5393 LU-3727 LU-4528 LU-5522 LU-5264 LU-6049 LU-6084 LU-5764

• Severity: 3
• Assignee: Bruno Faccini (Inactive)
• Reporter: Antoine Percher

Description

After our last lustre upgrade, On tera100 and tgcc site, some
lustre fs have meet the same corruption on the changelog_catalog
The robinhood node panic like in the LU-6471 and the crash analyze
show that is the changelog-catalog file that have a corruption.
The file is too big than the maximum size of this type of file and
the record who produces the panic is not in the right place.

Issue Links

• is related to: LU-7138 LBUG: (osd_handler.c:1017:osd_trans_start()) ASSERTION( get_current()->journal_info == ((void *)0) ) failed:
• is related to: LU-6634 (osd_handler.c:901:osd_trans_start()) ASSERTION( get_current()->journal_info == ((void *)0) ) failed: when reaching Catalog full condition
• is related to: LU-7241 sanity test_60a: there are no more free slots in catalog
• is related to: LU-6954 LustreError: 12934:0:(mdd_device.c:305:mdd_changelog_llog_init()) fsrzb-MDD0000: changelog init failed: rc = -5
• is related to: LU-4528 osd_trans_exec_op()) ASSERTION( oti->oti_declare_ops_rb[rb] > 0 ) failed: rb = 0
• is related to: LU-7329 sanity test_60a timeouts with “* invoking oom-killer”

Comments

[Bruno Faccini (Inactive)]

I have created LU-6634 because during my testing of my patch for this/LU-6556 ticket, I wanted to check what happen (expected ENOSPC return!) after Catalog has looped-back and fills up, but I got a "(osd_handler.c:901:osd_trans_start()) ASSERTION( get_current()->journal_info == ((void *)0) ) failed:" LBUG. I have identified this to be caused by the fact that in the error path for llog_cat_new_log(), llog_destroy() is called to destroy the new plain LLOG for which the reference can't be recorded into Catalog because there is no slot available to do so, and this triggers the Assertion because there is already a started transaction from llog_cat_add(), when llog_destroy() wants to start its own transaction.

[Gerrit Updater]

Faccini Bruno ([e-mail]) uploaded a new patch: http://review.whamcloud.com/14912
Subject: LU-6556 obdclass: re-allow catalog loopback
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 8112eb1a2d3dd5beed00c75d76295ae1362ab19a

[Antoine Percher]

Please find in attachment customer trace analyze file (log_lu-6556_b.txt)
to understand the changelog-catalog coruption

[Bruno Faccini (Inactive)]

Peter, no in this particular case the similar LBUG than for LU-6471 is valid since it is triggered due to the ChangeLog Catalog corruption I just explained before in my previous comment.

[Bruno Faccini (Inactive)]

Assigning to me since I have been working on this issue when being on-site with Antoine.

In fact, after our joint analysis, it seems that the crash has occurred a few time after the upgrade+reboot of all nodes because of the 2 combined things :
1) a ChangeLog Catalog that has already looped-back
2) integrating/running with LU-4528 patch that seems to have introduced a regression where looped-back Catalogs are not handled correctly and are only expected to grow when it is not the case.

Will push a patch soon to master where the problem seems to be also present (but still undetected due to the need of a looped-back Catalog situation to trigger which should not occur so frequently ...).

[Peter Jones]

Bruno

Can you please confirm whether this does indeed meet the profile of LU-6471?

Thanks

Peter