[LU-10527] LustreError: 7830:0:(llog_cat.c:313:llog_cat_current_log()) ASSERTION( llh )

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.12.0, Lustre 2.10.7
• Severity: 3
• Assignee: Patrick Farrell (Inactive)
• Reporter: Bruno Faccini (Inactive)

Description

[894175.107791] LustreError: 21232:0:(mdd_dir.c:835:mdd_changelog_ns_store()) Skipped 71 previous similar messages
[894200.594288] Lustre: 21803:0:(llog_cat.c:91:llog_cat_new_log()) snx11253-MDD0000: there are no more free slots in catalog
[894200.606214] Lustre: 21803:0:(llog_cat.c:91:llog_cat_new_log()) Skipped 15668 previous similar messages
[894207.538039] LustreError: 21789:0:(mdd_dir.c:835:mdd_changelog_ns_store()) snx11253-MDD0000: cannot store changelog record: type = 2, name = 'LTPfstat01.9.PxDvLp.1496380725', t = [0x2800484d7:0xca6c:0x0], p = [0x280048380:0x1:0x0]: rc = -28
[894207.561210] LustreError: 21789:0:(mdd_dir.c:835:mdd_changelog_ns_store()) Skipped 166 previous similar messages
[894264.576106] Lustre: 19178:0:(llog_cat.c:91:llog_cat_new_log()) snx11253-MDD0000: there are no more free slots in catalog
[894264.588022] Lustre: 19178:0:(llog_cat.c:91:llog_cat_new_log()) Skipped 341 previous similar messages
[894271.531429] LustreError: 18644:0:(mdd_dir.c:835:mdd_changelog_ns_store()) snx11253-MDD0000: cannot store changelog record: type = 2, name = 'CL_LTPfcntl04.3.o1pIM1.1496380789', t = [0x2800484d7:0xcae9:0x0], p = [0x280048380:0x1:0x0]: rc = -28
[894271.554925] LustreError: 18644:0:(mdd_dir.c:835:mdd_changelog_ns_store()) Skipped 312 previous similar messages
[894333.383425] LustreError: 17029:0:(llog_cat.c:329:llog_cat_current_log()) snx11253-MDD0000: next log does not exist!
[894333.395645] LustreError: 7830:0:(llog_cat.c:313:llog_cat_current_log()) ASSERTION( llh ) failed:
[894333.405845] LustreError: 7830:0:(llog_cat.c:313:llog_cat_current_log()) LBUG
[894333.414189] Pid: 7830, comm: mdt_rdpg00_016
[894333.419667]
Call Trace:
[894333.426088]  [<ffffffffa0912853>] libcfs_debug_dumpstack+0x53/0x80 [libcfs]
[894333.434292]  [<ffffffffa0912f95>] lbug_with_loc+0x45/0xc0 [libcfs]
[894333.441728]  [<ffffffffa0a288c0>] llog_cat_add_rec+0xea0/0xf10 [obdclass]
[894333.449715]  [<ffffffff811c18ae>] ? kmem_cache_alloc_trace+0x1ce/0x1f0
[894333.457475]  [<ffffffffa0a2109a>] llog_add+0x7a/0x1a0 [obdclass]
[894333.464707]  [<ffffffffa114c8f2>] mdd_changelog_store+0xf2/0x270 [mdd]
[894333.472425]  [<ffffffffa115ea1a>] mdd_changelog_data_store+0x23a/0x330 [mdd]
[894333.480641]  [<ffffffffa115f95d>] mdd_close+0x29d/0xc90 [mdd]
[894333.487534]  [<ffffffffa1241500>] ? mdt_reint_create+0xcd0/0xd10 [mdt]
[894333.495203]  [<ffffffffa124be29>] mdt_mfd_close+0x2a9/0xa30 [mdt]
[894333.502390]  [<ffffffffa0a4f6d3>] ? class_handle2object+0xb3/0x1b0 [obdclass]
[894333.510584]  [<ffffffffa1251abb>] mdt_close+0x30b/0xd30 [mdt]
[894333.517578]  [<ffffffffa0d48dfb>] tgt_request_handle+0x8fb/0x11f0 [ptlrpc]
[894333.525370]  [<ffffffffa0ceb54b>] ptlrpc_server_handle_request+0x21b/0xa90 [ptlrpc]
[894333.533906]  [<ffffffffa0924848>] ? lc_watchdog_touch+0x68/0x180 [libcfs]
[894333.541595]  [<ffffffffa0ce8618>] ? ptlrpc_wait_event+0x98/0x330 [ptlrpc]
[894333.549282]  [<ffffffffa0ceee78>] ptlrpc_main+0xc08/0x1f40 [ptlrpc]
[894333.556423]  [<ffffffffa0cee270>] ? ptlrpc_main+0x0/0x1f40 [ptlrpc]
[894333.563524]  [<ffffffff810a5b8f>] kthread+0xcf/0xe0
[894333.569214]  [<ffffffff810a5ac0>] ? kthread+0x0/0xe0
[894333.574986]  [<ffffffff81644e18>] ret_from_fork+0x58/0x90
[894333.581176]  [<ffffffff810a5ac0>] ? kthread+0x0/0xe0
[894333.586921]
[894333.589184] Kernel panic - not syncing: LBUG
[894333.594215] CPU: 0 PID: 7830 Comm: mdt_rdpg00_016 Tainted: P           OE  ------------   3.10.0-327.36.3.x3.0.15.x86_64 #1
[894333.606087] Hardware name: Seagate Laguna Seca/Laguna Seca, BIOS v02.0024 02/26/2016
[894333.614589]  ffffffffa0937ca3 00000000efc7dcf7 ffff880d5fc6b9c8 ffffffff816349d1
[894333.622804]  ffff880d5fc6ba48 ffffffff8162e260 ffffffff00000008 ffff880d5fc6ba58
[894333.631015]  ffff880d5fc6b9f8 00000000efc7dcf7 ffffffffa0aa9d00 ffffffffa0943525
[894333.639218] Call Trace:
[894333.642397]  [<ffffffff816349d1>] dump_stack+0x19/0x1b
[894333.648253]  [<ffffffff8162e260>] panic+0xd8/0x1e7
[894333.653749]  [<ffffffffa0912ffb>] lbug_with_loc+0xab/0xc0 [libcfs]
[894333.660640]  [<ffffffffa0a288c0>] llog_cat_add_rec+0xea0/0xf10 [obdclass]
[894333.668112]  [<ffffffff811c18ae>] ? kmem_cache_alloc_trace+0x1ce/0x1f0
[894333.675322]  [<ffffffffa0a2109a>] llog_add+0x7a/0x1a0 [obdclass]
[894333.681990]  [<ffffffffa114c8f2>] mdd_changelog_store+0xf2/0x270 [mdd]
[894333.689168]  [<ffffffffa115ea1a>] mdd_changelog_data_store+0x23a/0x330 [mdd]
[894333.696864]  [<ffffffffa115f95d>] mdd_close+0x29d/0xc90 [mdd]
[894333.703245]  [<ffffffffa1241500>] ? mdt_reint_create+0xcd0/0xd10 [mdt]
[894333.710397]  [<ffffffffa124be29>] mdt_mfd_close+0x2a9/0xa30 [mdt]
[894333.717101]  [<ffffffffa0a4f6d3>] ? class_handle2object+0xb3/0x1b0 [obdclass]
[894333.724834]  [<ffffffffa1251abb>] mdt_close+0x30b/0xd30 [mdt]
[894333.731188]  [<ffffffffa0d48dfb>] tgt_request_handle+0x8fb/0x11f0 [ptlrpc]
[894333.738669]  [<ffffffffa0ceb54b>] ptlrpc_server_handle_request+0x21b/0xa90 [ptlrpc]
[894333.746918]  [<ffffffffa0924848>] ? lc_watchdog_touch+0x68/0x180 [libcfs]
[894333.754319]  [<ffffffffa0ce8618>] ? ptlrpc_wait_event+0x98/0x330 [ptlrpc]
[894333.761770]  [<ffffffffa0ceee78>] ptlrpc_main+0xc08/0x1f40 [ptlrpc]

Issue Links

• is related to: LU-11943 many input/output error after soak running for couple of hours
• is related to: LU-7340 ChangeLogs catalog full condition should be handled more gracefully