[LU-6696] ASSERTION( rc == 0 || rc == LLOG_PROC_BREAK ) failed: 0 changes, 0 in progress, 0 in flight: -5

• Type: Bug
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.9.0
• Affects Version/s: Lustre 2.5.3, Lustre 2.8.0
• Severity: 2
• Assignee: Zhenyu Xu
• Reporter: Mahmoud Hanafi

Description

LustreError: 11-0: hw_nb-OST0016-osc-MDT0000: Communicating with 10.151.26.55@o2ib, operation ost_connect failed with -114.
LustreError: 6488:0:(llog_cat.c:866:llog_cat_init_and_process()) hw_nb-OST0024-osc-MDT0000: llog_process() with cat_cancel_cb failed: rc = -5
LustreError: 6580:0:(osp_sync.c:874:osp_sync_thread()) ASSERTION( rc == 0 || rc == LLOG_PROC_BREAK ) failed: 0 changes, 0 in progress, 0 in flight: -5
LustreError: 6580:0:(osp_sync.c:874:osp_sync_thread()) LBUG
Pid: 6580, comm: osp-syn-36-0

Call Trace:
 [<ffffffffa05cf895>] libcfs_debug_dumpstack+0x55/0x80 [libcfs]
 [<ffffffffa05cfe97>] lbug_with_loc+0x47/0xb0 [libcfs]
 [<ffffffffa10d9243>] osp_sync_thread+0x753/0x7d0 [osp]
 [<ffffffff81559b9e>] ? thread_return+0x4e/0x770
 [<ffffffffa10d8af0>] ? osp_sync_thread+0x0/0x7d0 [osp]

Entering kdb (current=0xffff8803b5e04080, pid 6580) on processor 3 Oops: (null)
due to oops @ 0x0
kdba_dumpregs: pt_regs not available, use bt* or pid to select a different task
[3]kdb> 

Issue Links

• is related to: LU-9068 Hardware problem resulting in bad blocks
• is related to: LU-8252 MDS kernel panic after aborting journal
• is related to: LU-7011 Kernel part of llog subsystem can do self-repairing in some cases
• is related to: LU-5056 osp_sync_thread()) ASSERTION( rc == 0 || rc == LLOG_PROC_BREAK ) failed: 6 changes, 8 in progress, 0 in flight: -5

Comments

[Joseph Gmitter (Inactive)]

Patch has landed to master for 2.9.0.

The tool patch is being tracked by LU-7011

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/19856/
Subject: LU-6696 llog: improve error handling
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 53d2f414d75ac1302b53017376ca2f1fda1f3d17

[Gerrit Updater]

Bobi Jam ([e-mail]) uploaded a new patch: http://review.whamcloud.com/19856
Subject: LU-6696 llog: improve error handling
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 12153490536bb3f1049631720b3629de68ad8574

[Mikhail Pershin]

Andreas, the 15250 is not in master, so can be ported, 15247 has master patch which is 15245 and is tracked under LU-7011, it is not landed but will not be lost after the closing of this ticket.

[Andreas Dilger]

It doesn't appear that either http://review.whamcloud.com/15250 or http://review.whamcloud.com/15247 have landed to master. Are these patches no longer needed (and should be abandoned) because of different patches to master, or do they need to be ported to master?

[Mahmoud Hanafi]

We can close this LU.

[Mikhail Pershin]

I wonder about -115 (EINPROGRESS) error code and think it is from obd_fid_alloc() which may do RPC to the master MDT. While we need better error handling in OSP, in this particular case I think it is also not right to return -EINPROGRESS from the FID/SEQ code at all, it should be handled inside.