[LU-6696] ASSERTION( rc == 0 || rc == LLOG_PROC_BREAK ) failed: 0 changes, 0 in progress, 0 in flight: -5

• Type: Bug
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.9.0
• Affects Version/s: Lustre 2.5.3, Lustre 2.8.0
• Severity: 2
• Assignee: Zhenyu Xu
• Reporter: Mahmoud Hanafi

Description

LustreError: 11-0: hw_nb-OST0016-osc-MDT0000: Communicating with 10.151.26.55@o2ib, operation ost_connect failed with -114.
LustreError: 6488:0:(llog_cat.c:866:llog_cat_init_and_process()) hw_nb-OST0024-osc-MDT0000: llog_process() with cat_cancel_cb failed: rc = -5
LustreError: 6580:0:(osp_sync.c:874:osp_sync_thread()) ASSERTION( rc == 0 || rc == LLOG_PROC_BREAK ) failed: 0 changes, 0 in progress, 0 in flight: -5
LustreError: 6580:0:(osp_sync.c:874:osp_sync_thread()) LBUG
Pid: 6580, comm: osp-syn-36-0

Call Trace:
 [<ffffffffa05cf895>] libcfs_debug_dumpstack+0x55/0x80 [libcfs]
 [<ffffffffa05cfe97>] lbug_with_loc+0x47/0xb0 [libcfs]
 [<ffffffffa10d9243>] osp_sync_thread+0x753/0x7d0 [osp]
 [<ffffffff81559b9e>] ? thread_return+0x4e/0x770
 [<ffffffffa10d8af0>] ? osp_sync_thread+0x0/0x7d0 [osp]

Entering kdb (current=0xffff8803b5e04080, pid 6580) on processor 3 Oops: (null)
due to oops @ 0x0
kdba_dumpregs: pt_regs not available, use bt* or pid to select a different task
[3]kdb> 

Issue Links

• is related to: LU-9068 Hardware problem resulting in bad blocks
• is related to: LU-8252 MDS kernel panic after aborting journal
• is related to: LU-7011 Kernel part of llog subsystem can do self-repairing in some cases
• is related to: LU-5056 osp_sync_thread()) ASSERTION( rc == 0 || rc == LLOG_PROC_BREAK ) failed: 6 changes, 8 in progress, 0 in flight: -5

Comments

[Peter Jones]

ok - thanks Mahmoud

[Mahmoud Hanafi]

I fixed the corrupted llog catalog and was able to mount the filesystem. You may lower the prio.

[Mikhail Pershin]

I've just pushed tool to fix corrupted llog catalog, it might be helpful in such cases to revive server quickly. Of course we can just destroy corrupted llog, but there will be orphaned plain llogs. So anyway this tool is helpful to show all valid records to find out IDs of plain llogs to delete.

[Gerrit Updater]

Mike Pershin ([e-mail]) uploaded a new patch: http://review.whamcloud.com/15247
Subject: LU-6696 llog: tool to fix corrupted llog catalog
Project: fs/lustre-release
Branch: b2_5
Current Patch Set: 1
Commit: 289722425f1195bc2438cd65dd9bcfa0243fcf46

[Gerrit Updater]

Mike Pershin ([e-mail]) uploaded a new patch: http://review.whamcloud.com/15245
Subject: LU-6696 llog: tool to fix corrupted llog catalog
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 000ed45ddda77930f00cf3d2791e9af703aab95d

[Mikhail Pershin]

Upload fixed llog catalog file

[Zhenyu Xu]

Mikhail, can you upload the llog?

[Jay Lan (Inactive)]

My manager asks to raise the priority (currently 3) because the production filesystem is not available.

[Mikhail Pershin]

well, llog is corrupted in some strange way, meanwhile I've found that llog contained 4 records with indeces 61,62,63,64. Llog itself contains only 3 records 62, 63 and 64. And everything before those records are just garbage. I've fixed llog manually so it looks healthy now and contains those three records:

# lustre/utils/llog_reader cb2000d_9c396a65_fixed 
Bit 0 of 3 not set
rec #62 type=1064553b len=64
rec #63 type=1064553b len=64
rec #64 type=1064553b len=64
Header size : 8192
Time : Fri Nov  7 09:00:21 2008
Number of records: 3
Target uuid :  
-----------------------
#62 (064)ogen=0 name=0x3bf:1
#63 (064)ogen=0 name=0x419:1
#64 (064)ogen=0 name=0x448:1

That might help to revive MDS with access at least to those plain llogs.

[Mikhail Pershin]

Mahmoud, the llog looks empty, can you upload it again and gzip it before, please?
That was my browser issue, false alarm. I have the file.

[Mahmoud Hanafi]

attached llog file to the LU.