|
Bobijam
Could you please advise here? Is this related to LU-5056 perhaps?
Peter
|
|
Can I have debug logs?
|
|
the mds crashes so I can't give any debug logs.
Do you want them from the oss? what specific debug settings?
|
|
I want to know the cause of the llog process failure, can you upload logs from OST0024?
|
|
I uploaded logs (lbug.OSS1.out.gz) to ftp site: /uploads/LU6696/lbug.OSS1.out.gz
|
|
the OSS shows that it cannot access MGS service
Are MDS and MGS different nodes, or MDT and MGT locate on different devices? It seems that it's MGS does not response and the llog process cannot move on.
Another thing to notice is that llog_cat_init_and_process() just discards the return value of llog_process_or_fork(), which makes the osp_sync_init() ignores the error in its osp_sync_llog_init(), I think this part also need fix.
|
|
The MDS and MGS where located on the same device. We got the errors. as part of debugging I moved the mgs and mdt to different devices. Did a tunefs.lustre --writeconf but got the same error.
What is the fix for the osp_sync_llog_init()
|
|
btw, it is only the single OST that is causing the LBUG.
|
|
I uploaded debug logs from mdt
ftp:/uploads/LU6696/lustre-log.1433912973.6315.txt
|
|
from the MDT log
We can see that the llog for this device (OST00024) is corrupted (looking for llog indexed 61, and skipped over to index 63, and find out that the reading offset (12224) is over the llog file size, and -EIO is returned.
Tappro, how can I disable the llog process of this OST device?
|
|
there is no way to stop llog processing, except the llog removal by hands. Meanwhile the record type 0x10645539 is llog_header and it is correct that its lrh_index is 0, so probably llog is not corrupted. The question is why header lies at offset 8192 or maybe block was read from offset 0 somehow? Do you know llog ID? Then it can be found in /O/ directory and analyzed, anyway we have to find it. Don't remove this llog, we need its header with all data in it at first.
|
|
00000004:00000040:1.0:1433912973.463121:0:6221:0:(osp_sync.c:949:osp_sync_llog_init()) hw_nb-OST0024-osc-MDT0000: Init llog for 36 - catid 0xcb2000d:1:9c396a65
So the llog ID is 0xcb2000d:1:9c396a65 ?
From code llog_process_thread(), the cur_offset is initialized as LLOG_CHUNK_SIZE, so the block was read from 8192, I don't quite know about llog somehow.
|
|
How do I get the llog ID?
|
|
nbphw-mds /mnt/lustre/hw_mdt/OBJECTS # llog_reader cb2000d:9c396a65
rec #0 type=10645539 len=8192
The log is corrupt (too big at 0)
Could not pack buffer; rc=-22
|
|
It should be possible to improve the error handling in this code so that it isn't an LASSERT(), and instead returns an error to the caller. We shouldn't have LASSERT() checks on data that comes from the disk.
|
|
Can you post that llog file here, please?
|
|
It looks like llog has another (or the same) header written from 8192 offset. That is wrong and I'd like to investigate this to understand how that was possible.
Andreas, I agree, OSP code is quite aggressive towards possible IO errors
|
|
attached llog file to the LU.
|
|
Mahmoud, the llog looks empty, can you upload it again and gzip it before, please?
That was my browser issue, false alarm. I have the file.
|
|
well, llog is corrupted in some strange way, meanwhile I've found that llog contained 4 records with indeces 61,62,63,64. Llog itself contains only 3 records 62, 63 and 64. And everything before those records are just garbage. I've fixed llog manually so it looks healthy now and contains those three records:
# lustre/utils/llog_reader cb2000d_9c396a65_fixed
Bit 0 of 3 not set
rec #62 type=1064553b len=64
rec #63 type=1064553b len=64
rec #64 type=1064553b len=64
Header size : 8192
Time : Fri Nov 7 09:00:21 2008
Number of records: 3
Target uuid :
-----------------------
#62 (064)ogen=0 name=0x3bf:1
#63 (064)ogen=0 name=0x419:1
#64 (064)ogen=0 name=0x448:1
That might help to revive MDS with access at least to those plain llogs.
|
|
My manager asks to raise the priority (currently 3) because the production filesystem is not available.
|
|
Mikhail, can you upload the llog?
|
|
Upload fixed llog catalog file
|
|
Mike Pershin (mike.pershin@intel.com) uploaded a new patch: http://review.whamcloud.com/15245
Subject: LU-6696 llog: tool to fix corrupted llog catalog
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 000ed45ddda77930f00cf3d2791e9af703aab95d
|
|
Mike Pershin (mike.pershin@intel.com) uploaded a new patch: http://review.whamcloud.com/15247
Subject: LU-6696 llog: tool to fix corrupted llog catalog
Project: fs/lustre-release
Branch: b2_5
Current Patch Set: 1
Commit: 289722425f1195bc2438cd65dd9bcfa0243fcf46
|
|
I've just pushed tool to fix corrupted llog catalog, it might be helpful in such cases to revive server quickly. Of course we can just destroy corrupted llog, but there will be orphaned plain llogs. So anyway this tool is helpful to show all valid records to find out IDs of plain llogs to delete.
|
|
I fixed the corrupted llog catalog and was able to mount the filesystem. You may lower the prio.
|
|
ok - thanks Mahmoud
|
|
Mike, thanks for getting this tool working for the customer so quickly.
It would be more useful in the long run if the kernel llog code would just skip the corrupted records itself, and/or have LFSCK repair them before use. That allows the filesystem to keep working, rather than taking an outage and requiring the admin to even figure out such a tool exists and have to run it, instead of the kernel dealing with this problem directly.
|
|
Hit this bug on master branch, replay-single test_60 failed. lustre-master build# 3175 RHEL7 DNE
https://testing.hpdd.intel.com/test_logs/d35a0490-54ed-11e5-9cd2-5254006e85c2/show_text
|
|
I wonder about -115 (EINPROGRESS) error code and think it is from obd_fid_alloc() which may do RPC to the master MDT. While we need better error handling in OSP, in this particular case I think it is also not right to return -EINPROGRESS from the FID/SEQ code at all, it should be handled inside.
|
|
We can close this LU.
|
|
It doesn't appear that either http://review.whamcloud.com/15250 or http://review.whamcloud.com/15247 have landed to master. Are these patches no longer needed (and should be abandoned) because of different patches to master, or do they need to be ported to master?
|
|
Andreas, the 15250 is not in master, so can be ported, 15247 has master patch which is 15245 and is tracked under LU-7011, it is not landed but will not be lost after the closing of this ticket.
|
|
Bobi Jam (bobijam@hotmail.com) uploaded a new patch: http://review.whamcloud.com/19856
Subject: LU-6696 llog: improve error handling
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 12153490536bb3f1049631720b3629de68ad8574
|
|
Oleg Drokin (oleg.drokin@intel.com) merged in patch http://review.whamcloud.com/19856/
Subject: LU-6696 llog: improve error handling
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 53d2f414d75ac1302b53017376ca2f1fda1f3d17
|
|
Patch has landed to master for 2.9.0.
The tool patch is being tracked by LU-7011
|
Generated at Sat Feb 10 02:02:27 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.
cb2000d_9c396a65