service160.bz2
service160.gz
[LU-1524] Parent doesn't exist Created: 14/Jun/12 Updated: 21/Mar/13 Resolved: 21/Mar/13 |
|
| Status: | Resolved |
| Project: | Lustre |
| Component/s: | None |
| Affects Version/s: | Lustre 2.1.1 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major |
| Reporter: | Mahmoud Hanafi | Assignee: | Zhenyu Xu |
| Resolution: | Not a Bug | Votes: | 0 |
| Labels: | None | ||
| Attachments: |
service160.bz2
service160.gz
|
| Severity: | 3 |
| Rank (Obsolete): | 4045 |
| Description |
|
mds starts to report "parent doesn't exist" Load in the mds became very high we ending up dumping the server. Have vmcore if needed. Could be a dup of Lustre: 4504:0:(mdt_handler.c:888:mdt_getattr_name_lock()) header@ffff880c18139480[0x0, 1, [0x325698b32c:0x9:0x0] hash lru] { ^M Lustre: 4504:0:(mdt_handler.c:888:mdt_getattr_name_lock()) ....mdt@ffff880c181394d8mdt-object@ffff880c18139480(ioepoch=0 flags=0x0, epochcount=0, writecount=0)^M Lustre: 4504:0:(mdt_handler.c:888:mdt_getattr_name_lock()) ....cmm@ffff880b36c71d40[local]^M Lustre: 4504:0:(mdt_handler.c:888:mdt_getattr_name_lock()) ....mdd@ffff8808ceb92a40mdd-object@ffff8808ceb92a40(open_count=0, valid=0, cltime=0, flags=0)^M Lustre: 4504:0:(mdt_handler.c:888:mdt_getattr_name_lock()) ....osd-ldiskfs@ffff8808ceb92980osd-ldiskfs-object@ffff8808ceb92980(i:(null):0/0)[plain]^M Lustre: 4504:0:(mdt_handler.c:888:mdt_getattr_name_lock()) } header@ffff880c18139480^M Lustre: 4946:0:(mdt_xattr.c:375:mdt_reint_setxattr()) client miss to set OBD_MD_FLCTIME when setxattr: [object [0x2f00600666:0x44:0x0]] [valid 68719476736]^M |
| Comments |
| Comment by Peter Jones [ 14/Jun/12 ] |
|
Bobijam Is this a duplicate of Peter |
| Comment by Zhenyu Xu [ 14/Jun/12 ] |
|
Mahmoud, What situation does this issue happen? During MDS recovery? Or what operaions made MDS start to dumping this info and hung there? |
| Comment by Mahmoud Hanafi [ 14/Jun/12 ] |
|
This happen after the Parent's... message. The MDS was not in recovery. but the same mds after recover is again seeing a high load. Jun 14 19:26:50 nbp2-mds kernel: Lustre: nbp2-MDT0000: sending delayed replies to recovered clients |
| Comment by Zhenyu Xu [ 15/Jun/12 ] |
|
Looks like the OSTs are busy deleteing orphan objects, and MDS's busy waiting for fid sequence update. Wangdi, could you help to have a look? |
| Comment by Zhenyu Xu [ 15/Jun/12 ] |
|
Whether did you observe that the MDS keep high load after recovery or did the high load keep a short time after recovery? Would you mind uploading vmcore or sysrq-trigger output, we'd like to know the whole picture of this situation. (lov_request.c:569:lov_update_create_set()) error creating fid 0x1 sub-object on OST idx 2/1: rc = -107) means the 1st object hasn't created on this OST yet, is this a production system or a testing system? |
| Comment by Mahmoud Hanafi [ 19/Jun/12 ] |
|
We hit this issue again. Uploading console logs with call traces. We are not able to upload the vmcore due to security. |
| Comment by Di Wang [ 20/Jun/12 ] |
|
According to the log. there is a LBUG here. <0>LustreError: 11533:0:(lu_object.c:113:lu_object_put()) ASSERTION(cfs_list_empty(&top->loh_lru)) failed^M Seems like 1013. |
| Comment by Mahmoud Hanafi [ 20/Jun/12 ] |
|
Yes the LBUG was after the system was rebooted and that is a different issue Sent from my iPhone |
| Comment by Di Wang [ 20/Jun/12 ] |
|
Yeah, saw that. btw: the MDT is almost full? |
| Comment by Di Wang [ 20/Jun/12 ] |
|
According to the console log. it seems the journal can not be synced to disk in time. See Pid: 18717, comm: mdt_mdss_343^M Then all other processes just wait there to get enough journal credit to start the journal. Hmm I do not know why the journal can not be synced to disk in time. Are you using internal or external journal for your system? Actually in And also we saw a lot lnet error msg here Lustre: 2056:0:(o2iblnd_cb.c:2326:kiblnd_passive_connect()) Conn stale 10.151.59.167@o2ib [old ver: 12, new ver: 12] Which usually means client is being reboot. Is that true? And how many clients in your system? Thanks. |
| Comment by Mahmoud Hanafi [ 20/Jun/12 ] |
|
As for disk space we have lots free We also noticed the slowness in the journal. We have check the RAID subsystem no issue were found. We do have a lot of clients over 10000. We are not planing to upgrade to 2.2 but we are looking at 2.1.2. We may be able to pull-in |
| Comment by Di Wang [ 20/Jun/12 ] |
|
Hmm, I found some error msg LustreError: 12820:0:(llog_cat.c:298:llog_cat_add_rec()) llog_write_rec -28: lh=ffff88014c7a2ec0^M Ah, that is because current log handle does not have space for the record, instead of the disk space. Hmm, the error msg is a little misleading. I will correct it a bit.
So there are more than 10k clients, do you expect they are being reboot during the time you collecting the console log, since we see a lot Lustre: 2056:0:(o2iblnd_cb.c:2326:kiblnd_passive_connect()) Conn stale 10.151.59.167@o2ib [old ver: 12, new ver: 12] in the message you upload. Also it seems you also met a lot LustreError: 18929:0:(mdt_recovery.c:793:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 171918688025, new: 171918688024 replay: 0. see Unfortunately, the fix is also landed on 2.2. (the patch on 2.1 is just a workaround fix). |
| Comment by Mahmoud Hanafi [ 20/Jun/12 ] |
|
does kiblnd_passive_connect always mean the client is being reconnecting from a reboot? Or is it just it's reconnecting, may due to fabric issue. |
| Comment by Di Wang [ 20/Jun/12 ] |
|
That is what I learned, but I am not lnet expert. Liang, could you please comment here? |
| Comment by Peter Jones [ 29/Jun/12 ] |
|
Added Liang as a watcher so that he can see Wang Di's question |
| Comment by Liang Zhen (Inactive) [ 29/Jun/12 ] |
I think we generate these messages only for rebooted remote peers. |
| Comment by Peter Jones [ 21/Mar/13 ] |
|
As per NASA ok to close ticket |