https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-959 Lustre <p>It seams the kuc channels are not reestablished after a MDS crash.<br/> In particular, process that are listening for changelogs remain stuck listening to the kernel-&gt;userspace pipe<br/> whereas no message is sent from the MDS, as clients do not re-register kuc listeners after reconnecting to the MDS.</p> <p>It would probably need an action in mdc_import_event() to reregister kuc listeners,<br/> something like:</p> <p>in mdc_import_event():</p> <p> case IMP_EVENT_ACTIVE: {<br/> rc = obd_notify_observer(obd, obd, OBD_NOTIFY_ACTIVE, NULL);<br/> + /* restore re-establish kuc registration after reconnecting */<br/> + if (rc == 0)<br/> + rc = mdc_kuc_reregister(imp);</p> LU-959

kuc channels not reestablished after MDS crash

Bug Minor Closed Cannot Reproduce Niu Yawei Thomas LEIBOVICI - CEA Wed, 4 Jan 2012 05:40:33 +0000 Wed, 11 Apr 2012 02:52:27 +0000 Wed, 11 Apr 2012 02:52:27 +0000 Lustre 2.0.0 Lustre 2.1.0 0 3 <p>Niu</p> <p>Could you please look into this one</p> <p>Thanks</p> <p>Peter</p> <p>Hi, Thomas</p> <p>I don't quite follow your description of this ticket, and I didn't find mdc_kuc_reregister() neither. Could you elaborate this ticket a little bit more or post the patch on gerrit for review? Thank you.</p> <p>OK, I'll try to explain the issue with more details.</p> <p>To receive MDT changelogs from a client, llapi_changelog_start() is called by the user space program (like lfs):<br/> 1) this creates a "kuc" channel (communuication channel between mdc and user space, which is implemented in libcfs/libcfs/kernel_user_comm.c)<br/> 2) it calls an ioctl which results in calling mdc_ioc_changelog_send() in mdc that sends an RPC to MDS, to notify it it must send CL records to this client.<br/> Then, the user space program calls llapi_changelog_recv() to get changelog records. This listens to incoming data from the kuc channel.</p> <p>The problem is there is no recovery mechanism of KUC channels when the MDS restarts:<br/> the client remains blocked in llapi_changelog_recv() an no more data is sent from the MDS (it forgot a client was listening for changelogs).<br/> I think there should be an internal mechanism in MDC to call mdc_ioc_changelog_send() again after a MDC/MDS reconnection.</p> <p>This is what I suggested: a mdc_kuc_reregister() should be implemented to be called in mdc_import_event(),<br/> so mdc_ioc_changelog_send() it called for each registered process in the kuc layer.</p> <p>Do you have a better understanding of this issue?<br/> Thanks</p> <p>Thanks a lot for the details, Thomas. I think I have much better understanding now, but I still don't see why client was blocked in llapi_changelog_recv() after MDS restarted: mdc_ioc_changelog_send() just use the llog APIs to read changelog on MDS then put it in the pipe, so when MDS restarts, no matter if the client llog process procedure break earlier for an RPC error, the CL_EOF will always be written, and llapi_changelog_recv() should receive this EOF record and break reading.</p> <p>Do you have the debug log and stack trace when the process stuck in llapi_changelog_recv()? </p> <p>Right, I see what you mean. Maybe my initial understanding of the problem is wrong.<br/> Unfortunately, I have no stack for you right now. Just something we noticed...<br/> So we'll have to wait for the next MDS crash to get a detailed stack, which will be hopefully not too soon <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Thomas, is it still relevant? can we close it?</p> <p>OK, let's close it. I'll reopen it in case of new occurrence.</p> <p>not reproduced, close it for now.</p> Development Rank 1|hzvhlb: Rank (Obsolete) 6499 Severity