Whamcloud Community JIRA

Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us

9.4.14

940014

05-12-2023

[LU-12198] lnetctl peer show hangs for ~2600 clients, ioctl getting E2BIG https://jira.whamcloud.com/browse/LU-12198 Lustre <p>command `lnetctl peer show` appears to hang, strace shows looping on:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>ioctl(3, _IOC(_IOC_READ|_IOC_WRITE, 0x65, 0x64, 0xb8), 0x7fffffffccf0) = -1 E2BIG (Argument list too long) </pre> </div></div> <p>There are 2605 lines in /sys/kernel/debug/lnet/peers.</p> x86 servers, 2.12 no patches, RHEL 7.6 LU-12198

lnetctl peer show hangs for ~2600 clients, ioctl getting E2BIG

Bug Minor Resolved Fixed Dominique Martinet Ruth Klundt Thu, 18 Apr 2019 20:19:05 +0000 Mon, 22 Jun 2020 17:46:17 +0000 Tue, 25 Feb 2020 14:21:51 +0000 Lustre 2.12.0 Lustre 2.14.0 Lustre 2.12.5 1 10 <p>Yes, this is because we have this limit of 1000 peers while allocating buffer to get the peer list- </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> 2996 count = 1000; 2997 size = count * sizeof(struct lnet_process_id); 2998 list = malloc(size); </pre> </div></div> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> 009 LIBCFS_IOC_INIT_V2(peer_info, prcfg_hdr); 3010 peer_info.prcfg_hdr.ioc_len = sizeof(peer_info); 3011 peer_info.prcfg_size = size; 3012 peer_info.prcfg_bulk = list; 3013 3014 l_errno = 0; 3015 rc = l_ioctl(LNET_DEV_ID, IOC_LIBCFS_GET_PEER_LIST, 3016 &peer_info); 3017 count = peer_info.prcfg_count; 3018 <span class="code-keyword">if</span> (rc == 0) 3019 <span class="code-keyword">break</span>; 3020 l_errno = errno; </pre> </div></div> <p> </p> <p> </p> <p> </p> <p>fyi a bit further down the code appears to retry with count and size returned from ioctl. those values must not be getting across correctly though because the call loops indefinitely. </p> <p>We are hitting this issue on our routers.</p> <p> lnetctl peer show</p> <p>will hang and strace show </p> <p> ioctl(3, _IOC(_IOC_READ|_IOC_WRITE, 0x65, 0x64, 0xb8), 0x7fffffffe6b0) = -1 E2BIG (Argument list too long)</p> <p>The problem is that the <tt>hdr</tt> header in kernel does not go back to userspace on error, so when <tt>lnet_get_peer_list</tt> writes back to <tt>*sizep</tt> it stays in kernel and does not fill back the value for lnetctl to grow the buffer.</p> <p>Dominique Martinet (dominique.martinet@cea.fr) uploaded a new patch: <a href="https://review.whamcloud.com/37559" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37559</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12198" title="lnetctl peer show hangs for ~2600 clients, ioctl getting E2BIG" class="issue-link" data-issue-key="LU-12198"><del>LU-12198</del></a> libcfs: always copy ioctl header back to user<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 818fc691a0e29e5764bfcd65d2a1918c5369fe7c</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/37559/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37559/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12198" title="lnetctl peer show hangs for ~2600 clients, ioctl getting E2BIG" class="issue-link" data-issue-key="LU-12198"><del>LU-12198</del></a> libcfs: always copy ioctl header back to user<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 9e02ef474f8caa833d6a1b5e0068d5323a57e8c4</p> <p>Landed for 2.14</p> <p>Minh Diep (mdiep@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/37720" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37720</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12198" title="lnetctl peer show hangs for ~2600 clients, ioctl getting E2BIG" class="issue-link" data-issue-key="LU-12198"><del>LU-12198</del></a> libcfs: always copy ioctl header back to user<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: 321a757880445089a48d26acfe0554853750ca3f</p> <p>It would be nice if this could land in the next 2.12.x.</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/37720/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37720/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12198" title="lnetctl peer show hangs for ~2600 clients, ioctl getting E2BIG" class="issue-link" data-issue-key="LU-12198"><del>LU-12198</del></a> libcfs: always copy ioctl header back to user<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: a3c687a943233a7c5ae7e3fb906d1913b063c95c</p> <p>I'm seeing something similar to this while calling both</p> <p><tt>lnetctl export</tt></p> <p>and </p> <p><tt>lnetctl peer show</tt></p> <p>These commands hang and running strace shows the above.</p> <p>Is this with 2.14? I suspect we are reaching the limits of using ioctls.</p> Related LU-9680 Development Rank 1|i00f1z: Rank (Obsolete) 9223372036854775807 Severity