https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-8172 Lustre <p>Trying to run simple lnet_selftest failed between 2 nodes with this error</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">00000400:00020000:13.0F:1463699033.377705:0:8764:0:(lib-ptl.c:190:lnet_try_match_md()) Matching packet from 12345-10.151.27.60@o2ib, match 1 length 172 too big: 160 left, 160 allowed 00000400:00000100:13.0:1463699033.420950:0:8764:0:(lib-move.c:1479:lnet_parse_put()) Dropping PUT from 12345-10.151.27.60@o2ib portal 51 match 1 offset 0 length 172: 4 </pre> </div></div> <p>It works within the same node.</p> 2.7.1 with <a href="https://jira.whamcloud.com/browse/LU-3322" title="ko2iblnd support for different map_on_demand and peer_credits between systems" class="issue-link" data-issue-key="LU-3322"><strike>LU-3322</strike></a> LU-8172

match 1 length 172 too big: 160 left, 160 allowed

Bug Major Closed Not a Bug Doug Oucharek Mahmoud Hanafi Thu, 19 May 2016 23:16:17 +0000 Tue, 24 May 2016 21:55:37 +0000 Tue, 24 May 2016 21:55:37 +0000 Lustre 2.7.0 0 4 <p>I rebased our nas-2.7.1-fe git repo with b2_7_fe on 5/12.<br/> <a href="https://jira.whamcloud.com/browse/LU-3322" title="ko2iblnd support for different map_on_demand and peer_credits between systems" class="issue-link" data-issue-key="LU-3322"><del>LU-3322</del></a> was pulled in with that rebase. We did not have the problem before the rebase.</p> <p>There are some LU patches we carry not in b2_7_fe:</p> <p><a href="https://jira.whamcloud.com/browse/LU-7808" title="Service threads hung at ptlrpc_abort_bulk" class="issue-link" data-issue-key="LU-7808"><del>LU-7808</del></a> lnet: Debug patch to investigate MD Unlink<br/> <a href="https://jira.whamcloud.com/browse/LU-7733" title="ptlrpc_lprocfs_svc_req_history_show (Seq_printf) time resolution should be higher than 1 second." class="issue-link" data-issue-key="LU-7733"><del>LU-7733</del></a> ptlrpc: print times in microseconds<br/> <a href="https://jira.whamcloud.com/browse/LU-7795" title="least_qunit should be tunnable" class="issue-link" data-issue-key="LU-7795"><del>LU-7795</del></a> quota: tuneable soft least qunit<br/> <a href="https://jira.whamcloud.com/browse/LU-6433" title="MDS deadlock in qouta " class="issue-link" data-issue-key="LU-6433"><del>LU-6433</del></a> quota: handle QUOTA_DQACQ in READPAGE portal<br/> <a href="https://jira.whamcloud.com/browse/LU-7569" title="IB leaf switch caused LNet routers to crash" class="issue-link" data-issue-key="LU-7569"><del>LU-7569</del></a> o2iblnd: avoid intensive reconnecting<br/> <a href="https://jira.whamcloud.com/browse/LU-7210" title="ASSERTION( peer-&gt;ibp_connecting == 0 )" class="issue-link" data-issue-key="LU-7210"><del>LU-7210</del></a> o2iblnd: take extra refcount in kiblnd_connreq_done<br/> <a href="https://jira.whamcloud.com/browse/LU-7054" title="ib_cm scalling issue when lustre clients connect to OSS" class="issue-link" data-issue-key="LU-7054"><del>LU-7054</del></a> o2iblnd: less intense allocating retry<br/> Revert "<a href="https://jira.whamcloud.com/browse/LU-7054" title="ib_cm scalling issue when lustre clients connect to OSS" class="issue-link" data-issue-key="LU-7054"><del>LU-7054</del></a> o2iblnd: less intense allocating retry"<br/> <a href="https://jira.whamcloud.com/browse/LU-7372" title="replay-dual test_26: test failed to respond and timed out" class="issue-link" data-issue-key="LU-7372"><del>LU-7372</del></a> mgs: reprocess all locks at device fini<br/> <a href="https://jira.whamcloud.com/browse/LU-5704" title="lfs getstripe hangs on fifo files" class="issue-link" data-issue-key="LU-5704"><del>LU-5704</del></a> utils: stop open hangs on fifo files<br/> <a href="https://jira.whamcloud.com/browse/LU-7054" title="ib_cm scalling issue when lustre clients connect to OSS" class="issue-link" data-issue-key="LU-7054"><del>LU-7054</del></a> o2iblnd: less intense allocating retry<br/> <a href="https://jira.whamcloud.com/browse/LU-7506" title="lfs quota -h should support petabytes output" class="issue-link" data-issue-key="LU-7506"><del>LU-7506</del></a> lfs: "lfs quota -h" should support petabytes output</p> <p>The git repo can be accessed at github. Peter Jones's account has access to it.<br/> <a href="https://github.com/NASAEarthExchange/lustre-nas-fe/commits/nas-2.7.1" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/NASAEarthExchange/lustre-nas-fe/commits/nas-2.7.1</a></p> <p>Doug</p> <p>Could you please advise?</p> <p>Thanks</p> <p>Peter</p> <p>Is the exact same build running on all the nodes involved in the self test? My interpretation of the error is that the message size we are anticipating is 12 bytes less than the amount which was sent. </p> <p>Are there any issue with running 2.7.1+<a href="https://jira.whamcloud.com/browse/LU-3322" title="ko2iblnd support for different map_on_demand and peer_credits between systems" class="issue-link" data-issue-key="LU-3322"><del>LU-3322</del></a> server and 2.5.3or2.7.1 none <a href="https://jira.whamcloud.com/browse/LU-3322" title="ko2iblnd support for different map_on_demand and peer_credits between systems" class="issue-link" data-issue-key="LU-3322"><del>LU-3322</del></a> clients?</p> <p>There is incompatibility of lst between lu-3322 builds and non lu-3322. This was the issue I was having</p> <p>Should lst work with a build without <a href="https://jira.whamcloud.com/browse/LU-3322" title="ko2iblnd support for different map_on_demand and peer_credits between systems" class="issue-link" data-issue-key="LU-3322"><del>LU-3322</del></a> and one with <a href="https://jira.whamcloud.com/browse/LU-3322" title="ko2iblnd support for different map_on_demand and peer_credits between systems" class="issue-link" data-issue-key="LU-3322"><del>LU-3322</del></a>? </p> <p>Mahmoud, our nas-2.7.1-4.1nasS server and nas-2.7.1-4nasC client both have <a href="https://jira.whamcloud.com/browse/LU-3322" title="ko2iblnd support for different map_on_demand and peer_credits between systems" class="issue-link" data-issue-key="LU-3322"><del>LU-3322</del></a>.<br/> The nas-2.5.3 client does not have <a href="https://jira.whamcloud.com/browse/LU-3322" title="ko2iblnd support for different map_on_demand and peer_credits between systems" class="issue-link" data-issue-key="LU-3322"><del>LU-3322</del></a>.</p> <p>after trying out different version it looks like "<a href="https://jira.whamcloud.com/browse/LU-7808" title="Service threads hung at ptlrpc_abort_bulk" class="issue-link" data-issue-key="LU-7808"><del>LU-7808</del></a> lnet: Debug patch to investigate MD Unlink" is where it breaks.</p> <p>Mahmoud</p> <p>I recall that this patch was quite different between the two versions. Do you have the appropriate version on both releases? Does everything work fine if you remove it altogether? </p> <p>Peter</p> <p>Peter,</p> <p>We have correct versions of the <a href="https://jira.whamcloud.com/browse/LU-7808" title="Service threads hung at ptlrpc_abort_bulk" class="issue-link" data-issue-key="LU-7808"><del>LU-7808</del></a> lnet debug patches.<br/> The b2_5_fe(#19643, patch set 4) and b2_7_fe(#19916 patch set 1) are different.</p> <p>The 3 new stats I added to the debug patch for <a href="https://jira.whamcloud.com/browse/LU-7808" title="Service threads hung at ptlrpc_abort_bulk" class="issue-link" data-issue-key="LU-7808"><del>LU-7808</del></a> add up to 12 bytes and are traded between the two nodes during an lnet-selftest. So it looks like one side is sending the new 3 stats but the other side is not expecting them. I checked the two patches and they both should be sending/receiving the new 3 stats.</p> <p>As well as LNet, the debug patches for <a href="https://jira.whamcloud.com/browse/LU-7808" title="Service threads hung at ptlrpc_abort_bulk" class="issue-link" data-issue-key="LU-7808"><del>LU-7808</del></a> changed the lnet-selftest module. Has the updated lnet-selftest module been updated on both nodes?</p> <p>It does look like this was the main issue and lst didn't match across all sides.</p> <p>You can close this issue.</p> Development Rank 1|hzyc9z: Rank (Obsolete) 9223372036854775807 Severity