Whamcloud Community JIRA

Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us

9.4.14

940014

05-12-2023

[LU-782] sanity.sh subtest test_220 failed with 3, error -107 -ENOTCONN https://jira.whamcloud.com/browse/LU-782 Lustre <p>This issue was created by maloo for Chris Gearing <chris@whamcloud.com></p> <p>This issue relates to the following test suite run: <a href="https://maloo.whamcloud.com/test_sets/af3669de-f9cd-11e0-b486-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/af3669de-f9cd-11e0-b486-52540025f9af</a>.</p> <p>The sub-test test_220 failed with the following error:</p> <blockquote> <p>test_220 failed with 3</p></blockquote> <p>Info required for matching: sanity 220</p> LU-782

sanity.sh subtest test_220 failed with 3, error -107 -ENOTCONN

Bug Minor Resolved Cannot Reproduce Zhenyu Xu Maloo Fri, 21 Oct 2011 05:17:55 +0000 Mon, 27 Feb 2017 23:35:15 +0000 Mon, 27 Feb 2017 23:35:15 +0000 0 6 <p>Bobijam</p> <p>Could you please look into this regular failure. Note the comments from Tappro on LU748 and the work in this area done by Andreas on LU614</p> <p>Thanks</p> <p>Peter</p> <p>another hit <a href="https://maloo.whamcloud.com/sub_tests/ec9a1b44-f451-11e0-908b-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/sub_tests/ec9a1b44-f451-11e0-908b-52540025f9af</a></p> <p>With unknown reason, while creating object, MDS reports ENOTCONN</p> <p>(lov_request.c:569:lov_update_create_set()) error creating fid 0x78d0 sub-object on OST idx 0/1: rc = -107</p> <p>But I havn't found the evidence of network error between MDS0 and OST0.</p> <p>Couldn't find the cause of this -107 error, need -1 bug log to reveal more details.</p> <p>Sarah</p> <p>Could you please try and repeat this test manually and gather the -1 debug log?</p> <p>Thanks</p> <p>Peter</p> <p>sure, will keep you updated</p> <p>Another failure instance:</p> <p><a href="https://maloo.whamcloud.com/test_sets/7d12e158-2c63-11e1-ab2e-5254004bbbd3" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/7d12e158-2c63-11e1-ab2e-5254004bbbd3</a></p> <p>We seem to hit the same issue.<br/> Some file creation onto specific OSTs fail after 50 seconds. The next available OST is then assigned to this file, with the MDS reporting the "error creating fix 0xXX sub-objects on OST idx XX/1: rc= -107" message.<br/> The failing OSTs are not always the same, but persist enough time to reproduce the issue for several minutes consistently.<br/> I will attach soon full debug logs from the client, the MDS and the OSS serving the failing OST during a 'lfs setstripe -c 1 -o 3 toto' which shows the problem.<br/> This looks like a communication problem between the MDS and OSS, but communication with other OSTs on the same OSS does not cause any problem.</p> <p>In the MDS debug log, Request x1405820398383011 is never seen as "Sent" and also never appears in the OSS debug logs.</p> <p>Full debug log of lfs setstripe -c 1 -o 3 toto on client, MDS and OSS.</p> <p>A crashdump of the MDS in this state is now available if needed.</p> <p>Is it easy to reproduce it? If so, would you mind collecting -1 debug log for it? </p> <p>This is easy to reproduce in the current state of the cluster.<br/> The tarball I already uploaded (issue-20120702.tar.gz) already contains a -1 debug log for it on client+mds+oss.<br/> The debug log on the mds mentions a RPC which times out after 50s, but this RPC number was never logged as "Sent", and the OSS log never mentions it as "Received".</p> <p>I think the log has not covered the life cycle of x1405820398383011 request, the first appearance of it in MDS is its timeout event, the log shows it timed out and it seems that it's a no-resend request, mostly goes through following path.</p> <div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader" style="border-bottom-width: 1px;"><b>ptlrpc_check_set()</b></div><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-comment">// </span> <span class="code-keyword">if</span> (ptlrpc_import_delay_req(imp, req, &status)){ /* put on delay list - only <span class="code-keyword">if</span> we wait * recovery finished - before send */ cfs_list_del_init(&req->rq_list); cfs_list_add_tail(&req->rq_list, &imp-> \ imp_delayed_list); cfs_spin_unlock(&imp->imp_lock); <span class="code-keyword">continue</span>; } <span class="code-keyword">if</span> (status != 0) { req->rq_status = status; ptlrpc_rqphase_move(req, RQ_PHASE_INTERPRET); cfs_spin_unlock(&imp->imp_lock); GOTO(interpret, req->rq_status); } <span class="code-keyword">if</span> (ptlrpc_no_resend(req) && !req->rq_wait_ctx) { <span class="code-comment">/////// HERE </span> req->rq_status = -ENOTCONN; ptlrpc_rqphase_move(req, RQ_PHASE_INTERPRET); cfs_spin_unlock(&imp->imp_lock); GOTO(interpret, req->rq_status); } </pre> </div></div> <p>and the request never got send, is it possible to get a log covering the life cycle of the -107 request?</p> <p>ps. what Lustre version do you use?</p> <p>Here is a new debug log session.<br/> MDS is helios17, OST used is scratch-OST000d which is mounted on helios26 and client is helios86.<br/> Their nids are respectively 10.4.72.51@o2ib, 10.4.72.8@o2ib and 10.4.67.2@o2ib.</p> <p>The request number leading to the -107 is 1406627952493697 and once again appears only once it times out on the MDS. I let 5 seconds between the activation of debug and trigger of the issue, so the whole lifecycle of the RPC should appear.</p> <p>We are using our internal Bull version Bull.2.213 which is based upon 2.1.1 + a few patches, including ORNL-22 and <a href="https://jira.whamcloud.com/browse/LU-1144" title="implement a NUMA aware ptlrpcd binding policy" class="issue-link" data-issue-key="LU-1144"><del>LU-1144</del></a>.<br/> The introduced ptlrpc_bind_policy parameter is set to 3.</p> <p>We should soon test the behavior with ptlrpc_bind_policy=1 to "revert" to the old code path.</p> <p>strangely the request birth still escaped our monitoring.</p> <div class="panel" style="border-width: 1px;"><div class="panelHeader" style="border-bottom-width: 1px;"><b>$ grep ffff880bcb72f800 lustre-debug-20120709.helios17</b></div><div class="panelContent"> <p>154252:00000100:00000040:11.0:1341848952.925668:0:5006:0:(client.c:1794:ptlrpc_expire_one_request()) @@@ Request x1406627952493697 sent from scratch-OST000d-osc-MDT0000 to NID 10.4.72.8@o2ib has timed out for sent delay: <span class="error">[sent 1341848902]</span> <span class="error">[real_sent 0]</span> <span class="error">[current 1341848952]</span> <span class="error">[deadline 50s]</span> <span class="error">[delay 0s]</span> req@ffff880bcb72f800 x1406627952493697/t0(0) o-1->scratch-OST000d_UUID@10.4.72.8@o2ib:0/0 lens 0/0 e 0 to 1 dl 1341848952 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1</p> <p>154265:00000100:00000040:11.0:1341848952.925691:0:5006:0:(lustre_net.h:1675:ptlrpc_rqphase_move()) @@@ move req "Rpc" <del>> "Interpret" req@ffff880bcb72f800 x1406627952493697/t0(0) o-1</del>>scratch-OST000d_UUID@10.4.72.8@o2ib:0/0 lens 0/0 e 0 to 1 dl 1341848952 ref 1 fl Rpc:XN/ffffffff/ffffffff rc -107/-1</p> <p>154276:00000100:00000001:11.0:1341848952.937088:0:5006:0:(client.c:2440:ptlrpc_request_addref()) Process leaving (rc=18446612182972168192 : -131890737383424 : ffff880bcb72f800)</p> <p>154277:00000100:00000040:11.0:1341848952.937089:0:5006:0:(lustre_net.h:1675:ptlrpc_rqphase_move()) @@@ move req "Interpret" <del>> "Complete" req@ffff880bcb72f800 x1406627952493697/t0(0) o-1</del>>scratch-OST000d_UUID@10.4.72.8@o2ib:0/0 lens 0/0 e 0 to 1 dl 1341848952 ref 2 fl Interpret:XN/ffffffff/ffffffff rc -107/-1</p> <p>154279:00000100:00000040:11.0:1341848952.937093:0:5006:0:(client.c:2201:__ptlrpc_req_finished()) @@@ refcount now 1 req@ffff880bcb72f800 x1406627952493697/t0(0) o-1->scratch-OST000d_UUID@10.4.72.8@o2ib:0/0 lens 0/0 e 0 to 1 dl 1341848952 ref 2 fl Complete:XN/ffffffff/ffffffff rc -107/-1</p> <p>154285:00000100:00000040:11.0:1341848952.937101:0:5006:0:(client.c:2201:__ptlrpc_req_finished()) @@@ refcount now 0 req@ffff880bcb72f800 x1406627952493697/t0(0) o-1->scratch-OST000d_UUID@10.4.72.8@o2ib:0/0 lens 0/0 e 0 to 1 dl 1341848952 ref 1 fl Complete:XN/ffffffff/ffffffff rc -107/-1</p> <p>154290:00000100:00000010:11.0:1341848952.937107:0:5006:0:(client.c:2166:__ptlrpc_free_req()) kfreed 'request': 968 at ffff880bcb72f800.</p> <p>...</p> </div></div> <p>Close old bug.</p> Development Rank 1|hzv51z: Rank (Obsolete) 4352 Severity