https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-518 Lustre <p>replay-single test 45 failed as follows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== replay-single test 45: Handle failed close == 07:35:08 (1311086108) multiop /mnt/lustre/f45 vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.17755 Can't lstat /mnt/lustre/f45: Cannot send after transport endpoint shutdown replay-single test_45: @@@@@@ FAIL: test_45 failed with 2 Dumping lctl log to /home/yujian/test_logs/2011-07-19/054342/replay-single.test_45.*.1311086110.log </pre> </div></div> <p>Dmesg on the client node fat-amd-3-ib:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close == 07:35:08 (1311086108) Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request LustreError: 18821:0:(file.c:155:ll_close_inode_openhandle()) inode 144115440136749057 mdc close failed: rc = -108 LustreError: 18826:0:(client.c:1057:ptlrpc_import_delay_req()) @@@ IMP_INVALID req@ffff8804079b3400 x1374766420142719/t0(0) o-1-&gt;lustre-MDT0000_UUID@192.168.4.2@o2ib:12/10 lens 544/880 e 0 to 0 dl 0 ref 2 fl Rpc:/ffffffff/ffffffff rc 0/-1 LustreError: 18826:0:(mdc_locks.c:722:mdc_enqueue()) ldlm_cli_enqueue: -108 LustreError: 18826:0:(file.c:2165:ll_inode_revalidate_fini()) failure -108 inode 29 Lustre: DEBUG MARKER: replay-single test_45: @@@@@@ FAIL: test_45 failed with 2 </pre> </div></div> <p>Maloo report: <a href="https://maloo.whamcloud.com/test_sets/7f371672-b281-11e0-b33f-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/7f371672-b281-11e0-b33f-52540025f9af</a></p> <p>This is an known issue on Lustre master branch: <a href="https://bugzilla.lustre.org/show_bug.cgi?id=22981" class="external-link" target="_blank" rel="nofollow noopener">bug 22981</a>. Some instances were reported in <a href="https://bugzilla.lustre.org/show_bug.cgi?id=20997" class="external-link" target="_blank" rel="nofollow noopener">bug 20997</a>.</p> <br/> Lustre Tag: v2_0_65_0<br/> Lustre Build: <a href="http://newbuild.whamcloud.com/job/lustre-master/204/">http://newbuild.whamcloud.com/job/lustre-master/204/</a><br/> e2fsprogs Build: <a href="http://newbuild.whamcloud.com/job/e2fsprogs-master/42/">http://newbuild.whamcloud.com/job/e2fsprogs-master/42/</a><br/> Distro/Arch: RHEL6/x86_64(in-kernel OFED, kernel version: 2.6.32-131.2.1.el6)<br/> ENABLE_QUOTA=yes<br/> FAILURE_MODE=HARD<br/> <br/> MGS/MDS Nodes: client-2-ib(active), client-4-ib(passive)<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\ /<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;1 combined MGS/MDT<br/> <br/> OSS Nodes: fat-amd-1-ib(active), fat-amd-2-ib(active)<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\ /<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;OST1 (active in fat-amd-1-ib)<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;OST2 (active in fat-amd-2-ib)<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;OST3 (active in fat-amd-1-ib)<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;OST4 (active in fat-amd-2-ib)<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;OST5 (active in fat-amd-1-ib)<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;OST6 (active in fat-amd-2-ib)<br/> <br/> Client Nodes: fat-amd-3-ib, client-6-ib<br/> LU-518

replay-single test_45: Can't lstat /mnt/lustre/f45: Cannot send after transport endpoint shutdown

Bug Minor Closed Won't Fix WC Triage Jian Yu Wed, 20 Jul 2011 03:33:47 +0000 Tue, 16 Aug 2016 16:32:26 +0000 Tue, 16 Aug 2016 16:32:26 +0000 Lustre 2.0.0 Lustre 2.1.0 0 1 <p>for unknown reason, the recovery reconnect request just failed, and made the MDC import to MDS invalid henceforward.</p> <div class="panel" style="border-width: 1px;"><div class="panelHeader" style="border-bottom-width: 1px;"><b>client2 (fat-amd-3-ib) debug log</b></div><div class="panelContent"> <p>1311086109.335453:0:18824:0:(recover.c:276:ptlrpc_set_import_active()) setting import lustre-MDT0000_UUID VALID<br/> 1311086109.335467:0:18824:0:(import.c:167:ptlrpc_set_import_discon()) lustre-MDT0000-mdc-ffff880117b9d400: Connection to service lustre-MDT0000 via nid 192.168.4.4@o2ib was lost; in progress operations using this service will wait for recovery to complete.<br/> 1311086109.335474:0:18824:0:(import.c:177:ptlrpc_set_import_discon()) ffff880218423000 lustre-MDT0000_UUID: changing import state from FULL to DISCONN<br/> 1311086109.335482:0:18824:0:(import.c:621:ptlrpc_connect_import()) ffff880218423000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING<br/> 1311086109.335489:0:18824:0:(import.c:478:import_select_connection()) lustre-MDT0000-mdc-ffff880117b9d400: connect to NID 192.168.4.4@o2ib last attempt 4305440055<br/> 1311086109.335495:0:18824:0:(import.c:478:import_select_connection()) lustre-MDT0000-mdc-ffff880117b9d400: connect to NID 192.168.4.2@o2ib last attempt 4305416002<br/> 1311086109.335505:0:18824:0:(import.c:550:import_select_connection()) Changing connection for lustre-MDT0000-mdc-ffff880117b9d400 to 192.168.4.2@o2ib/192.168.4.2@o2ib<br/> 1311086109.335509:0:18824:0:(import.c:556:import_select_connection()) lustre-MDT0000-mdc-ffff880117b9d400: import ffff880218423000 using connection 192.168.4.2@o2ib/192.168.4.2@o2ib<br/> 1311086109.335543:0:18824:0:(import.c:720:ptlrpc_connect_import()) @@@ (re)connect request (timeout 5) req@ffff880319ce9400 x1374766420142717/t0(0) o-1-&gt;lustre-MDT0000_UUID@192.168.4.2@o2ib:12/10 lens 368/392 e 0 to 0 dl 0 ref 1 fl New:N/ffffffff/ffffffff rc 0/-1<br/> 1311086109.335575:0:18824:0:(recover.c:344:ptlrpc_recover_import_no_retry()) lustre-MDT0000_UUID: recovery started, waiting<br/> 1311086109.335587:0:15748:0:(client.c:1392:ptlrpc_send_new_req()) Sending RPC pname:cluuid:pid:xid:nid:opc ptlrpcd-rcv:93a2f5a2-2e73-de1c-53c5-2e51c11c95b2:15748:1374766420142717:192.168.4.2@o2ib:38<br/> 1311086109.336967:0:15748:0:(client.c:1775:ptlrpc_expire_one_request()) @@@ Request x1374766420142717 sent from lustre-MDT0000-mdc-ffff880117b9d400 to NID 192.168.4.2@o2ib has failed <font color="red">due to network error:</font> <span class="error">&#91;sent 1311086109&#93;</span> <span class="error">&#91;real_sent 1311086109&#93;</span> <span class="error">&#91;current 1311086109&#93;</span> <span class="error">&#91;deadline 26s&#93;</span> <span class="error">&#91;delay -26s&#93;</span> req@ffff880319ce9400 x1374766420142717/t0(0) o-1-&gt;lustre-MDT0000_UUID@192.168.4.2@o2ib:12/10 lens 368/392 e 0 to 1 dl 1311086135 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1<br/> 1311086109.336994:0:15748:0:(client.c:1807:ptlrpc_expire_one_request()) @@@ err <del>110, sent_state=CONNECTING (now=CONNECTING) req@ffff880319ce9400 x1374766420142717/t0(0) o-1</del>&gt;lustre-MDT0000_UUID@192.168.4.2@o2ib:12/10 lens 368/392 e 0 to 1 dl 1311086135 ref 1 fl Rpc:XN/ffffffff/ffffffff rc 0/-1<br/> 1311086109.337009:0:15748:0:(import.c:1120:ptlrpc_connect_interpret()) ffff880218423000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN<br/> 1311086109.337015:0:15748:0:(import.c:1166:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.4.2@o2ib failed (-110)</p> </div></div> <p>Old ticket for unsupported version</p> Bugzilla ID 22981.0 Development Rank 1|hzvehj: Rank (Obsolete) 5988 Severity