Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 [LU-3751] disable OUT_PORTAL on OST for now https://jira.whamcloud.com/browse/LU-3751 Lustre <p>Disable OUT_PORTAL on OST, otherwise it will confuse the receiver when MDT and OST are on the same node, which might make ll_ost_outxx handle CONNECT requests from MDTs. Then it would cause panic like</p> <p>Lustre: 13261:0:(client.c:1896:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: <span class="error">&#91;sent 1401492648/real 1401492648&#93;</span> req@ffff880044bf9000 x1469570233011472/t0(0) o1000-&gt;lustre-MDT0000-osp-MDT0001@0@lo:24/10 lens 8416/8416 e 0 to 1 dl 1401492655 ref 2 fl Rpc:X/0/ffffffff rc 0/-1<br/> Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete<br/> Lustre: Skipped 5 previous similar messages<br/> Lustre: lustre-MDT0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting<br/> LustreError: 4239:0:(mdt_handler.c:3191:mdt_tgt_connect()) ASSERTION( mti != ((void *)0) ) failed: <br/> LustreError: 4239:0:(mdt_handler.c:3191:mdt_tgt_connect()) LBUG<br/> Pid: 4239, comm: ll_ost_out01_00</p> <p>Call Trace:<br/> <span class="error">&#91;&lt;ffffffffa0603905&gt;&#93;</span> libcfs_debug_dumpstack+0x55/0x80 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0603f07&gt;&#93;</span> lbug_with_loc+0x47/0xb0 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0ddab95&gt;&#93;</span> mdt_tgt_connect+0x515/0x550 <span class="error">&#91;mdt&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0939f5d&gt;&#93;</span> tgt_request_handle+0x57d/0xe30 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa08f6718&gt;&#93;</span> ptlrpc_server_handle_request+0x398/0xc60 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa06045be&gt;&#93;</span> ? cfs_timer_arm+0xe/0x10 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa061629f&gt;&#93;</span> ? lc_watchdog_touch+0x6f/0x170 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa08edd09&gt;&#93;</span> ? ptlrpc_wait_event+0xa9/0x290 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa06147e1&gt;&#93;</span> ? libcfs_debug_msg+0x41/0x50 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffff810533f3&gt;&#93;</span> ? __wake_up+0x53/0x70<br/> <span class="error">&#91;&lt;ffffffffa08f7aac&gt;&#93;</span> ptlrpc_main+0xacc/0x1750 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa08f6fe0&gt;&#93;</span> ? ptlrpc_main+0x0/0x1750 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff81091d66&gt;&#93;</span> kthread+0x96/0xa0<br/> <span class="error">&#91;&lt;ffffffff8100c14a&gt;&#93;</span> child_rip+0xa/0x20<br/> <span class="error">&#91;&lt;ffffffff81091cd0&gt;&#93;</span> ? kthread+0x0/0xa0<br/> <span class="error">&#91;&lt;ffffffff8100c140&gt;&#93;</span> ? child_rip+0x0/0x20</p> LU-3751 disable OUT_PORTAL on OST for now Bug Blocker Resolved Fixed Di Wang Di Wang Tue, 13 Aug 2013 20:39:23 +0000 Wed, 8 Jan 2014 18:57:57 +0000 Thu, 29 Aug 2013 18:10:57 +0000 Lustre 2.5.0 Lustre 2.5.0 0 6 <p><a href="http://review.whamcloud.com/7323" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7323</a></p> <p>I used the following:</p> <p>diff --git a/lustre/include/lustre/lustre_idl.h b/lustre/include/lustre/lustre_idl.h<br/> index 3ee0c7e..109ae00 100644<br/> &#8212; a/lustre/include/lustre/lustre_idl.h<br/> +++ b/lustre/include/lustre/lustre_idl.h<br/> @@ -141,6 +141,7 @@<br/> #define SEQ_DATA_PORTAL 31<br/> #define SEQ_CONTROLLER_PORTAL 32<br/> #define MGS_BULK_PORTAL 33<br/> +#define OUT_OST_PORTAL 34</p> <p> /* Portal 63 is reserved for the Cray Inc DVS - nic@cray.com, roe@cray.com, n8851@cray.com */</p> <p>diff --git a/lustre/ost/ost_handler.c b/lustre/ost/ost_handler.c<br/> index 7880341..2f4421f 100644<br/> &#8212; a/lustre/ost/ost_handler.c<br/> +++ b/lustre/ost/ost_handler.c<br/> @@ -2797,7 +2797,7 @@ static int ost_setup(struct obd_device <b>obd, struct lustre_cfg</b> lcfg)<br/> .bc_buf_size = OUT_BUFSIZE,<br/> .bc_req_max_size = OUT_MAXREQSIZE,<br/> .bc_rep_max_size = OUT_MAXREPSIZE,</p> <ul class="alternate" type="square"> <li>.bc_req_portal = OUT_PORTAL,<br/> + .bc_req_portal = OUT_OST_PORTAL,<br/> .bc_rep_portal = OSC_REPLY_PORTAL,<br/> },<br/> /*</li> </ul> <p>Alex, could you please push your patch to review and try to land it?</p> <p>What are the implications here for unified targets? Isn't the whole point of UT that the same RPC to the same portal will execute the same operation on an OSD? If the "OST OUT" has a different portal than the "MDT OUT", we will need to handle OST updates separately from MDT updates forever in the future, which doesn't make sense to me.</p> <p>My preference would be Di's patch that just disables this code for 2.5 (rather than changing the protocol forever in the future as Alex's patch does). However, disabling the OST OUT handler will cause problems for LFSCK Phase 2, which is supposed to be using this service for MDT-&gt;OST communications, though that will only become a problem in 2.6.</p> <p>Mike, there needs to be some way for OUT to handle RPCs for both MDT and OST devices. Is that part of your later UT patch series?</p> <p>I'm not saying that change is the right thing in the long term, but at least it gives me a way to develop stuff before UT is complete.</p> <p>Problem should gone when MDT part of UT patch will be landed, no need to change protocol. Alex, could you base your changes on later patches in UT series? I expect it should work with <a href="http://review.whamcloud.com/6973" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/6973</a></p> <p>sure, I will do.</p> <p>With Change, 7323 now landed to Master can this ticket be closed or is additional work needed?</p> <p>The patch has landed to master. Let me know if more work is needed and I will reopen this ticket.</p> Duplicate LU-3728 Development Rank 1|hzvxvz: Rank (Obsolete) 9674 Severity