https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4913 Lustre <p>&lt;0&gt;[ 231.434653] LustreError: 26880:0:(import.c:286:ptlrpc_invalidate_import()) ASSERTION( imp-&gt;imp_invalid ) failed: <br/> &lt;0&gt;[ 231.445129] LustreError: 26880:0:(import.c:286:ptlrpc_invalidate_import()) LBUG<br/> &lt;4&gt;[ 231.452651] Pid: 26880, comm: mount.lustre<br/> &lt;4&gt;[ 231.456892] <br/> &lt;4&gt;[ 231.456892] Call Trace:<br/> &lt;4&gt;[ 231.461143] <span class="error">&#91;&lt;ffffffffa057b825&gt;&#93;</span> libcfs_debug_dumpstack+0x55/0x80 <span class="error">&#91;libcfs&#93;</span><br/> &lt;4&gt;[ 231.468255] <span class="error">&#91;&lt;ffffffffa057be27&gt;&#93;</span> lbug_with_loc+0x47/0xb0 <span class="error">&#91;libcfs&#93;</span><br/> &lt;4&gt;[ 231.474624] <span class="error">&#91;&lt;ffffffffa08a3597&gt;&#93;</span> ptlrpc_invalidate_import+0x377/0x8b0 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt;[ 231.482172] <span class="error">&#91;&lt;ffffffffa08a1a16&gt;&#93;</span> ? ptlrpc_set_import_discon+0xf6/0x600 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt;[ 231.489805] <span class="error">&#91;&lt;ffffffffa08a3b03&gt;&#93;</span> ptlrpc_reconnect_import+0x33/0x1a0 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt;[ 231.497100] <span class="error">&#91;&lt;ffffffffa05870d1&gt;&#93;</span> ? libcfs_debug_msg+0x41/0x50 <span class="error">&#91;libcfs&#93;</span><br/> &lt;4&gt;[ 231.503869] <span class="error">&#91;&lt;ffffffffa0380aec&gt;&#93;</span> mgc_set_info_async+0x32c/0xc80 <span class="error">&#91;mgc&#93;</span><br/> &lt;4&gt;[ 231.510570] <span class="error">&#91;&lt;ffffffffa0722d75&gt;&#93;</span> obd_set_info_async.clone.3+0xf5/0x3a0 <span class="error">&#91;obdclass&#93;</span><br/> &lt;4&gt;[ 231.518377] <span class="error">&#91;&lt;ffffffffa06fe1d6&gt;&#93;</span> ? class_name2dev+0x56/0xe0 <span class="error">&#91;obdclass&#93;</span><br/> &lt;4&gt;[ 231.525164] <span class="error">&#91;&lt;ffffffffa072b471&gt;&#93;</span> lustre_start_mgc+0x12a1/0x2020 <span class="error">&#91;obdclass&#93;</span><br/> &lt;4&gt;[ 231.532310] <span class="error">&#91;&lt;ffffffffa0724927&gt;&#93;</span> ? server_kernel_mount+0x667/0xf30 <span class="error">&#91;obdclass&#93;</span><br/> &lt;4&gt;[ 231.539761] <span class="error">&#91;&lt;ffffffffa072ece8&gt;&#93;</span> lustre_fill_super+0x2d8/0xfc0 <span class="error">&#91;obdclass&#93;</span><br/> &lt;4&gt;[ 231.546818] <span class="error">&#91;&lt;ffffffffa072ea10&gt;&#93;</span> ? lustre_fill_super+0x0/0xfc0 <span class="error">&#91;obdclass&#93;</span><br/> &lt;4&gt;[ 231.553851] <span class="error">&#91;&lt;ffffffff8117997f&gt;&#93;</span> get_sb_nodev+0x5f/0xa0<br/> &lt;4&gt;[ 231.559341] <span class="error">&#91;&lt;ffffffffa071f765&gt;&#93;</span> lustre_get_sb+0x25/0x30 <span class="error">&#91;obdclass&#93;</span><br/> &lt;4&gt;[ 231.565850] <span class="error">&#91;&lt;ffffffff811795db&gt;&#93;</span> vfs_kern_mount+0x7b/0x1b0<br/> &lt;4&gt;[ 231.571567] <span class="error">&#91;&lt;ffffffff81179782&gt;&#93;</span> do_kern_mount+0x52/0x130<br/> &lt;4&gt;[ 231.577210] <span class="error">&#91;&lt;ffffffff81197da2&gt;&#93;</span> do_mount+0x2d2/0x8d0<br/> &lt;4&gt;[ 231.582493] <span class="error">&#91;&lt;ffffffff81198430&gt;&#93;</span> sys_mount+0x90/0xe0<br/> &lt;4&gt;[ 231.587687] <span class="error">&#91;&lt;ffffffff8100b0f2&gt;&#93;</span> system_call_fastpath+0x16/0x1b</p> LU-4913

mgc import reconnect race

Bug Critical Resolved Fixed Cliff White Andriy Skulysh patch Wed, 16 Apr 2014 09:30:29 +0000 Tue, 31 May 2016 21:00:05 +0000 Fri, 23 May 2014 14:24:37 +0000 Lustre 2.6.0 0 7 <p>mgc import can be reconnected by pinger or ptlrpc_reconnect_import().<br/> ptlrpc_invalidate_import() isn't protected against alteration of imp_invalid state. Import can be<br/> reconnected by pinger which makes imp_invalid equal to false. Thus LASSERT(imp-&gt;imp_invalid) fails<br/> in ptlrpc_invalidate_import().</p> <p>patch: <a href="http://review.whamcloud.com/9967" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9967</a></p> <p>A test was added to reproduce the issue. Here is dmesg log showing the failure:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> &lt;4&gt;Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race == 16:25:18 (1397136318) &lt;6&gt;LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: &lt;6&gt;LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: &lt;4&gt;Lustre: 9028:0:(ldlm_lib.c:1005:target_handle_connect()) MGS: connection from afc19ceb-785b-4037-43da-94710aee9fb7@0@lo t0 exp (null) cur 1397136328 last 0 &lt;4&gt;Lustre: lustre-MDT0000: used disk, loading &lt;4&gt;Lustre: 9078:0:(debug.c:320:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release. &lt;6&gt;LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: &lt;6&gt;LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: &lt;6&gt;Lustre: lustre-OST0000: Now serving lustre-OST0000 on /dev/loop1 with recovery enabled &lt;4&gt;Lustre: 9202:0:(genops.c:1498:obd_export_evict_by_uuid()) MGS: evicting afc19ceb-785b-4037-43da-94710aee9fb7 at adminstrative request &lt;3&gt;LustreError: 11-0: MGC192.168.122.169@tcp: Communicating with 0@lo, operation ldlm_enqueue failed with -107. &lt;3&gt;LustreError: 166-1: MGC192.168.122.169@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail &lt;3&gt;LustreError: 9027:0:(fail.c:124:__cfs_fail_timeout_set()) cfs_fail_timeout id 906 sleeping for 10000ms &lt;4&gt;Lustre: 8745:0:(client.c:1960:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1397136328/real 1397136328] req@ffff88007a9c3bc0 x1465003620171816/t0(0) o8-&gt;lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 368/512 e 0 to 1 dl 1397136333 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 &lt;4&gt;Lustre: 9136:0:(ldlm_lib.c:1005:target_handle_connect()) lustre-OST0000: connection from lustre-MDT0000-mdtlov_UUID@0@lo t0 exp (null) cur 1397136338 last 0 &lt;4&gt;Lustre: lustre-OST0000: received MDS connection from 0@lo &lt;6&gt;Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0000_UUID now active, resetting orphans &lt;4&gt;Lustre: 8745:0:(client.c:1960:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1397136333/real 1397136333] req@ffff88007a9c3440 x1465003620171868/t0(0) o250-&gt;MGC192.168.122.169@tcp@0@lo:26/25 lens 368/512 e 0 to 1 dl 1397136339 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 &lt;3&gt;LustreError: 9027:0:(fail.c:128:__cfs_fail_timeout_set()) cfs_fail_timeout id 906 awake &lt;4&gt;Lustre: 9027:0:(ldlm_lib.c:1005:target_handle_connect()) MGS: connection from afc19ceb-785b-4037-43da-94710aee9fb7@0@lo t0 exp (null) cur 1397136343 last 0 &lt;3&gt;LustreError: 9029:0:(fail.c:124:__cfs_fail_timeout_set()) cfs_fail_timeout id 906 sleeping for 10000ms &lt;3&gt;LustreError: 9029:0:(fail.c:128:__cfs_fail_timeout_set()) cfs_fail_timeout id 906 awake &lt;4&gt;Lustre: MGS: Client afc19ceb-785b-4037-43da-94710aee9fb7 (at 0@lo) reconnecting &lt;4&gt;Lustre: 9029:0:(ldlm_lib.c:1005:target_handle_connect()) MGS: connection from afc19ceb-785b-4037-43da-94710aee9fb7@0@lo t0 exp ffff88007a58ec00 cur 1397136353 last 1397136343 &lt;3&gt;LustreError: 9029:0:(obd_class.h:514:obd_set_info_async()) obd_set_info_async: dev 0 no operation &lt;4&gt;Lustre: Evicted from MGS (at MGC192.168.122.169@tcp_0) after server handle changed from 0x49a8420081f448d2 to 0x49a8420081f44981 &lt;3&gt;LustreError: 9205:0:(fail.c:124:__cfs_fail_timeout_set()) cfs_fail_timeout id 906 sleeping for 15000ms &lt;6&gt;LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: &lt;6&gt;LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: &lt;3&gt;LustreError: 9231:0:(fail.c:124:__cfs_fail_timeout_set()) cfs_fail_timeout id 906 sleeping for 15000ms &lt;3&gt;LustreError: 9205:0:(fail.c:128:__cfs_fail_timeout_set()) cfs_fail_timeout id 906 awake &lt;6&gt;Lustre: MGC192.168.122.169@tcp: Connection restored to MGS (at 0@lo) &lt;3&gt;LustreError: 9231:0:(fail.c:128:__cfs_fail_timeout_set()) cfs_fail_timeout id 906 awake &lt;0&gt;LustreError: 9231:0:(import.c:287:ptlrpc_invalidate_import()) ASSERTION( imp-&gt;imp_invalid ) failed: &lt;0&gt;LustreError: 9231:0:(import.c:287:ptlrpc_invalidate_import()) LBUG &lt;4&gt;Pid: 9231, comm: mount.lustre &lt;4&gt; &lt;4&gt;Call Trace: &lt;4&gt; [&lt;ffffffffa0d9d825&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] &lt;4&gt; [&lt;ffffffffa0d9de27&gt;] lbug_with_loc+0x47/0xb0 [libcfs] &lt;4&gt; [&lt;ffffffffa068df2f&gt;] ptlrpc_invalidate_import+0x84f/0x8e0 [ptlrpc] &lt;4&gt; [&lt;ffffffffa068bed6&gt;] ? ptlrpc_set_import_discon+0xf6/0x600 [ptlrpc] &lt;4&gt; [&lt;ffffffffa068dff3&gt;] ptlrpc_reconnect_import+0x33/0x1a0 [ptlrpc] &lt;4&gt; [&lt;ffffffffa0da90f1&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] &lt;4&gt; [&lt;ffffffffa092bb3c&gt;] mgc_set_info_async+0x32c/0xc80 [mgc] &lt;4&gt; [&lt;ffffffffa0da90f1&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] &lt;4&gt; [&lt;ffffffffa050c115&gt;] obd_set_info_async.clone.3+0xf5/0x3a0 [obdclass] &lt;4&gt; [&lt;ffffffffa04e71f6&gt;] ? class_name2dev+0x56/0xe0 [obdclass] &lt;4&gt; [&lt;ffffffffa05148f3&gt;] lustre_start_mgc+0x1383/0x1fd0 [obdclass] &lt;4&gt; [&lt;ffffffffa050dd10&gt;] ? server_kernel_mount+0x6b0/0xf30 [obdclass] &lt;4&gt; [&lt;ffffffffa0518038&gt;] lustre_fill_super+0x2d8/0xfd0 [obdclass] &lt;4&gt; [&lt;ffffffffa0517d60&gt;] ? lustre_fill_super+0x0/0xfd0 [obdclass] &lt;4&gt; [&lt;ffffffff81179a2f&gt;] get_sb_nodev+0x5f/0xa0 &lt;4&gt; [&lt;ffffffffa0508b05&gt;] lustre_get_sb+0x25/0x30 [obdclass] &lt;4&gt; [&lt;ffffffff8117968b&gt;] vfs_kern_mount+0x7b/0x1b0 &lt;4&gt; [&lt;ffffffff81179832&gt;] do_kern_mount+0x52/0x130 &lt;4&gt; [&lt;ffffffff81197eb2&gt;] do_mount+0x2d2/0x8d0 &lt;4&gt; [&lt;ffffffff81198540&gt;] sys_mount+0x90/0xe0 &lt;4&gt; [&lt;ffffffff8100b0f2&gt;] system_call_fastpath+0x16/0x1b </pre> </div></div> <p>Landed for 2.6</p> <p>FYI, in <a href="https://jira.whamcloud.com/browse/LU-5420" title="Failure on test suite sanity test_17m: mount MDS failed, Input/output error" class="issue-link" data-issue-key="LU-5420"><del>LU-5420</del></a> it was reported that the fix for this bug caused that regression.</p> Related LU-5420 LU-8206 Development Rank 1|hzwk8v: Rank (Obsolete) 13574 Severity