https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3333 Lustre <p>I was suspicious when I hit this for the first time, but I hit it once more, so I am filing a bug.</p> <p>First time was in sanity test 182</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[78415.059119] LustreError: 17692:0:(mdt_recovery.c:390:mdt_last_rcvd_update()) Trying to overwrite bigger transno:on-disk: 25769825268, new: 25769825267 replay: 0. see LU-617. [78415.061562] LustreError: 17692:0:(mdt_recovery.c:390:mdt_last_rcvd_update()) Skipped 8 previous similar messages [78417.103412] LustreError: 31938:0:(pack_generic.c:1032:lustre_msg_get_opc()) incorrect message magic: a0b03b5e(msg:ffff8800aba18df0) [78417.103916] LustreError: 31938:0:(pack_generic.c:1033:lustre_msg_get_opc()) LBUG [78417.104332] Pid: 31938, comm: createmany [78417.104542] [78417.104543] Call Trace: [78417.104925] [&lt;ffffffffa0b028a5&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] [78417.105213] [&lt;ffffffffa0b02ea7&gt;] lbug_with_loc+0x47/0xb0 [libcfs] [78417.105490] [&lt;ffffffffa0d5a4ba&gt;] lustre_msg_get_opc+0x8a/0x110 [ptlrpc] [78417.105778] [&lt;ffffffffa0d5d97b&gt;] _debug_req+0x57b/0x680 [ptlrpc] [78417.106087] [&lt;ffffffff81047fde&gt;] ? kernel_map_pages+0xfe/0x110 [78417.106345] [&lt;ffffffffa0d4965e&gt;] ptlrpc_free_committed+0x20e/0x620 [ptlrpc] [78417.106619] [&lt;ffffffffa0d4b313&gt;] after_reply+0x7a3/0xd90 [ptlrpc] [78417.106878] [&lt;ffffffffa0d502c3&gt;] ptlrpc_check_set+0x1093/0x1da0 [ptlrpc] [78417.107137] [&lt;ffffffff8107eca0&gt;] ? process_timeout+0x0/0x10 [78417.107389] [&lt;ffffffffa0d512fa&gt;] ptlrpc_set_wait+0x32a/0x880 [ptlrpc] [78417.107639] [&lt;ffffffff8105ad10&gt;] ? default_wake_function+0x0/0x20 [78417.107907] [&lt;ffffffffa0d5b4c6&gt;] ? lustre_msg_set_jobid+0xb6/0x140 [ptlrpc] [78417.108187] [&lt;ffffffffa0d518cf&gt;] ptlrpc_queue_wait+0x7f/0x220 [ptlrpc] [78417.108453] [&lt;ffffffffa0d2d9c5&gt;] ldlm_cli_enqueue+0x365/0x770 [ptlrpc] [78417.119499] [&lt;ffffffff814fc8bc&gt;] ? __mutex_lock_slowpath+0x21c/0x2c0 [78417.119779] [&lt;ffffffffa0d326e0&gt;] ? ldlm_completion_ast+0x0/0x950 [ptlrpc] [78417.120071] [&lt;ffffffffa0f389e0&gt;] ? ll_md_blocking_ast+0x0/0x750 [lustre] [78417.120334] [&lt;ffffffffa0415f55&gt;] mdc_enqueue+0x795/0x18c0 [mdc] [78417.120583] [&lt;ffffffffa041727d&gt;] mdc_intent_lock+0x1fd/0x5b4 [mdc] [78417.120847] [&lt;ffffffffa0f389e0&gt;] ? ll_md_blocking_ast+0x0/0x750 [lustre] [78417.121427] [&lt;ffffffffa0d326e0&gt;] ? ldlm_completion_ast+0x0/0x950 [ptlrpc] [78417.121688] [&lt;ffffffffa03c8667&gt;] ? lmv_fid_alloc+0x117/0x4b0 [lmv] [78417.121952] [&lt;ffffffffa03e02d6&gt;] lmv_intent_open+0x1f6/0x8c0 [lmv] [78417.122206] [&lt;ffffffffa0f389e0&gt;] ? ll_md_blocking_ast+0x0/0x750 [lustre] [78417.122464] [&lt;ffffffff811661ca&gt;] ? cache_alloc_debugcheck_after+0x14a/0x210 [78417.122720] [&lt;ffffffff81168671&gt;] ? __kmalloc+0x191/0x2a0 [78417.122958] [&lt;ffffffffa03e0c4b&gt;] lmv_intent_lock+0x2ab/0x370 [lmv] [78417.123211] [&lt;ffffffffa0f389e0&gt;] ? ll_md_blocking_ast+0x0/0x750 [lustre] [78417.123471] [&lt;ffffffffa0f37a9e&gt;] ? ll_i2gids+0x2e/0xd0 [lustre] [78417.123718] [&lt;ffffffffa0f1e4ba&gt;] ? ll_prep_md_op_data+0xfa/0x3a0 [lustre] [78417.123983] [&lt;ffffffffa0f3d014&gt;] ll_lookup_it+0x3a4/0xbf0 [lustre] [78417.124273] [&lt;ffffffffa0f389e0&gt;] ? ll_md_blocking_ast+0x0/0x750 [lustre] [78417.124539] [&lt;ffffffffa0f3d8e9&gt;] ll_lookup_nd+0x89/0x3b0 [lustre] [78417.124784] [&lt;ffffffff8119b456&gt;] ? d_alloc+0x166/0x1d0 [78417.125087] [&lt;ffffffff8118e8e2&gt;] __lookup_hash+0x102/0x160 [78417.125322] [&lt;ffffffff8118ef8a&gt;] lookup_hash+0x3a/0x50 [78417.125550] [&lt;ffffffff81192edb&gt;] do_filp_open+0x2eb/0xe00 [78417.125790] [&lt;ffffffff814fae88&gt;] ? thread_return+0x4e/0x776 [78417.126034] [&lt;ffffffff8118f37b&gt;] ? getname+0x3b/0x250 [78417.126259] [&lt;ffffffff8119fd7b&gt;] ? alloc_fd+0xab/0x160 [78417.126490] [&lt;ffffffff8117e399&gt;] do_sys_open+0x69/0x140 [78417.126717] [&lt;ffffffff8100c575&gt;] ? math_state_restore+0x45/0x60 [78417.126965] [&lt;ffffffff8117e4b0&gt;] sys_open+0x20/0x30 [78417.127190] [&lt;ffffffff8100b0b2&gt;] system_call_fastpath+0x16/0x1b [78417.127428] [78417.169930] Kernel panic - not syncing: LBUG </pre> </div></div> <p>Crashdumps and modules for this one are in:<br/> /exports/crashdumps/192.168.10.224-2013-05-09-20\:43\:05</p> <p>The other crash was like this:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[79955.565718] Lustre: DEBUG MARKER: == sanity test 71: Running dbench on lustre (don't segment fault) ====== 21:01:15 (1368493275) [80393.197212] LustreError: 25536:0:(pack_generic.c:1032:lustre_msg_get_opc()) incorrect message magic: a0b03b5e(msg:ffff8800464ccdf0) [80393.197711] LustreError: 25536:0:(pack_generic.c:1033:lustre_msg_get_opc()) LBUG [80393.209355] Pid: 25536, comm: dbench [80393.209559] [80393.209559] Call Trace: [80393.209964] [&lt;ffffffffa0b028a5&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] [80393.210228] [&lt;ffffffffa0b02ea7&gt;] lbug_with_loc+0x47/0xb0 [libcfs] [80393.210502] [&lt;ffffffffa0d5a4ba&gt;] lustre_msg_get_opc+0x8a/0x110 [ptlrpc] [80393.212853] [&lt;ffffffffa0d5d97b&gt;] _debug_req+0x57b/0x680 [ptlrpc] [80393.213149] [&lt;ffffffff81047fde&gt;] ? kernel_map_pages+0xfe/0x110 [80393.213412] [&lt;ffffffffa0d4965e&gt;] ptlrpc_free_committed+0x20e/0x620 [ptlrpc] [80393.213689] [&lt;ffffffffa0d4b313&gt;] after_reply+0x7a3/0xd90 [ptlrpc] [80393.213967] [&lt;ffffffffa0d502c3&gt;] ptlrpc_check_set+0x1093/0x1da0 [ptlrpc] [80393.214225] [&lt;ffffffff8107eca0&gt;] ? process_timeout+0x0/0x10 [80393.214477] [&lt;ffffffffa0d512fa&gt;] ptlrpc_set_wait+0x32a/0x880 [ptlrpc] [80393.214733] [&lt;ffffffff8105ad10&gt;] ? default_wake_function+0x0/0x20 [80393.214995] [&lt;ffffffffa0d5b4c6&gt;] ? lustre_msg_set_jobid+0xb6/0x140 [ptlrpc] [80393.215271] [&lt;ffffffffa0d518cf&gt;] ptlrpc_queue_wait+0x7f/0x220 [ptlrpc] [80393.215526] [&lt;ffffffffa0401f59&gt;] mdc_sync+0xb9/0x230 [mdc] [80393.215773] [&lt;ffffffffa03d576e&gt;] lmv_sync+0x2ae/0x550 [lmv] [80393.216022] [&lt;ffffffffa0f0a6ab&gt;] ll_fsync+0x1ab/0x860 [lustre] [80393.216290] [&lt;ffffffff811b2401&gt;] vfs_fsync_range+0xa1/0xe0 [80393.216525] [&lt;ffffffff811b24ad&gt;] vfs_fsync+0x1d/0x20 [80393.216777] [&lt;ffffffff811b24ee&gt;] do_fsync+0x3e/0x60 [80393.217003] [&lt;ffffffff811b2540&gt;] sys_fsync+0x10/0x20 [80393.217230] [&lt;ffffffff8100b0b2&gt;] system_call_fastpath+0x16/0x1b [80393.217471] [80393.228801] Kernel panic - not syncing: LBUG </pre> </div></div> <p>/exports/crashdumps/192.168.10.224-2013-05-13-21\:08\:35/vmcore, same modules as before.</p> LU-3333

lustre_msg_get_opc()) incorrect message magic: a0b03b5 LBUG

Bug Blocker Resolved Fixed Oleg Drokin Oleg Drokin Tue, 14 May 2013 14:41:52 +0000 Fri, 6 Mar 2015 00:04:36 +0000 Thu, 1 May 2014 17:17:21 +0000 Lustre 2.3.0 Lustre 2.4.0 Lustre 2.1.5 Lustre 1.8.9 Lustre 2.6.0 Lustre 2.5.2 0 6 <p>Separate from fixing the root cause of this patch, it doesn't make sense to LBUG() on anything coming off of the network. At worst this should be turned into an error.</p> <p>I still hit this fairly frequently and caught one instance live today.</p> <p>I found the request that caused the LBUG, but in the request the rq_reqmsg does NOT match the lustre_msg pointer printed in the assertion.<br/> lustre_msg pointer seems to be pointing at some sort of structure that's has mfd handle embedded in it later on (rep buffer content? certainly sounds plausible), how did that happen I am totally not clear.<br/> the "lustre_msg" magic obviously does not match a proper one, but also does not match what's in assertion message, instead of the long number (that varies from recompile to recompile, but is the same in the same module version crashes) printed in assertion message, the in-memory value is always 0x2341</p> <p>I wonder if it's an artifact of rq_msg replacement when enlarging.<br/> Added some debug and going to rerun.</p> <p>My hunch was right, it seems.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000002:00100000:0.0:1369322005.401613:0:7254:0:(mdc_locks.c:635:mdc_finish_enqueue()) Attempting to enlarge req ffff880010aa67f0 with body ffff8800647f79f8 02000000:00100000:0.0:1369322005.401615:0:7254:0:(sec_null.c:266:null_enlarge_reqbuf()) Moved message ffff88002415ddf0 to ffff88004c1efbf0 ... 00000100:00020000:4.0:1369322005.402571:0:7256:0:(pack_generic.c:1032:lustre_msg_get_opc()) incorrect message magic: a0b04b5e(msg:ffff88002415ddf0) </pre> </div></div> <p>Unfortunately such replacement happens with no locking whatsoever and the open msg is already on the replay list. It's only a matter of (bad) luck for the race to happen where while trying to print req-&gt;rq_reqmsg the message get's moved to another place and the old one first overwritten and then freed leading to the assertion like what we've got here.</p> <p>It seems that we need to do this reallocate BEFORE we add the request to the replay list, though the comment in the addition to the list (that I remember I added some time ago before there was such an enlargement code) is advising of urgency of such adding for recovery reason.<br/> I think it's still much easier to move the list adding past request extending and increase the race a tiny bit (that would probably be still very unrealistic to hit) than add locking all around req-&gt;rq_reqmsg access.</p> <p>Tentative patch in <a href="http://review.whamcloud.com/6467" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/6467</a></p> <p>I seem to have hit this on Hyperion:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">LustreError: 9374:0:(pack_generic.c:487:lustre_msg_buf()) ASSERTION( 0 ) failed: incorrect message magic: 0000000d(msg:ffff881054aab0f8) LustreError: 9374:0:(pack_generic.c:487:lustre_msg_buf()) LBUG Pid: 9374, comm: ior Call Trace: [&lt;ffffffffa043d895&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] [&lt;ffffffffa043de97&gt;] lbug_with_loc+0x47/0xb0 [libcfs] [&lt;ffffffffa07cc5aa&gt;] lustre_msg_buf+0x4a/0x60 [ptlrpc] [&lt;ffffffffa07f894d&gt;] _sptlrpc_enlarge_msg_inplace+0x5d/0x1b0 [ptlrpc] [&lt;ffffffffa08086d5&gt;] null_enlarge_reqbuf+0xd5/0x200 [ptlrpc] [&lt;ffffffffa07f607c&gt;] sptlrpc_cli_enlarge_reqbuf+0x5c/0x160 [ptlrpc] [&lt;ffffffffa098ece6&gt;] mdc_finish_enqueue+0xb26/0x1190 [mdc] [&lt;ffffffffa07d18a0&gt;] ? lustre_swab_ldlm_reply+0x0/0x40 [ptlrpc] [&lt;ffffffffa0990860&gt;] mdc_enqueue+0xfc0/0x1b00 [mdc] [&lt;ffffffffa099159e&gt;] mdc_intent_lock+0x1fe/0x63f [mdc] [&lt;ffffffffa0a76020&gt;] ? ll_md_blocking_ast+0x0/0x7f0 [lustre] [&lt;ffffffffa07aaac0&gt;] ? ldlm_completion_ast+0x0/0x930 [ptlrpc] [&lt;ffffffffa093b6aa&gt;] ? lmv_fid_alloc+0x24a/0x3c0 [lmv] [&lt;ffffffffa0956165&gt;] lmv_intent_open+0x325/0x9c0 [lmv] [&lt;ffffffffa0a76020&gt;] ? ll_md_blocking_ast+0x0/0x7f0 [lustre] [&lt;ffffffffa0956abb&gt;] lmv_intent_lock+0x2bb/0x380 [lmv] [&lt;ffffffffa0a76020&gt;] ? ll_md_blocking_ast+0x0/0x7f0 [lustre] [&lt;ffffffffa0a57b18&gt;] ? ll_prep_md_op_data+0x1a8/0x490 [lustre] [&lt;ffffffffa0a7789a&gt;] ll_lookup_it+0x26a/0xad0 [lustre] [&lt;ffffffffa0a76020&gt;] ? ll_md_blocking_ast+0x0/0x7f0 [lustre] [&lt;ffffffffa0a7818e&gt;] ll_lookup_nd+0x8e/0x5d0 [lustre] [&lt;ffffffff811965c2&gt;] __lookup_hash+0x102/0x160 [&lt;ffffffff81196cea&gt;] lookup_hash+0x3a/0x50 [&lt;ffffffff8119b7fe&gt;] do_filp_open+0x2de/0xd20 [&lt;ffffffff81196a76&gt;] ? final_putname+0x26/0x50 [&lt;ffffffff8114fd1b&gt;] ? __vm_enough_memory+0x3b/0x190 [&lt;ffffffff811a82e2&gt;] ? alloc_fd+0x92/0x160 [&lt;ffffffff81185df9&gt;] do_sys_open+0x69/0x140 [&lt;ffffffff81185f10&gt;] sys_open+0x20/0x30 [&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b </pre> </div></div> <p>Log from the LBUG</p> <p>The landed patch from change 6467 has been reverted due to a but it introduced with sec_null types.<br/> Instead I resurrected the patchset #1 from there at <a href="http://review.whamcloud.com/10074" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/10074</a> that does not have the bug.</p> <p>Cliff,<br/> Can you confirm this fix and close this ticket if appropriate?<br/> Thank you!</p> <p>Reversal of patch completed.</p> Duplicate LU-4949 LU-4956 LU-3309 Related LU-6172 Development Rank 1|hzvqvr: Rank (Obsolete) 8241 Severity