https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4330 Lustre <p>Hi,</p> <p>More and more compute nodes of several different customer clusters are hitting an LBUG on this 'assertion failed' issue:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2013-11-21 14:06:54 LustreError: 46336:0:(events.c:433:ptlrpc_master_callback()) ASSERTION( callback == request_out_callback || callback == reply_in_callback || callback == client_bulk_callback || callback == request_in_callback || callback == reply_out_callback || callback == server_bulk_callback ) failed: 2013-11-21 14:06:54 LustreError: 46336:0:(events.c:433:ptlrpc_master_callback()) LBUG 2013-11-21 14:06:54 Nov 21 14:06:54 Pid: 46336, comm: kiblnd_sd_00 2013-11-21 14:06:54 compute5666 kernel 2013-11-21 14:06:54 : LustreError: 4Call Trace: 2013-11-21 14:06:54 6336:0:(events.c [&lt;ffffffffa041c7f5&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] 2013-11-21 14:06:54 :433:ptlrpc_mast [&lt;ffffffffa041ce07&gt;] lbug_with_loc+0x47/0xb0 [libcfs] 2013-11-21 14:06:54 er_callback()) A [&lt;ffffffffa06a426c&gt;] ptlrpc_master_callback+0xcc/0xd0 [ptlrpc] 2013-11-21 14:06:54 SSERTION( callba [&lt;ffffffffa048ebd2&gt;] lnet_enq_event_locked+0x62/0xd0 [lnet] 2013-11-21 14:06:54 ck == request_ou [&lt;ffffffffa048ecdb&gt;] lnet_finalize+0x9b/0x2f0 [lnet] 2013-11-21 14:06:54 t_callback || ca [&lt;ffffffffa083d073&gt;] kiblnd_recv+0x103/0x570 [ko2iblnd] 2013-11-21 14:06:54 llback == reply_ [&lt;ffffffffa04928dd&gt;] lnet_ni_recv+0xad/0x2f0 [lnet] 2013-11-21 14:06:54 in_callback || c [&lt;ffffffffa0492c06&gt;] lnet_recv_put+0xe6/0x120 [lnet] 2013-11-21 14:06:54 allback == clien [&lt;ffffffffa0499c33&gt;] lnet_parse+0x1273/0x1b80 [lnet] 2013-11-21 14:06:54 t_bulk_callback [&lt;ffffffff81042ca3&gt;] ? enqueue_task+0x43/0x90 2013-11-21 14:06:54 || callback == r [&lt;ffffffffa083d7ab&gt;] kiblnd_handle_rx+0x2cb/0x680 [ko2iblnd] 2013-11-21 14:06:54 equest_in_callba [&lt;ffffffffa083e590&gt;] kiblnd_rx_complete+0x2d0/0x440 [ko2iblnd] 2013-11-21 14:06:54 ck || callback = [&lt;ffffffff81042a63&gt;] ? __wake_up+0x53/0x70 2013-11-21 14:06:54 = reply_out_call [&lt;ffffffffa083e762&gt;] kiblnd_complete+0x62/0xe0 [ko2iblnd] 2013-11-21 14:06:54 back || callback [&lt;ffffffffa083eb19&gt;] kiblnd_scheduler+0x339/0x7a0 [ko2iblnd] 2013-11-21 14:06:54 == server_bulk_ [&lt;ffffffff8104a320&gt;] ? default_wake_function+0x0/0x20 2013-11-21 14:06:54 callback ) faile [&lt;ffffffffa083e7e0&gt;] ? kiblnd_scheduler+0x0/0x7a0 [ko2iblnd] 2013-11-21 14:06:54 d: 2013-11-21 14:06:54 Nov 21 14:06 [&lt;ffffffff8100412a&gt;] child_rip+0xa/0x20 2013-11-21 14:06:54 :54 compute5666 ke [&lt;ffffffffa083e7e0&gt;] ? kiblnd_scheduler+0x0/0x7a0 [ko2iblnd] 2013-11-21 14:06:54 rnel: LustreErro [&lt;ffffffffa083e7e0&gt;] ? kiblnd_scheduler+0x0/0x7a0 [ko2iblnd] 2013-11-21 14:06:54 r: 46336:0:(even [&lt;ffffffff81004120&gt;] ? child_rip+0x0/0x20 2013-11-21 14:06:54 ts.c:433:ptlrpc_ 2013-11-21 14:06:54 master_callback()) LBUG 2013-11-21 14:06:54 Nov 21 14:06:54 compute566Kernel panic - not syncing: LBUG </pre> </div></div> <p>For information, systems are running with kernel boot parameter 'tolerant=1' set.</p> <p>We have a crash dump that we will upload to ftp.</p> <p>Sebastien.</p> LU-4330

LustreError: 46336:0:(events.c:433:ptlrpc_master_callback()) ASSERTION( callback == request_out_callback || callback == reply_in_callback || callback == client_bulk_callback || callback == request_in_callback || callback == reply_out_callback ... ) failed

Bug Major Reopened Unresolved Bruno Faccini Sebastien Buisson Mon, 2 Dec 2013 14:40:13 +0000 Thu, 14 Jun 2018 21:41:20 +0000 Lustre 2.1.6 0 13 <p>Hello Sebastien, <br/> According to <a href="https://jira.whamcloud.com/browse/LU-1734" title="ptlrpc_master_callback() LBUG" class="issue-link" data-issue-key="LU-1734"><del>LU-1734</del></a>/<a href="https://jira.whamcloud.com/browse/LU-3010" title="client crashes on RHEL6 with Lustre 1.8.8" class="issue-link" data-issue-key="LU-3010"><del>LU-3010</del></a> already describing the same issue/LBUG, it has been identified to be caused by some IB-related memory corruption.<br/> Thus I am wondering if the affected systems are running with ipoib-locking-fix.patch described in <a href="https://jira.whamcloud.com/browse/LU-3848" title="Compute node crashes due to error in Lustre : ASSERTION( me == md-&gt;md_me ) failed" class="issue-link" data-issue-key="LU-3848"><del>LU-3848</del></a> (where you recently noticed "please also note that this patch on IPoIB was removed as it was found to introduce a deadlock") ?</p> <p>just uploaded vmcore + vmlinux and Lustre modules to Intel's FTP: /uploads/<a href="https://jira.whamcloud.com/browse/LU-4330" title="LustreError: 46336:0:(events.c:433:ptlrpc_master_callback()) ASSERTION( callback == request_out_callback || callback == reply_in_callback || callback == client_bulk_callback || callback == request_in_callback || callback == reply_out_callback ... ) failed" class="issue-link" data-issue-key="LU-4330">LU-4330</a>* </p> <p>Hi Bruno,</p> <p>Just a quick comment to mention that we are not running with the ipoib-locking-fix.patch, as we already run an OFED stack that integrates a more complete version of this patch, and it does not help.</p> <p>Cheers,<br/> Sebastien.</p> <p>Hugo, thanks for uploading the crash-dump, but I need the exact names to be able to get them.<br/> Sebastien, thanks for the precision.</p> <p>filenames are:<br/> <a href="https://jira.whamcloud.com/browse/LU-4330" title="LustreError: 46336:0:(events.c:433:ptlrpc_master_callback()) ASSERTION( callback == request_out_callback || callback == reply_in_callback || callback == client_bulk_callback || callback == request_in_callback || callback == reply_out_callback ... ) failed" class="issue-link" data-issue-key="LU-4330">LU-4330</a>-vmcore-tcn91-LBUG<br/> <a href="https://jira.whamcloud.com/browse/LU-4330" title="LustreError: 46336:0:(events.c:433:ptlrpc_master_callback()) ASSERTION( callback == request_out_callback || callback == reply_in_callback || callback == client_bulk_callback || callback == request_in_callback || callback == reply_out_callback ... ) failed" class="issue-link" data-issue-key="LU-4330">LU-4330</a>-vmlinux-systemmap-weak_updates.zip<br/> Let me know if you can find them, otherwise I'll upload again...<br/> Hugo</p> <p>Thanks Hugo, I got them and was able to look at the crash-dump then!<br/> BTW, analysis conclusion looks very similar of the one I just made for <a href="https://jira.whamcloud.com/browse/LU-3848" title="Compute node crashes due to error in Lustre : ASSERTION( me == md-&gt;md_me ) failed" class="issue-link" data-issue-key="LU-3848"><del>LU-3848</del></a> :</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>crash&gt; p/x ((lnet_me_t *)0xffff88053cc10dc0)-&gt;me_md-&gt;md_user_ptr $32 = 0xffff88087aa2c03d crash&gt; crash&gt; p/x *(struct ptlrpc_cb_id *)0xffff88087aa2c03d $8 = { cbid_fn = 0xffa07a2b10ffff88, cbid_arg = 0x87aa2c000ffffff } crash&gt; p/x *(struct ptlrpc_cb_id *)0xffff88087aa2c040 $9 = { cbid_fn = 0xffffffffa07a2b10, cbid_arg = 0xffff88087aa2c000 } crash&gt; x/i 0xffffffffa07a2b10 0xffffffffa07a2b10 &lt;request_in_callback&gt;: push %rbp crash&gt; </pre> </div></div> <p>which means that current MD's md_user_ptr field/pointer seems to have been decremented by 3 !!! Just like the ME's md_me pointer in crash-dump for <a href="https://jira.whamcloud.com/browse/LU-3848" title="Compute node crashes due to error in Lustre : ASSERTION( me == md-&gt;md_me ) failed" class="issue-link" data-issue-key="LU-3848"><del>LU-3848</del></a> that occured on an other Client of same Cluster/Customer.<br/> This is again in Slabs size-128, no OOM logs but VTune/Emon driver loaded.</p> <p>So, if we add <a href="https://jira.whamcloud.com/browse/LU-3848" title="Compute node crashes due to error in Lustre : ASSERTION( me == md-&gt;md_me ) failed" class="issue-link" data-issue-key="LU-3848"><del>LU-3848</del></a> and this ticket's symptoms, it looks like we suffer a new situation like the one described in <a href="https://jira.whamcloud.com/browse/LU-3010" title="client crashes on RHEL6 with Lustre 1.8.8" class="issue-link" data-issue-key="LU-3010"><del>LU-3010</del></a> !!!</p> <p>Since the same kind of corruption occured in a ME for <a href="https://jira.whamcloud.com/browse/LU-3848" title="Compute node crashes due to error in Lustre : ASSERTION( me == md-&gt;md_me ) failed" class="issue-link" data-issue-key="LU-3848"><del>LU-3848</del></a> and in the MD for this ticket, both being allocated in Slab/kmem-cache size-128. <br/> So, as per Liang's advice, I will try to develop a patch where MEs and small MDs (&lt;128bytes) will be allocated in a specific kmem-cache. On the other hand and specifically for MDs, I may also try to use/define LNET_USE_LIB_FREELIST, this way we will use full-size and pre-allocated MDs …</p> <p>It is still a bit early to directly link this ticket to <a href="https://jira.whamcloud.com/browse/LU-3848" title="Compute node crashes due to error in Lustre : ASSERTION( me == md-&gt;md_me ) failed" class="issue-link" data-issue-key="LU-3848"><del>LU-3848</del></a>, because even if it concern the same kind of corruption (-3), the same size-128 kmem-cache, a node of the same cluster/system, the offset in the affected Slab/structure is different.</p> <p>Hi all, best wishes for 2014!!</p> <p>after upgrading the clients to 2.1.6, this crash has again occured, this hopefully helps in finding the underlying problem...</p> <p>1388685619 2014 Jan 2 19:00:19 tcn19 kern emerg kernel LustreError: 3259:0:(events.c:433:ptlrpc_master_callback()) ASSERTION( callback == request<br/> _out_callback || callback == reply_in_callback || callback == client_bulk_callback || callback == request_in_callback || callback == reply_out_cal<br/> lback || callback == server_bulk_callback ) failed: <br/> 1388685619 2014 Jan 2 19:00:19 tcn19 kern emerg kernel LustreError: 3259:0:(events.c:433:ptlrpc_master_callback()) LBUG<br/> 1388685619 2014 Jan 2 19:00:19 tcn19 kern warning kernel Pid: 3259, comm: kiblnd_sd_04<br/> 1388685619 2014 Jan 2 19:00:19 tcn19 kern warning kernel <br/> 1388685619 2014 Jan 2 19:00:19 tcn19 kern warning kernel Call Trace:<br/> 1388685619 2014 Jan 2 19:00:19 tcn19 kern warning kernel <span class="error">&#91;&lt;ffffffffa05317f5&gt;&#93;</span> libcfs_debug_dumpstack+0x55/0x80 <span class="error">&#91;libcfs&#93;</span><br/> 1388685619 2014 Jan 2 19:00:19 tcn19 kern warning kernel <span class="error">&#91;&lt;ffffffffa0531e07&gt;&#93;</span> lbug_with_loc+0x47/0xb0 <span class="error">&#91;libcfs&#93;</span><br/> 1388685619 2014 Jan 2 19:00:19 tcn19 kern warning kernel <span class="error">&#91;&lt;ffffffffa07a225c&gt;&#93;</span> ptlrpc_master_callback+0xcc/0xd0 <span class="error">&#91;ptlrpc&#93;</span><br/> 1388685619 2014 Jan 2 19:00:19 tcn19 kern warning kernel <span class="error">&#91;&lt;ffffffffa05a3bd2&gt;&#93;</span> lnet_enq_event_locked+0x62/0xd0 <span class="error">&#91;lnet&#93;</span><br/> 1388685619 2014 Jan 2 19:00:19 tcn19 kern warning kernel <span class="error">&#91;&lt;ffffffffa05a3cdb&gt;&#93;</span> lnet_finalize+0x9b/0x2f0 <span class="error">&#91;lnet&#93;</span><br/> 1388685619 2014 Jan 2 19:00:19 tcn19 kern warning kernel <span class="error">&#91;&lt;ffffffffa093b043&gt;&#93;</span> kiblnd_recv+0x103/0x570 <span class="error">&#91;ko2iblnd&#93;</span><br/> 1388685619 2014 Jan 2 19:00:19 tcn19 kern warning kernel <span class="error">&#91;&lt;ffffffffa05a78dd&gt;&#93;</span> lnet_ni_recv+0xad/0x2f0 <span class="error">&#91;lnet&#93;</span><br/> 1388685619 2014 Jan 2 19:00:19 tcn19 kern warning kernel <span class="error">&#91;&lt;ffffffffa05a7c06&gt;&#93;</span> lnet_recv_put+0xe6/0x120 <span class="error">&#91;lnet&#93;</span><br/> 1388685619 2014 Jan 2 19:00:19 tcn19 kern warning kernel <span class="error">&#91;&lt;ffffffffa05aec33&gt;&#93;</span> lnet_parse+0x1273/0x1b80 <span class="error">&#91;lnet&#93;</span><br/> 1388685619 2014 Jan 2 19:00:19 tcn19 kern warning kernel <span class="error">&#91;&lt;ffffffffa093b77b&gt;&#93;</span> kiblnd_handle_rx+0x2cb/0x680 <span class="error">&#91;ko2iblnd&#93;</span><br/> 1388685619 2014 Jan 2 19:00:19 tcn19 kern warning kernel <span class="error">&#91;&lt;ffffffff8103d349&gt;&#93;</span> ? __wake_up_common+0x59/0x90<br/> 1388685619 2014 Jan 2 19:00:19 tcn19 kern warning kernel <span class="error">&#91;&lt;ffffffffa093c560&gt;&#93;</span> kiblnd_rx_complete+0x2d0/0x440 <span class="error">&#91;ko2iblnd&#93;</span><br/> 1388685619 2014 Jan 2 19:00:19 tcn19 kern warning kernel <span class="error">&#91;&lt;ffffffff81042cf3&gt;&#93;</span> ? __wake_up+0x53/0x70<br/> 1388685619 2014 Jan 2 19:00:19 tcn19 kern warning kernel <span class="error">&#91;&lt;ffffffffa093c732&gt;&#93;</span> kiblnd_complete+0x62/0xe0 <span class="error">&#91;ko2iblnd&#93;</span><br/> 1388685619 2014 Jan 2 19:00:19 tcn19 kern warning kernel <span class="error">&#91;&lt;ffffffffa093cae9&gt;&#93;</span> kiblnd_scheduler+0x339/0x7a0 <span class="error">&#91;ko2iblnd&#93;</span><br/> 1388685619 2014 Jan 2 19:00:19 tcn19 kern warning kernel <span class="error">&#91;&lt;ffffffff81048df0&gt;&#93;</span> ? default_wake_function+0x0/0x20<br/> 1388685619 2014 Jan 2 19:00:19 tcn19 kern warning kernel <span class="error">&#91;&lt;ffffffffa093c7b0&gt;&#93;</span> ? kiblnd_scheduler+0x0/0x7a0 <span class="error">&#91;ko2iblnd&#93;</span><br/> 1388685619 2014 Jan 2 19:00:19 tcn19 kern warning kernel <span class="error">&#91;&lt;ffffffff8100412a&gt;&#93;</span> child_rip+0xa/0x20<br/> 1388685620 2014 Jan 2 19:00:20 tcn19 kern warning kernel <span class="error">&#91;&lt;ffffffffa093c7b0&gt;&#93;</span> ? kiblnd_scheduler+0x0/0x7a0 <span class="error">&#91;ko2iblnd&#93;</span><br/> 1388685620 2014 Jan 2 19:00:20 tcn19 kern warning kernel <span class="error">&#91;&lt;ffffffffa093c7b0&gt;&#93;</span> ? kiblnd_scheduler+0x0/0x7a0 <span class="error">&#91;ko2iblnd&#93;</span><br/> 1388685620 2014 Jan 2 19:00:20 tcn19 kern warning kernel <span class="error">&#91;&lt;ffffffff81004120&gt;&#93;</span> ? child_rip+0x0/0x20</p> <p>Hello Hugo, Can you also upload this new crash-dump (and required vmlinux/lustre-modules additionnal stuff) with 2.1.6, so I can double-check if we still suffer the same kind of corruption ??</p> <p>On the other hand and if still same issue as very strongly expected, I will try to come up with a patch to implement the work-around (move allocation of MEs and small MDs out of &lt;size-128&gt; SLABs kmem-cache) described before.</p> <p>I'm afraid no crash-dump on this one available, I will send one on the next occurance...</p> <p>Bruno, Liang: This weekend I was able to obtain kernel coredumps again with this error (after a stable period os 3-4 weeks). As mentioned earlier the lustre version has been upgraded to 2.1.6. The uploaded file is LU4330-tcn82-vmcore-vmlinux-systemmap-weak-updates.zip; please let me know if you need additional files...</p> <p>Hello Hugo, I will review the new crash-dump you uploaded to verify we still encounter the same hind of corruption.</p> <p>BTW, I have pushed a patch that should help to better identify the root cause of the MEs/MDs corruption in Slabs, for both <a href="https://jira.whamcloud.com/browse/LU-3848" title="Compute node crashes due to error in Lustre : ASSERTION( me == md-&gt;md_me ) failed" class="issue-link" data-issue-key="LU-3848"><del>LU-3848</del></a> and this ticket, it is available at <a href="http://review.whamcloud.com/8819/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8819/</a>. It has already been successfully exposed to autotests, and you may want to give it a try. May be Sebastien could also validate it and try to integrate it in Bull distro?</p> <p>Since this patch extracts LNET MEs and small-MDs allocation from generic &lt;size-128&gt; kmem_cache, I strongly expect that you will no longer encounter <a href="https://jira.whamcloud.com/browse/LU-3848" title="Compute node crashes due to error in Lustre : ASSERTION( me == md-&gt;md_me ) failed" class="issue-link" data-issue-key="LU-3848"><del>LU-3848</del></a>/<a href="https://jira.whamcloud.com/browse/LU-4330" title="LustreError: 46336:0:(events.c:433:ptlrpc_master_callback()) ASSERTION( callback == request_out_callback || callback == reply_in_callback || callback == client_bulk_callback || callback == request_in_callback || callback == reply_out_callback ... ) failed" class="issue-link" data-issue-key="LU-4330">LU-4330</a> LBUGs (and may be suffer new issues with others Kernel codes using &lt;size-128&gt; …). But if not, and you are able to reproduce the LBUGs, then I will need to investigate further.</p> <p>just to keep the info with <a href="https://jira.whamcloud.com/browse/LU-3848" title="Compute node crashes due to error in Lustre : ASSERTION( me == md-&gt;md_me ) failed" class="issue-link" data-issue-key="LU-3848"><del>LU-3848</del></a> in sync:<br/> Hi Bruno and Liang; <br/> @Bruno: Thanks for the patch, I've asked Sebastien to prepare it for me to test at the client site. I do have plenty more of crashes available right now, but I'm not sure if this would be a huge help at this time.<br/> @Liang: Your patch has been integrated to our tree, but just after our upgrade to 2.1.6, so I'm not running this patch yet. It will be in the next lustre package.<br/> I'll try to give you feedback on these as soon as possible.</p> <p>Hello Hugo,<br/> Thanks again for your help on this!<br/> I just had a look into the recent crash-dump you uploaded in LU4330-tcn82-vmcore-vmlinux-systemmap-weak-updates.zip, and I can confirm this the same scenario, with the same/-3 corruption of MD's md_user_ptr field/pointer.</p> <p>I need to indicate that after successful auto-tests, I made local extensive testing of <a href="http://review.whamcloud.com/8819/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8819/</a> and I can confirm it works as expected and with no MEs/MDs leak in Slabs.</p> <p>As an update: the fix has now been installed for a couple of weeks and no more of these LBUG's have been seen, so the fix looks good...</p> <p>Hello Hugo, thank's for your feedback!<br/> The fact that both asserts for this ticket and <a href="https://jira.whamcloud.com/browse/LU-3848" title="Compute node crashes due to error in Lustre : ASSERTION( me == md-&gt;md_me ) failed" class="issue-link" data-issue-key="LU-3848"><del>LU-3848</del></a> disappeared seems to confirm what I suspected from the beginning, that LNET/Lustre were not involved with the corruptions but only victims. BTW, did you notice any other sub-system (networking, ...) disfunction since ?</p> <p>On the other-hand I am now working on a more generic (not based on hard-coded 128 bytes length) patch version to address Liang's last input, and also answer to Isaac comments too. Ans also push a master version as Isaac requested.</p> <p>Hi Bruno, no disfunction seen, the fix seems to solve the issue completely; thanks! Hugo</p> <p>Faccini Bruno (bruno.faccini@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/18586" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/18586</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-4330" title="LustreError: 46336:0:(events.c:433:ptlrpc_master_callback()) ASSERTION( callback == request_out_callback || callback == reply_in_callback || callback == client_bulk_callback || callback == request_in_callback || callback == reply_out_callback ... ) failed" class="issue-link" data-issue-key="LU-4330">LU-4330</a> lnet: Allocate MEs and small MDs in own kmem_caches<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 11e4b34d75505476f363c1cf4400755a7f30f766</p> <p>The scary thing is that there would continue to be random memory corruptions in the size-128 slab, but they will just be corrupting some other part of memory.</p> <p>If this problem can be found in a relatively short amount of testing time, then there are debugging patches available that could be applied to the kernel to make all kmalloc() calls actually map to vmalloc() internally and have vmalloc() always use a new memory address, and then when the memory is freed the page is unmapped and the address never used again. If another thread is incorrectly accessing an unmapped address (use after free) it will fault and then the source of the corruption may be found. Unfortunately, this impacts the performance and can only be used for debugging and not in production.</p> <p>Patches are available in <a href="https://bugzilla.lustre.org/show_bug.cgi?id=22471" class="external-link" target="_blank" rel="nofollow noopener">https://bugzilla.lustre.org/show_bug.cgi?id=22471</a> but the would likely need to be updated for newer kernels. They can definitely help find memory corruption problems that are otherwise very difficult to find.</p> <p>Bruno, if you update those patches for newer kernels, please submit the new patches into lustre/kernel_patches/patches so they are available for use in the future, since bugzilla may disappear at some point.</p> <p>I think I might know what the problem is. A recent patch for ko2iblnd in the upstream kernel landed that exposed a serious memory corruption.<br/> The commit is 3d1477309806459d39e13d8c3206ba35d183c34a "Replace sg++ with sg = sg_next(sg)" The scatter gather list is from <br/> tx-&gt;tx_frags which is IBLND_MAX_RDMA_FRAGS in size. Since you write at an offset into the tx_frags that mean you really need IBLND_MAX_RDMA_FRAGS + 1 in size for the frags. Currently the upstream clients will crash when you attempt to access scatter list entry IBLND_MAX_RDMA_FRAGS + 1.</p> <p>James,<br/> Thanks for your input.<br/> But well, I can not tell for this old case/crash, but on my side I have seen recent &lt;size-128&gt; Slabs corruption cases that are ext4/ldiskfs related (see <a href="https://jira.whamcloud.com/browse/LU-7980" title="Overrun in generic &lt;size-128&gt; kmem_cache Slabs causing OSS to crash" class="issue-link" data-issue-key="LU-7980"><del>LU-7980</del></a>).</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/18586/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/18586/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-4330" title="LustreError: 46336:0:(events.c:433:ptlrpc_master_callback()) ASSERTION( callback == request_out_callback || callback == reply_in_callback || callback == client_bulk_callback || callback == request_in_callback || callback == reply_out_callback ... ) failed" class="issue-link" data-issue-key="LU-4330">LU-4330</a> lnet: Allocate MEs and small MDs in own kmem_caches<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 9d9bb678d6b3707623845e0ce67dd7fd07a12fe9</p> <p>Landed for 2.9</p> <p>Attempted to merge the patch for <a href="https://jira.whamcloud.com/browse/LU-4330" title="LustreError: 46336:0:(events.c:433:ptlrpc_master_callback()) ASSERTION( callback == request_out_callback || callback == reply_in_callback || callback == client_bulk_callback || callback == request_in_callback || callback == reply_out_callback ... ) failed" class="issue-link" data-issue-key="LU-4330">LU-4330</a> upstream and it was rejected since according to Greg it will not fix the problem. See email</p> <p><a href="http://driverdev.linuxdriverproject.org/pipermail/driverdev-devel/2016-June/090787.html" class="external-link" target="_blank" rel="nofollow noopener">http://driverdev.linuxdriverproject.org/pipermail/driverdev-devel/2016-June/090787.html</a></p> <p>Bruno can you reproduce this problem? Have you tried debugging with slub?</p> <p>James, sorry to be late to comment/answer you.<br/> In fact, I should have re-phrase my commit msg before submitting patch upstream, seems that Greg thought that this patch has been created to fix the corruotions issues, when it has been created to pursue with the corruption isolation, by first avoiding LNet MEs/small-MDs to be impacted. Then the idea came up that moving LNet MEs/small-MDs to a specific kmem_cache has been developed and seen as a possible performance enhancement.<br/> BTW, Greg seems curious about getting some numbers/stats about this enhancement!</p> <p>Regardless of what upstream think of this fix, I think that we no longer need to track it for 2.9 so I will remove the fix version</p> Related LU-3848 LU-8362 Development Rank 1|hzwacn: Rank (Obsolete) 11850 Severity