https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1044 Lustre <p>2012-01-26 10:35:09 LustreError: 15531:0:(lustre_net.h:761:lustre_set_req_swabbed()) ASSERTION((req-&gt;rq_req_swab_mask &amp; (1 &lt;&lt; index)) == 0) failed<br/> 2012-01-26 10:35:09 LustreError: 15531:0:(lustre_net.h:761:lustre_set_req_swabbed()) LBUG<br/> 2012-01-26 10:35:09 Jan 26 10:35:09 Pid: 15531, comm: ldlm_cn_09<br/> 2012-01-26 10:35:09 momus31 kernel:<br/> 2012-01-26 10:35:09 LustreError: 15531:0:(lustre_netCall Trace:<br/> 2012-01-26 10:35:09 .h:761:lustre_se <span class="error">&#91;&lt;ffffffffa054e895&gt;&#93;</span> libcfs_debug_dumpstack+0x55/0x80 <span class="error">&#91;libcfs&#93;</span><br/> 2012-01-26 10:35:09 t_req_swabbed()) <span class="error">&#91;&lt;ffffffffa054edc7&gt;&#93;</span> lbug_with_loc+0x47/0xb0 <span class="error">&#91;libcfs&#93;</span><br/> 2012-01-26 10:35:09 LBUG<br/> 2012-01-26 10:35:09 <span class="error">&#91;&lt;ffffffffa055842d&gt;&#93;</span> libcfs_assertion_failed+0x2d/0x30 <span class="error">&#91;libcfs&#93;</span><br/> 2012-01-26 10:35:09 <span class="error">&#91;&lt;ffffffffa07f8189&gt;&#93;</span> ptlrpc_buf_set_swabbed+0xc9/0x130 <span class="error">&#91;ptlrpc&#93;</span><br/> 2012-01-26 10:35:09 <span class="error">&#91;&lt;ffffffffa08182a1&gt;&#93;</span> __req_capsule_get+0x3f1/0x5a0 <span class="error">&#91;ptlrpc&#93;</span><br/> 2012-01-26 10:35:09 <span class="error">&#91;&lt;ffffffffa07f8590&gt;&#93;</span> ? lustre_swab_ldlm_request+0x0/0x30 <span class="error">&#91;ptlrpc&#93;</span><br/> 2012-01-26 10:35:09 <span class="error">&#91;&lt;ffffffffa0818585&gt;&#93;</span> req_capsule_client_get+0x15/0x20 <span class="error">&#91;ptlrpc&#93;</span><br/> 2012-01-26 10:35:09 <span class="error">&#91;&lt;ffffffffa07d7b7a&gt;&#93;</span> ldlm_cancel_hpreq_check+0xba/0x1e0 <span class="error">&#91;ptlrpc&#93;</span><br/> 2012-01-26 10:35:09 <span class="error">&#91;&lt;ffffffffa07fea53&gt;&#93;</span> ptlrpc_server_handle_req_in+0x7a3/0x8e0 <span class="error">&#91;ptlrpc&#93;</span><br/> 2012-01-26 10:35:09 <span class="error">&#91;&lt;ffffffffa0802b96&gt;&#93;</span> ptlrpc_main+0x6e6/0x1690 <span class="error">&#91;ptlrpc&#93;</span><br/> 2012-01-26 10:35:09 <span class="error">&#91;&lt;ffffffffa08024b0&gt;&#93;</span> ? ptlrpc_main+0x0/0x1690 <span class="error">&#91;ptlrpc&#93;</span><br/> 2012-01-26 10:35:09 <span class="error">&#91;&lt;ffffffff8100c14a&gt;&#93;</span> child_rip+0xa/0x20<br/> 2012-01-26 10:35:09 <span class="error">&#91;&lt;ffffffffa08024b0&gt;&#93;</span> ? ptlrpc_main+0x0/0x1690 <span class="error">&#91;ptlrpc&#93;</span><br/> 2012-01-26 10:35:09 <span class="error">&#91;&lt;ffffffffa08024b0&gt;&#93;</span> ? ptlrpc_main+0x0/0x1690 <span class="error">&#91;ptlrpc&#93;</span><br/> 2012-01-26 10:35:09 <span class="error">&#91;&lt;ffffffff8100c140&gt;&#93;</span> ? child_rip+0x0/0x20<br/> 2012-01-26 10:35:09<br/> 2012-01-26 10:35:09 Kernel panic - not syncing: LBUG<br/> 2012-01-26 10:35:09 Pid: 15531, comm: ldlm_cn_09 Not tainted 2.6.32-220.2.1.3chaos.ch5.x86_64 #1<br/> 2012-01-26 10:35:09 Jan 26 10:35:09 Call Trace:<br/> 2012-01-26 10:35:09 momus31 kernel: <span class="error">&#91;&lt;ffffffff814edddb&gt;&#93;</span> ? panic+0x78/0x143<br/> 2012-01-26 10:35:09 Kernel panic - n <span class="error">&#91;&lt;ffffffffa054ee1b&gt;&#93;</span> ? lbug_with_loc+0x9b/0xb0 <span class="error">&#91;libcfs&#93;</span><br/> 2012-01-26 10:35:09 ot syncing: LBUG <span class="error">&#91;&lt;ffffffffa055842d&gt;&#93;</span> ? libcfs_assertion_failed+0x2d/0x30 <span class="error">&#91;libcfs&#93;</span><br/> 2012-01-26 10:35:09<br/> 2012-01-26 10:35:09 <span class="error">&#91;&lt;ffffffffa07f8189&gt;&#93;</span> ? ptlrpc_buf_set_swabbed+0xc9/0x130 <span class="error">&#91;ptlrpc&#93;</span><br/> 2012-01-26 10:35:09 <span class="error">&#91;&lt;ffffffffa08182a1&gt;&#93;</span> ? __req_capsule_get+0x3f1/0x5a0 <span class="error">&#91;ptlrpc&#93;</span><br/> 2012-01-26 10:35:09 <span class="error">&#91;&lt;ffffffffa07f8590&gt;&#93;</span> ? lustre_swab_ldlm_request+0x0/0x30 <span class="error">&#91;ptlrpc&#93;</span><br/> 2012-01-26 10:35:09 <span class="error">&#91;&lt;ffffffffa0818585&gt;&#93;</span> ? req_capsule_client_get+0x15/0x20 <span class="error">&#91;ptlrpc&#93;</span><br/> 2012-01-26 10:35:09 <span class="error">&#91;&lt;ffffffffa07d7b7a&gt;&#93;</span> ? ldlm_cancel_hpreq_check+0xba/0x1e0 <span class="error">&#91;ptlrpc&#93;</span><br/> 2012-01-26 10:35:09 <span class="error">&#91;&lt;ffffffffa07fea53&gt;&#93;</span> ? ptlrpc_server_handle_req_in+0x7a3/0x8e0 <span class="error">&#91;ptlrpc&#93;</span><br/> 2012-01-26 10:35:09 <span class="error">&#91;&lt;ffffffffa0802b96&gt;&#93;</span> ? ptlrpc_main+0x6e6/0x1690 <span class="error">&#91;ptlrpc&#93;</span><br/> 2012-01-26 10:35:09 <span class="error">&#91;&lt;ffffffffa08024b0&gt;&#93;</span> ? ptlrpc_main+0x0/0x1690 <span class="error">&#91;ptlrpc&#93;</span><br/> 2012-01-26 10:35:09 <span class="error">&#91;&lt;ffffffff8100c14a&gt;&#93;</span> ? child_rip+0xa/0x20<br/> 2012-01-26 10:35:09 <span class="error">&#91;&lt;ffffffffa08024b0&gt;&#93;</span> ? ptlrpc_main+0x0/0x1690 <span class="error">&#91;ptlrpc&#93;</span><br/> 2012-01-26 10:35:09 <span class="error">&#91;&lt;ffffffffa08024b0&gt;&#93;</span> ? ptlrpc_main+0x0/0x1690 <span class="error">&#91;ptlrpc&#93;</span><br/> 2012-01-26 10:35:09 <span class="error">&#91;&lt;ffffffff8100c140&gt;&#93;</span> ? child_rip+0x0/0x20</p> toss 2.0-beta12.ch5<br/> lustre-2.1.0-17chaos_2.6.32_220.2.1.3chaos.ch5.x86_64.x86_64 LU-1044

ASSERTION((req->rq_req_swab_mask & (1 << index)) == 0) failed

Bug Blocker Resolved Cannot Reproduce Jinshan Xiong Prakash Surya Thu, 26 Jan 2012 17:03:32 +0000 Mon, 5 Mar 2012 09:23:20 +0000 Mon, 5 Mar 2012 09:23:20 +0000 Lustre 2.1.0 Lustre 1.8.x (1.8.0 - 1.8.5) 0 6 <p>Initially, it appears as though the swap bit in question is being modified in between access in this code path:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">if</span> (ptlrpc_buf_need_swab(req, inout, offset)) { lustre_swab_ptlrpc_body(pb); ptlrpc_buf_set_swabbed(req, inout, offset); } </pre> </div></div> <p>`ptlrpc_buf_need_swab` will only return true if `req-&gt;rq_req_swab_mask &amp; (1 &lt;&lt; index)` is false. Thus, if it's calling `ptlrpc_buf_set_swabbed`, that must mean that `req-&gt;rq_req_swab_mask &amp; (1 &lt;&lt; index)` <em>is</em> false when it's executed within `ptlrpc_buf_need_swab`. But then when that same check it executed in `ptlrpc_buf_set_swabbed` its true (hence the LBUG).</p> <p>Where is the lock taken on `req` in this code path? The only way I can see this happening is if another thread swabs `req` in between the two above accesses. And that would imply improper lock handling on `req`.</p> <p>I was going down the wrong path earlier, the real call to `ptlrpc_buf_set_swabbed` I believe happened here:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">if</span> (ptlrpc_buf_need_swab(pill-&gt;rc_req, inout, offset) &amp;&amp; swabber != NULL &amp;&amp; value != NULL) do_swab = 1; <span class="code-keyword">else</span> do_swab = 0; <span class="code-keyword">if</span> (!(field-&gt;rmf_flags &amp; RMF_F_STRUCT_ARRAY)) { <span class="code-keyword">if</span> (dump &amp;&amp; field-&gt;rmf_dumper) { CDEBUG(D_RPCTRACE, <span class="code-quote">"Dump of %sfield %s follows\n"</span>, do_swab ? <span class="code-quote">"unswabbed "</span> : "", field-&gt;rmf_name); field-&gt;rmf_dumper(value); } <span class="code-keyword">if</span> (!do_swab) <span class="code-keyword">return</span>; swabber(value); ptlrpc_buf_set_swabbed(pill-&gt;rc_req, inout, offset); <span class="code-keyword">if</span> (dump) { CDEBUG(D_RPCTRACE, <span class="code-quote">"Dump of swabbed field %s "</span> <span class="code-quote">"follows\n"</span>, field-&gt;rmf_name); field-&gt;rmf_dumper(value); } <span class="code-keyword">return</span>; } </pre> </div></div> <p>Although, I think the same logic regarding `req-&gt;rq_req_swab_mask` changing in between calls to `ptlrpc_buf_need_swab` and `ptlrpc_buf_set_swabbed` still stands.</p> <p>I did a quick search for ldlm_cancel_hpreq_check and I don't see it anywhere in our code neither 2.x nor 1.8.x.<br/> Is this some sort of a patch you carry? The only patch I am able to find that has a function with this name is some sort of super old version in bug 22107, and even then it's for 1.x only</p> <p>Also did this crash happen on a client? What version was the client running?</p> <p>Also, is the client a PPC BlueGene system?</p> <p>Oleg: The ldlm_cancel_hpreq_check was introduced by one of the <a href="https://jira.whamcloud.com/browse/LU-874" title="Client eviction on lock callback timeout " class="issue-link" data-issue-key="LU-874"><del>LU-874</del></a> patches we are carrying locally (since they haven't landed yet AFAIK). As such, it might be a good idea to point Jinshan to this thread since he was a big contributor to those patches. In particular, that function was introduced by change-id: I1a300916b4b3b592ca565ffc06cb3658d699d7a0.</p> <p>Andreas: I can't say for sure, but after talking with Chris and Ned, I think the consensus is that we do still indeed have a PPC BG machine out in our environment mounting this filesystem. I don't know of any other reason why scrubbing would be necessary. Although, I'll check with one of the admins to make sure of this.</p> <p>jinshan</p> <p>Could you please look into this possible problem with one of the LU874 patches?</p> <p>Thanks</p> <p>Peter</p> <p>And just for completeness, here's the <a href="https://jira.whamcloud.com/browse/LU-874" title="Client eviction on lock callback timeout " class="issue-link" data-issue-key="LU-874"><del>LU-874</del></a> patch stack we are carrying locally:</p> <p>Change-id: If14eff6361f55d2b2eeb2db7146789dda4c32060<br/> Change-id: I2fa99a0f09a42e1333589fc7bc2a6eebef4924b6<br/> Change-id: I33e2d113d7929a56389741c06dffb5efb6bf28a3<br/> Change-id: I1a300916b4b3b592ca565ffc06cb3658d699d7a0</p> <p>I can confirm, we do have PPC bluegene clients mounting this file system. Both IO nodes, and login nodes.</p> <p>So, is the crash happening on OST?</p> <p>Oleg: Yeah, the OSS.</p> <p>Oleg: Our branch is available at github.com/chaos/lustre.</p> <p>Can you please use this patch(in attachment) as a workaround fix? Because <a href="https://jira.whamcloud.com/browse/LU-874" title="Client eviction on lock callback timeout " class="issue-link" data-issue-key="LU-874"><del>LU-874</del></a> <a href="http://review.whamcloud.com/#change,1918" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,1918</a> is still in inspection, so I'd like to merge it into that patch instead of having a standalone one.</p> <p>Thanks oleg for helping me find this out issue. There is a similar issue(<a href="https://jira.whamcloud.com/browse/LU-898" title="When copy data to 2.1 cluster via 1.8 client, all 2.1 OSS servers panic." class="issue-link" data-issue-key="LU-898"><del>LU-898</del></a> review.whamcloud.com/1799) which was merged, but we may not need that patch to make it work.</p> <p>Jinshan, since this will be running in a production environment, we are reluctant to apply the patch with it not submitted through Gerrit. Would you be able to merge it into that patch and push a new revision?</p> <p>Hi Prakash, please take a look at patch set 10 of <a href="http://review.whamcloud.com/#change,1918" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,1918</a>. I'll urge guys to inspect it.</p> <p>Thanks, Jinshan.</p> <p>No additional instances of this bug. I am ready to believe it introduced by the earlier revision <a href="https://jira.whamcloud.com/browse/LU-874" title="Client eviction on lock callback timeout " class="issue-link" data-issue-key="LU-874"><del>LU-874</del></a>. This can be closed or at least the priority reduced to "minor".</p> <p>ok then let's close for now and reopen if it reappears</p> Development Rank 1|hzvhg7: Rank (Obsolete) 6476 Severity