https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-8573 Lustre <p>Attempted IOR test on both sets of OSTs. <br/> No progress made on test.<br/> Client and OST logs are full of:<br/> OST</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">LustreError: 5245:0:(events.c:449:server_bulk_callback()) event type 3, status -113, desc ffff88104dc53200 LustreError: 9071:0:(events.c:449:server_bulk_callback()) event type 3, status -103, desc ffff880b0c956200 LustreError: 5245:0:(events.c:449:server_bulk_callback()) event type 5, status -113, desc ffff880b13e95200 LustreError: 5245:0:(events.c:449:server_bulk_callback()) event type 3, status -113, desc ffff880b13e95200 LustreError: 5245:0:(events.c:449:server_bulk_callback()) event type 5, status -113, desc ffff881032b5a800 LustreError: 5245:0:(events.c:449:server_bulk_callback()) event type 3, status -113, desc ffff881032b5a800 </pre> </div></div> <p>Client</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">[12378.313197] LustreError: 3138:0:(events.c:203:client_bulk_callback()) event type 1, status -5, desc ffff8807442f9c00 [12378.313223] LustreError: 3136:0:(events.c:203:client_bulk_callback()) event type 1, status -5, desc ffff8807442f9800 [12378.337649] LustreError: 3136:0:(events.c:203:client_bulk_callback()) event type 1, status -5, desc ffff8810102c4200 [12378.337671] LustreError: 3136:0:(events.c:203:client_bulk_callback()) event type 1, status -5, desc ffff880f55547400 [12378.337677] LustreError: 3138:0:(events.c:203:client_bulk_callback()) event type 1, status -5, desc ffff880729970000 [12378.362167] LustreError: 3137:0:(events.c:203:client_bulk_callback()) event type 1, status -5, desc ffff880804b78800 [12378.374324] LustreError: 3137:0:(events.c:203:client_bulk_callback()) event type 1, status -5, desc ffff8807299e5200 [12378.410886] LustreError: 3137:0:(events.c:203:client_bulk_callback()) event type 1, status -5, desc ffff880f53ce1c00 </pre> </div></div> <p>Eventually, one client LBUGS</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">[12377.942289] LustreError: 3153:0:(niobuf.c:319:ptlrpc_register_bulk()) ASSERTION( desc-&gt;bd_md_count == 0 ) failed: [12378.564210] LustreError: 3153:0:(niobuf.c:319:ptlrpc_register_bulk()) LBUG [12378.564210] LustreError: 3153:0:(niobuf.c:319:ptlrpc_register_bulk()) LBUG [12378.571890] Pid: 3153, comm: ptlrpcd_01_02 [12378.576468] Call Trace: [12378.580868] [&lt;ffffffffa08557d3&gt;] libcfs_debug_dumpstack+0x53/0x80 [libcfs] [12378.588652] [&lt;ffffffffa0855d75&gt;] lbug_with_loc+0x45/0xc0 [libcfs] [12378.595597] [&lt;ffffffffa0c3e661&gt;] ptlrpc_register_bulk+0x831/0x9c0 [ptlrpc] [12378.603389] [&lt;ffffffffa08cace2&gt;] ? LNetMDUnlink+0xe2/0x180 [lnet] [12378.610322] [&lt;ffffffffa0c6be76&gt;] ? sptlrpc_import_sec_ref+0x36/0x40 [ptlrpc] [12378.618321] [&lt;ffffffffa0c3f1af&gt;] ptl_send_rpc+0x1ff/0xda0 [ptlrpc] [12378.625361] [&lt;ffffffffa0c39256&gt;] ptlrpc_check_set.part.23+0x1896/0x1dd0 [ptlrpc] [12378.633743] [&lt;ffffffffa0c397eb&gt;] ptlrpc_check_set+0x5b/0xe0 [ptlrpc] [12378.640976] [&lt;ffffffffa0c643fb&gt;] ptlrpcd_check+0x4eb/0x5e0 [ptlrpc] [12378.648095] [&lt;ffffffffa0c647ab&gt;] ptlrpcd+0x2bb/0x560 [ptlrpc] [12378.654613] [&lt;ffffffff810b8910&gt;] ? default_wake_function+0x0/0x20 [12378.661550] [&lt;ffffffffa0c644f0&gt;] ? ptlrpcd+0x0/0x560 [ptlrpc] [12378.668067] [&lt;ffffffff810a5b2f&gt;] kthread+0xcf/0xe0 [12378.673515] [&lt;ffffffff810a5a60&gt;] ? kthread+0x0/0xe0 [12378.679058] [&lt;ffffffff81646a98&gt;] ret_from_fork+0x58/0x90 [12378.685086] [&lt;ffffffff810a5a60&gt;] ? kthread+0x0/0xe0 [12378.690626] [12378.692396] Kernel panic - not syncing: LBUG [12378.697164] CPU: 9 PID: 3153 Comm: ptlrpcd_01_02 Tainted: G OE ------------ 3.10.0-327.28.3.el7.x86_64 #1 [12378.709098] Hardware name: Intel Corporation S2600GZ/S2600GZ, BIOS SE5C600.86B.02.01.0002.082220131453 08/22/2013 [12378.720548] ffffffffa0872def 000000003210ab53 ffff881000c1ba80 ffffffff81636453 [12378.728852] ffff881000c1bb00 ffffffff8162fce7 ffffffff00000008 ffff881000c1bb10 [12378.737165] ffff881000c1bab0 000000003210ab53 ffffffffa0c90d30 0000000000000246 [12378.745463] Call Trace: [12378.748204] [&lt;ffffffff81636453&gt;] dump_stack+0x19/0x1b [12378.753939] [&lt;ffffffff8162fce7&gt;] panic+0xd8/0x1e7 [12378.759295] [&lt;ffffffffa0855ddb&gt;] lbug_with_loc+0xab/0xc0 [libcfs] [12378.766235] [&lt;ffffffffa0c3e661&gt;] ptlrpc_register_bulk+0x831/0x9c0 [ptlrpc] [12378.774009] [&lt;ffffffffa08cace2&gt;] ? LNetMDUnlink+0xe2/0x180 [lnet] [12378.780949] [&lt;ffffffffa0c6be76&gt;] ? sptlrpc_import_sec_ref+0x36/0x40 [ptlrpc] [12378.788950] [&lt;ffffffffa0c3f1af&gt;] ptl_send_rpc+0x1ff/0xda0 [ptlrpc] [12378.795981] [&lt;ffffffffa0c39256&gt;] ptlrpc_check_set.part.23+0x1896/0x1dd0 [ptlrpc] [12378.804368] [&lt;ffffffffa0c397eb&gt;] ptlrpc_check_set+0x5b/0xe0 [ptlrpc] [12378.811596] [&lt;ffffffffa0c643fb&gt;] ptlrpcd_check+0x4eb/0x5e0 [ptlrpc] [12378.818725] [&lt;ffffffffa0c647ab&gt;] ptlrpcd+0x2bb/0x560 [ptlrpc] [12378.825237] [&lt;ffffffff810b8910&gt;] ? wake_up_state+0x20/0x20 [12378.831492] [&lt;ffffffffa0c644f0&gt;] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc] [12378.838778] [&lt;ffffffff810a5b2f&gt;] kthread+0xcf/0xe0 [12378.844224] [&lt;ffffffff810a5a60&gt;] ? kthread_create_on_node+0x140/0x140 [12378.851510] [&lt;ffffffff81646a98&gt;] ret_from_fork+0x58/0x90 [12378.857534] [&lt;ffffffff810a5a60&gt;] ? kthread_create_on_node+0x140/0x140 </pre> </div></div> <p>vmcore is available on Spirit for analysis</p> Lustre-master 2.8.56_68_gd4a4c07 build 3430, RHEL7, Spirit performance cluster Revision: d4a4c0795b3befb87d47a5bf441adeba3b1c36f8<br/> LU-8573

IOR: niobuf.c:319:ptlrpc_register_bulk()) ASSERTION( desc->bd_md_count == 0 ) failed

Bug Blocker Resolved Fixed Hongchao Zhang Cliff White Wed, 31 Aug 2016 06:23:40 +0000 Sun, 2 Dec 2018 16:18:47 +0000 Sat, 1 Dec 2018 05:34:34 +0000 Lustre 2.9.0 Lustre 2.10.3 Lustre 2.9.0 0 12 <p>What's the path to vmcore and the debuginfo kernel and modules to make use of the vmcore</p> <p>The vmcore is in /scratch/dumps/spirit-33* on any Spirit node. The related RPMS are at /scratch/rpms or /scratch/hudson.</p> <p>File a DCO ticket requesting access, include your public key (DSA/RSA) The login is passwordless. <br/> The spirit head node (where you login) has all the relevant data under /scratch<br/> /scratch/dumps/&lt;hostname&gt;<br/> /scratch/rpms/ or /scratch/hudson has the RPMS for each install. I am going on sabbatical, Frank Heckes is taking over and should be able to help you. </p> <p>Hongchao Zhang (hongchao.zhang@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/22378" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/22378</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8573" title="IOR: niobuf.c:319:ptlrpc_register_bulk()) ASSERTION( desc-&gt;bd_md_count == 0 ) failed" class="issue-link" data-issue-key="LU-8573"><del>LU-8573</del></a> ptlrpc: always unregister bulk<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 5b1832e89bd0c9432592602149e8496e7c5d4fdf</p> <p>the error also occurs running <a href="https://build.hpdd.intel.com/job/lustre-reviews/41665/" class="external-link" target="_blank" rel="nofollow noopener">https://build.hpdd.intel.com/job/lustre-reviews/41665/</a> (<a href="http://review.whamcloud.com/#/c/22739/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/22739/</a> )<br/> Test was running on centOS-7.2 using patch set 2 of original fix #22738<br/> I'll attach debug logs later (approximately in 1 hour)</p> <p>P.S.: Sorry for using your account on spirit to push the change</p> <p>Hi Frank,<br/> Where do you put the debug logs?<br/> Thanks!</p> <p>Here are the debug logs from the OSTs, MDS and couple of clients</p> <p>Syslog and console messages from the run</p> <p>Hi Cliff,</p> <p>Thanks for the logs!<br/> I'm sorry that I can't find the logs of the client which encountered the "ASSERTION( desc-&gt;bd_md_count == 0 )" in the attached logs,<br/> could you attach the logs (syslog, console and debug log) of the client which encountered the ASSERTION?<br/> Thanks!</p> <p>Here is the system log with the assertion, It looks like the debug logs are no longer on the system</p> <p>Hi Cliff,</p> <p>In the newly attach log of spirit-33, the assertion is occurred at Aug 31, and there is no recent assertion.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Aug 31 01:36:14 spirit-33 kernel: Pid: 3153, comm: ptlrpcd_01_02 Aug 31 01:36:14 spirit-33 kernel: #012Call Trace: Aug 31 01:36:14 spirit-33 kernel: [&lt;ffffffffa08557d3&gt;] libcfs_debug_dumpstack+0x53/0x80 [libcfs] Aug 31 01:36:14 spirit-33 kernel: [&lt;ffffffffa0855d75&gt;] lbug_with_loc+0x45/0xc0 [libcfs] Aug 31 01:36:14 spirit-33 kernel: [&lt;ffffffffa0c3e661&gt;] ptlrpc_register_bulk+0x831/0x9c0 [ptlrpc] Aug 31 01:36:14 spirit-33 kernel: [&lt;ffffffffa08cace2&gt;] ? LNetMDUnlink+0xe2/0x180 [lnet] Aug 31 01:36:14 spirit-33 kernel: [&lt;ffffffffa0c6be76&gt;] ? sptlrpc_import_sec_ref+0x36/0x40 [ptlrpc] Aug 31 01:36:14 spirit-33 kernel: [&lt;ffffffffa0c3f1af&gt;] ptl_send_rpc+0x1ff/0xda0 [ptlrpc] Aug 31 01:36:14 spirit-33 kernel: [&lt;ffffffffa0c39256&gt;] ptlrpc_check_set.part.23+0x1896/0x1dd0 [ptlrpc] Aug 31 01:36:14 spirit-33 kernel: [&lt;ffffffffa0c397eb&gt;] ptlrpc_check_set+0x5b/0xe0 [ptlrpc] Aug 31 01:36:14 spirit-33 kernel: [&lt;ffffffffa0c643fb&gt;] ptlrpcd_check+0x4eb/0x5e0 [ptlrpc] Aug 31 01:36:14 spirit-33 kernel: [&lt;ffffffffa0c647ab&gt;] ptlrpcd+0x2bb/0x560 [ptlrpc] Aug 31 01:36:14 spirit-33 kernel: [&lt;ffffffff810b8910&gt;] ? default_wake_function+0x0/0x20 Aug 31 01:36:14 spirit-33 kernel: [&lt;ffffffffa0c644f0&gt;] ? ptlrpcd+0x0/0x560 [ptlrpc] Aug 31 01:36:14 spirit-33 kernel: [&lt;ffffffff810a5b2f&gt;] kthread+0xcf/0xe0 Aug 31 01:36:14 spirit-33 kernel: [&lt;ffffffff810a5a60&gt;] ? kthread+0x0/0xe0 Aug 31 01:36:14 spirit-33 kernel: [&lt;ffffffff81646a98&gt;] ret_from_fork+0x58/0x90 Aug 31 01:36:14 spirit-33 kernel: [&lt;ffffffff810a5a60&gt;] ? kthread+0x0/0xe0 </pre> </div></div> <p>Yes, August 31 was the last time this test was run.</p> <p>But Frank said it should run around Sept 29.</p> <blockquote> <p>Frank Heckes added a comment - 29/Sep/16 1:48 AM</p> <p>the error also occurs running <a href="https://build.hpdd.intel.com/job/lustre-reviews/41665/" class="external-link" target="_blank" rel="nofollow noopener">https://build.hpdd.intel.com/job/lustre-reviews/41665/</a> (<a href="http://review.whamcloud.com/#/c/22739/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/22739/</a> )<br/> Test was running on centOS-7.2 using patch set 2 of original fix #22738</p></blockquote> <p>Test again with latest master, still have this issue. LDEV-510 has more details</p> <p>Repeated test with latest master, same result. Nodes are currently halted with this issue, can be examined.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Oct 17 17:13:27 spirit-5 kernel: Lustre: DEBUG MARKER: == parallel-scale test iorfpp: iorfpp ================================================================ 17:13:46 (1476724426) Oct 17 17:13:27 spirit-5 sshd[23824]: Received disconnect from 10.10.1.11: 11: disconnected by user Oct 17 17:13:27 spirit-5 sshd[23824]: pam_unix(sshd:session): session closed <span class="code-keyword">for</span> user root Oct 17 17:13:27 spirit-5 systemd-logind: Removed session 94. Oct 17 17:13:31 spirit-5 kernel: Lustre: lustre-OST000c: Client 2af8c060-dda3-f8ed-5532-73d48d5bec75 (at 192.168.1.21@o2ib) reconnecting Oct 17 17:13:31 spirit-5 kernel: Lustre: lustre-OST000c: Connection restored to 78757673-8c34-2ea0-d2f0-d29a18f40898 (at 192.168.1.16@o2ib) Oct 17 17:13:31 spirit-5 kernel: Lustre: Skipped 64 previous similar messages Oct 17 17:13:31 spirit-5 kernel: LustreError: 19542:0:(ldlm_lib.c:3184:target_bulk_io()) @@@ bulk WRITE failed: rc -107 req@ffff88074fd57c50 x1548457026272736/t0(0) o4-&gt;b765b4c7-de22-04f2-d85c-19567447ab65@192.168.1.15@o2ib:-1/-1 lens 608/448 e 0 to 0 dl 1476724421 ref 1 fl Interpret:/0/0 rc 0/0 Oct 17 17:13:31 spirit-5 kernel: LustreError: 19542:0:(ldlm_lib.c:3184:target_bulk_io()) Skipped 12 previous similar messages Oct 17 17:13:31 spirit-5 kernel: Lustre: lustre-OST0000: Bulk IO write error with b765b4c7-de22-04f2-d85c-19567447ab65 (at 192.168.1.15@o2ib), client will retry: rc -107 Oct 17 17:13:31 spirit-5 kernel: Lustre: Skipped 145 previous similar messages Oct 17 17:13:31 spirit-5 kernel: LustreError: 19363:0:(ldlm_lib.c:3234:target_bulk_io()) @@@ Reconnect on bulk WRITE req@ffff88074fd53050 x1548457087089584/t0(0) o4-&gt;2af8c060-dda3-f8ed-5532-73d48d5bec75@192.168.1.21@o2ib:-1/-1 lens 608/448 e 0 to 0 dl 1476724421 ref 1 fl Interpret:/0/0 rc 0/0 Oct 17 17:13:31 spirit-5 kernel: LustreError: 19363:0:(ldlm_lib.c:3234:target_bulk_io()) Skipped 1319 previous similar messages ... Oct 17 17:14:28 spirit-5 kernel: LNetError: 14218:0:(o2iblnd_cb.c:3131:kiblnd_check_txs_locked()) Timed out tx: active_txs, 7 seconds Oct 17 17:14:28 spirit-5 kernel: LNetError: 14218:0:(o2iblnd_cb.c:3131:kiblnd_check_txs_locked()) Skipped 3 previous similar messages Oct 17 17:14:28 spirit-5 kernel: LNetError: 14218:0:(o2iblnd_cb.c:3194:kiblnd_check_conns()) Timed out RDMA with 192.168.1.11@o2ib (1): c: 103, oc: 0, rc: 124 Oct 17 17:14:28 spirit-5 kernel: LNetError: 14218:0:(o2iblnd_cb.c:3194:kiblnd_check_conns()) Skipped 3 previous similar messages Oct 17 17:14:28 spirit-5 kernel: LustreError: 14218:0:(events.c:446:server_bulk_callback()) event type 5, status -103, desc ffff880faf285a00 Oct 17 17:14:28 spirit-5 kernel: LustreError: 14218:0:(events.c:446:server_bulk_callback()) event type 3, status -103, desc ffff880faf285a00 Oct 17 17:14:28 spirit-5 kernel: LustreError: 14218:0:(events.c:446:server_bulk_callback()) event type 5, status -103, desc ffff88084d834e00 Oct 17 17:14:28 spirit-5 kernel: LustreError: 14218:0:(events.c:446:server_bulk_callback()) event type 3, status -103, desc ffff88084d834e00 </pre> </div></div> <p>Client side</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Oct 17 17:14:47 spirit-11 kernel: LNetError: 762:0:(o2iblnd_cb.c:1074:kiblnd_init_rdma()) RDMA is too large <span class="code-keyword">for</span> peer 192.168.1.5@o2ib (131072), src size: 1024000 dst size: 1024000 Oct 17 17:14:47 spirit-11 kernel: LNetError: 762:0:(o2iblnd_cb.c:1698:kiblnd_reply()) Can't setup rdma <span class="code-keyword">for</span> GET from 192.168.1.5@o2ib: -90 Oct 17 17:14:47 spirit-11 kernel: LNetError: 762:0:(o2iblnd_cb.c:1698:kiblnd_reply()) Skipped 10 previous similar messages Oct 17 17:14:47 spirit-11 kernel: LustreError: 762:0:(events.c:199:client_bulk_callback()) event type 1, status -5, desc ffff88080cef7a00 Oct 17 17:14:47 spirit-11 kernel: Lustre: 4800:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1476724429/real 1476724429] req@ffff8807c3730300 x1548457007399104/t0(0) o4-&gt;lustre-OST000c-osc-ffff880036709000@192.168.1.5@o2ib:6/4 lens 608/448 e 6 to 1 dl 1476724517 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1 Oct 17 17:14:47 spirit-11 kernel: LustreError: 4787:0:(events.c:199:client_bulk_callback()) event type 1, status -5, desc ffff88080c746600 Oct 17 17:14:47 spirit-11 kernel: Lustre: lustre-OST0012-osc-ffff880036709000: Connection to lustre-OST0012 (at 192.168.1.5@o2ib) was lost; in progress operations using <span class="code-keyword">this</span> service will wait <span class="code-keyword">for</span> recovery to complete Oct 17 17:14:47 spirit-11 kernel: Lustre: Skipped 9 previous similar messages Oct 17 17:14:47 spirit-11 kernel: LustreError: 4785:0:(events.c:199:client_bulk_callback()) event type 1, status -5, desc ffff880ffa415800 Oct 17 17:14:47 spirit-11 kernel: Lustre: lustre-OST0012-osc-ffff880036709000: Connection restored to 192.168.1.5@o2ib (at 192.168.1.5@o2ib) Oct 17 17:14:47 spirit-11 kernel: Lustre: Skipped 9 previous similar messages Oct 17 17:14:47 spirit-11 kernel: Lustre: 4800:0:(client.c:2111:ptlrpc_expire_one_request()) Skipped 12 previous similar messages Oct 17 17:14:47 spirit-11 kernel: LustreError: 4791:0:(events.c:199:client_bulk_callback()) event type 1, status -5, desc ffff8807c360b800 Oct 17 17:14:47 spirit-11 kernel: LustreError: 4788:0:(events.c:199:client_bulk_callback()) event type 1, status -5, desc ffff8807c360ba00 Oct 17 17:14:47 spirit-11 kernel: LustreError: 4791:0:(events.c:199:client_bulk_callback()) event type 1, status -5, desc ffff8807c3711e00 Oct 17 17:14:47 spirit-11 kernel: LustreError: 4791:0:(events.c:199:client_bulk_callback()) event type 1, status -5, desc ffff88080cdc2a00 Oct 17 17:14:47 spirit-11 kernel: LustreError: 4789:0:(events.c:199:client_bulk_callback()) event type 1, status -5, desc ffff88080c014400 </pre> </div></div> <p>Lustre logs dumped from client (spirit-11) and OSS nodes with debug=-1. Logs dumped by hand after failure. </p> <p>Hi Cliff,</p> <p>Thanks!<br/> Could you please attach the logs of the node where the "ASSERTION( desc-&gt;bd_md_count == 0 )" is encountered? I can't find it in the attached logs (spirit-5,6,11).</p> <p>The logs were taken several hours after the failure, if the ASSERT isn't in those logs, then it must have been over-written. I can re-run the test, and you can login to the nodes if you wish.</p> <p>Hi Cliff, </p> <p>I saw there are several tests of "parallel-scale test iorfpp: iorfpp" after "Oct 18 00:00:00" at spirit-11. Do you still use spirit-11 to test<br/> this issue? there is no "client_bulk_callback" errors in the following tests and no "ASSERTION" was found in the /var/log/messages<br/> (the messages after Sep 26).</p> <p>With the patch <a href="http://review.whamcloud.com/#/c/22739/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/22739/</a> applied, there should be no "ASSERTION( desc-&gt;bd_md_count == 0 )" any longer. Because the patch deregisters the the RPC's MD unconditionally before such ASSERTION() check. That is almost equal to removing the ASSERTION(). But in fact, such ASSERTION() means we assume that the RPC environment should be sane, no need to force cleanup via deregister RPC's MD. So we should find out why the RPC environment is not sane at that time, not only simply 'bypass' such ASSERTION().</p> <p>On the other hand, it is NOT to say the patch <a href="http://review.whamcloud.com/#/c/22739/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/22739/</a> is meaningless, because if we still trigger the ASSERTION() with the patch applied, that means there are other logic paths not handled properly.</p> <p>I'm not familiar with the related code (still studying..), but it looks like ptlrpc_unregister_bulk() is missing in some codepath..</p> <p>Fan Yong (fan.yong@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/23354" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/23354</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8573" title="IOR: niobuf.c:319:ptlrpc_register_bulk()) ASSERTION( desc-&gt;bd_md_count == 0 ) failed" class="issue-link" data-issue-key="LU-8573"><del>LU-8573</del></a> ptlrpc: cleanup bulks for ptlrpc_register_bulk failure<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 683841cc2ea9ee0d16f9fe1c5baf77ed668c15fb</p> <p>IOR continues to fail<br/> Server logs from start of tests</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Oct 26 18:45:05 spirit-aeon-1 kernel: Lustre: zfstest-OST0002: Client de07d034-8d22-f659-1cbd-768d0030c923 (at 192.168.1.26@o2ib) reconnecting Oct 26 18:45:05 spirit-aeon-1 kernel: Lustre: zfstest-OST0002: Connection restored to de07d034-8d22-f659-1cbd-768d0030c923 (at 192.168.1.26@o2ib) Oct 26 18:45:05 spirit-aeon-1 kernel: Lustre: Skipped 9 previous similar messages Oct 26 18:45:05 spirit-aeon-1 kernel: LustreError: 29216:0:(ldlm_lib.c:3183:target_bulk_io()) @@@ bulk WRITE failed: rc -107 req@ffff8808089b4850 x1549270578154064/t0(0) o4-&gt;7522edd5-eea9-2c67-68cd-2b45f3964653@192.168.1.17@o2ib:-1/-1 lens 608/448 e 0 to 0 dl 1477507516 ref 1 fl Interpret:/0/0 rc 0/0 Oct 26 18:45:05 spirit-aeon-1 kernel: Lustre: zfstest-OST0002: Bulk IO write error with 7522edd5-eea9-2c67-68cd-2b45f3964653 (at 192.168.1.17@o2ib), client will retry: rc -107 Oct 26 18:45:05 spirit-aeon-1 kernel: Lustre: zfstest-OST0000: Client ec385605-4e60-f4be-b60f-cd3e019315dd (at 192.168.1.21@o2ib) reconnecting Oct 26 18:45:05 spirit-aeon-1 kernel: Lustre: Skipped 23 previous similar messages Oct 26 18:45:06 spirit-aeon-1 kernel: LustreError: 25234:0:(ldlm_lib.c:3233:target_bulk_io()) @@@ Reconnect on bulk WRITE req@ffff8808089b7850 x1549270581336640/t0(0) o4-&gt;de07d034-8d22-f659-1cbd-768d0030c923@192.168.1.26@o2ib:-1/-1 lens 608/448 e 0 to 0 dl 1477507516 ref 1 fl Interpret:/0/0 rc 0/0 Oct 26 18:45:06 spirit-aeon-1 kernel: LustreError: 25234:0:(ldlm_lib.c:3233:target_bulk_io()) Skipped 2 previous similar messages Oct 26 18:45:06 spirit-aeon-1 kernel: Lustre: zfstest-OST0002: Connection restored to 027d9577-68a8-de83-149f-38f3670e21f2 (at 192.168.1.14@o2ib) Oct 26 18:45:06 spirit-aeon-1 kernel: Lustre: Skipped 27 previous similar messages Oct 26 18:45:06 spirit-aeon-1 kernel: LustreError: 29238:0:(ldlm_lib.c:3233:target_bulk_io()) @@@ Reconnect on bulk WRITE req@ffff880808853850 x1549270577110928/t0(0) o4-&gt;77fffece-ca00-9432-9fd5-857674816ee4@192.168.1.28@o2ib:-1/-1 lens 608/448 e 0 to 0 dl 1477507516 ref 1 fl Interpret:/0/0 rc 0/0 Oct 26 18:45:06 spirit-aeon-1 kernel: Lustre: zfstest-OST0000: Client de07d034-8d22-f659-1cbd-768d0030c923 (at 192.168.1.26@o2ib) reconnecting Oct 26 18:45:06 spirit-aeon-1 kernel: Lustre: Skipped 19 previous similar messages Oct 26 18:45:06 spirit-aeon-1 kernel: LustreError: 29238:0:(ldlm_lib.c:3233:target_bulk_io()) Skipped 24 previous similar messages Oct 26 18:45:07 spirit-aeon-1 kernel: LustreError: 29274:0:(ldlm_lib.c:3183:target_bulk_io()) @@@ bulk WRITE failed: rc -107 req@ffff88007fd76850 x1549270578161152/t0(0) o4-&gt;1b29c57b-535c-46b6-486f-7405da7c1796@192.168.1.18@o2ib:-1/-1 lens 608/448 e 0 to 0 dl 1477507516 ref 1 fl Interpret:/2/0 rc 0/0 Oct 26 18:45:07 spirit-aeon-1 kernel: Lustre: zfstest-OST0000: Bulk IO write error with 1b29c57b-535c-46b6-486f-7405da7c1796 (at 192.168.1.18@o2ib), client will retry: rc -107 Oct 26 18:45:07 spirit-aeon-1 kernel: LustreError: 29261:0:(ldlm_lib.c:3233:target_bulk_io()) @@@ Reconnect on bulk WRITE req@ffff88079810d050 x1549270582366336/t0(0) o4-&gt;f71ccf2b-e961-3680-30e4-be748de9fb17@192.168.1.27@o2ib:-1/-1 lens 608/448 e 0 to 0 dl 1477507516 ref 1 fl Interpret:/2/0 rc 0/0 Oct 26 18:45:07 spirit-aeon-1 kernel: LustreError: 29261:0:(ldlm_lib.c:3233:target_bulk_io()) Skipped 21 previous similar messages Oct 26 18:45:08 spirit-aeon-1 kernel: Lustre: zfstest-OST0000: Connection restored to 51f91233-8032-15d2-cd74-03a850dab34b (at 192.168.1.11@o2ib) Oct 26 18:45:08 spirit-aeon-1 kernel: Lustre: Skipped 38 previous similar messages Oct 26 18:45:08 spirit-aeon-1 kernel: Lustre: zfstest-OST0000: Client 7f61e263-a95b-3193-8481-b48b561e6e29 (at 192.168.1.25@o2ib) reconnecting Oct 26 18:45:08 spirit-aeon-1 kernel: Lustre: Skipped 35 previous similar messages Oct 26 18:45:10 spirit-aeon-1 kernel: LustreError: 29306:0:(ldlm_lib.c:3233:target_bulk_io()) @@@ Reconnect on bulk WRITE req@ffff88072a949850 x1549270584459216/t0(0) o4-&gt;b38521f1-8de6-5f02-305a-3c91f4caeb2d@192.168.1.24@o2ib:-1/-1 lens 608/448 e 0 to 0 dl 1477507518 ref 1 fl Interpret:/2/0 rc 0/0 Oct 26 18:45:10 spirit-aeon-1 kernel: LustreError: 29306:0:(ldlm_lib.c:3233:target_bulk_io()) Skipped 44 previous similar messages Oct 26 18:45:12 spirit-aeon-1 kernel: Lustre: zfstest-OST0000: Connection restored to a8341b8c-a880-ca64-9830-6416dc1b988f (at 192.168.1.15@o2ib) Oct 26 18:45:12 spirit-aeon-1 kernel: Lustre: Skipped 54 previous similar messages Oct 26 18:45:12 spirit-aeon-1 kernel: Lustre: zfstest-OST0000: Client de07d034-8d22-f659-1cbd-768d0030c923 (at 192.168.1.26@o2ib) reconnecting Oct 26 18:45:12 spirit-aeon-1 kernel: Lustre: Skipped 56 previous similar messages Oct 26 18:45:14 spirit-aeon-1 kernel: LustreError: 29376:0:(ldlm_lib.c:3233:target_bulk_io()) @@@ Reconnect on bulk WRITE req@ffff88075d19bc50 x1549270582366368/t0(0) o4-&gt;f71ccf2b-e961-3680-30e4-be748de9fb17@192.168.1.27@o2ib:-1/-1 lens 608/448 e 0 to 0 dl 1477507522 ref 1 fl Interpret:/2/0 rc 0/0 Oct 26 18:45:14 spirit-aeon-1 kernel: LustreError: 29376:0:(ldlm_lib.c:3233:target_bulk_io()) Skipped 68 previous similar messages Oct 26 18:45:56 spirit-aeon-1 kernel: LNetError: 21200:0:(o2iblnd_cb.c:3131:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds Oct 26 18:45:56 spirit-aeon-1 kernel: LNetError: 21200:0:(o2iblnd_cb.c:3194:kiblnd_check_conns()) Timed out RDMA with 192.168.1.11@o2ib (6): c: 127, oc: 0, rc: 94 Oct 26 18:45:56 spirit-aeon-1 kernel: LustreError: 21200:0:(events.c:446:server_bulk_callback()) event type 5, status -103, desc ffff8804c83c8000 Oct 26 18:45:56 spirit-aeon-1 kernel: LustreError: 21200:0:(events.c:446:server_bulk_callback()) event type 3, status -103, desc ffff8804c83c8000 Oct 26 18:45:56 spirit-aeon-1 kernel: LustreError: 21200:0:(events.c:446:server_bulk_callback()) event type 5, status -103, desc ffff8804c83c8000 Oct 26 18:45:56 spirit-aeon-1 kernel: LustreError: 21200:0:(events.c:446:server_bulk_callback()) event type 3, status -103, desc ffff8804c83c8000 Oct 26 18:45:56 spirit-aeon-1 kernel: LustreError: 21200:0:(events.c:446:server_bulk_callback()) event type 5, status -103, desc ffff8804c83c8000 Oct 26 18:45:56 spirit-aeon-1 kernel: LustreError: 21200:0:(events.c:446:server_bulk_callback()) event type 3, status -103, desc ffff8804c83c8000 Oct 26 18:45:56 spirit-aeon-1 kernel: LustreError: 21200:0:(events.c:446:server_bulk_callback()) event type 5, status -103, desc ffff8804c83c8000 Oct 26 18:45:56 spirit-aeon-1 kernel: LustreError: 21200:0:(events.c:446:server_bulk_callback()) event type 3, status -103, desc ffff8804c83c8000 Oct 26 18:45:56 spirit-aeon-1 kernel: LustreError: 21200:0:(events.c:446:server_bulk_callback()) event type 5, status -103, desc ffff88054023b400 Oct 26 18:45:56 spirit-aeon-1 kernel: Lustre: zfstest-OST0002: Bulk IO write error with 51f91233-8032-15d2-cd74-03a850dab34b (at 192.168.1.11@o2ib), client will retry: rc -110 Oct 26 18:45:56 spirit-aeon-1 kernel: LustreError: 212 </pre> </div></div> <p>Client-side messages</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Oct 26 18:45:02 spirit-11 kernel: Lustre: DEBUG MARKER: == parallel-scale test iorfpp: iorfpp ================================================================ 18:45:02 (1477507502) Oct 26 18:45:02 spirit-11 sshd[21261]: Received disconnect from 10.10.1.11: 11: disconnected by user Oct 26 18:45:02 spirit-11 sshd[21261]: pam_unix(sshd:session): session closed <span class="code-keyword">for</span> user root Oct 26 18:45:02 spirit-11 systemd-logind: Removed session 42. Oct 26 18:45:05 spirit-11 kernel: LNetError: 3587:0:(o2iblnd_cb.c:1074:kiblnd_init_rdma()) RDMA is too large <span class="code-keyword">for</span> peer 192.168.2.2@o2ib (131072), src size: 1048576 dst size: 1048576 Oct 26 18:45:05 spirit-11 kernel: LNetError: 3586:0:(o2iblnd_cb.c:1698:kiblnd_reply()) Can't setup rdma <span class="code-keyword">for</span> GET from 192.168.2.2@o2ib: -90 Oct 26 18:45:05 spirit-11 kernel: LustreError: 3586:0:(events.c:199:client_bulk_callback()) event type 1, status -5, desc ffff88100d076800 Oct 26 18:45:05 spirit-11 kernel: LustreError: 3586:0:(events.c:199:client_bulk_callback()) event type 1, status -5, desc ffff88100d074e00 Oct 26 18:45:05 spirit-11 kernel: LustreError: 3585:0:(events.c:199:client_bulk_callback()) event type 1, status -5, desc ffff88100d074e00 Oct 26 18:45:05 spirit-11 kernel: Lustre: 3616:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1477507505/real 1477507505] req@ffff88077d8d6600 x1549270577199552/t0(0) o4-&gt;zfstest-OST0001-osc-ffff880ff9be7000@192.168.2.2@o2ib:6/4 lens 608/448 e 0 to 1 dl 1477507517 ref 2 fl Rpc:eX/0/ffffffff rc 0/-1 Oct 26 18:45:05 spirit-11 kernel: Lustre: zfstest-OST0001-osc-ffff880ff9be7000: Connection to zfstest-OST0001 (at 192.168.2.2@o2ib) was lost; in progress operations using <span class="code-keyword">this</span> service will wait <span class="code-keyword">for</span> recovery to complete Oct 26 18:45:05 spirit-11 kernel: Lustre: zfstest-OST0001-osc-ffff880ff9be7000: Connection restored to 192.168.2.2@o2ib (at 192.168.2.2@o2ib) Oct 26 18:45:05 spirit-11 kernel: LustreError: 3582:0:(events.c:199:client_bulk_callback()) event type 1, status -5, desc ffff88100d074400 Oct 26 18:45:05 spirit-11 kernel: LustreError: 3582:0:(events.c:199:client_bulk_callback()) event type 1, status -5, desc ffff88100d074400 Oct 26 18:45:05 spirit-11 kernel: LustreError: 3581:0:(events.c:199:client_bulk_callback()) event type 1, status -5, desc ffff88100d074400 Oct 26 18:45:05 spirit-11 kernel: LustreError: 3584:0:(events.c:199:client_bulk_callback()) event type 1, status -5, desc ffff88100d074400 Oct 26 18:45:05 spirit-11 kernel: LNetError: 3587:0:(o2iblnd_cb.c:1074:kiblnd_init_rdma()) Skipped 7 previous similar messages Oct 26 18:45:05 spirit-11 kernel: LustreError: 3587:0:(events.c:199:client_bulk_callback()) event type 1, status -5, desc ffff88100d076800 Oct 26 18:45:05 spirit-11 kernel: LustreError: 3588:0:(events.c:199:client_bulk_callback()) event type 1, status -5, desc ffff88003666fc00 Oct 26 18:45:05 spirit-11 kernel: LustreError: 3586:0:(events.c:199:client_bulk_callback()) event type 1, status -5, desc ffff88003666fc00 Oct 26 18:45:05 spirit-11 kernel: LustreError: 3585:0:(events.c:199:client_bulk_callback()) event type 1, status -5, desc ffff88003666fc00 </pre> </div></div> <p>Lustre logs from both OSS nodes. backfstype was ZFS for this test, error can be reproduced on either ZFS or ldiskfs</p> <p>MDS and sample client lustre logs. </p> <p>I think the IO errors aside, it's still important to fix the assertion since twe don't want to have the crashes.<br/> Now if io erors happen to be due to network errors later - then we can dig into this separately, but at least we won't be crashing anything.</p> <p>One note: the IOR test makes no progress due to these errors, it may not be hitting an ASSERT, but the code is completely un-useable. </p> <p>Repeated the fault with ldiskfs, set debug=-1 and dumped lustre logs after the hang occurred. <br/> MDS = spirit-4 OSS= spirit-5/6, all others are clients. </p> <p>Received patch from Alex, "Doug thinks that can be introduced by <a href="https://jira.whamcloud.com/browse/LU-7650" title="ko2iblnd map_on_demand can&#39;t negotitate when page sizes are different between nodes." class="issue-link" data-issue-key="LU-7650"><del>LU-7650</del></a>.. I just pushed a patch to revert that: <a href="http://review.whamcloud.com/23439" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/23439</a>"<br/> I was able to successfully complete IOR runs on both sets of servers, will retry ZFS today.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/23439/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/23439/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8573" title="IOR: niobuf.c:319:ptlrpc_register_bulk()) ASSERTION( desc-&gt;bd_md_count == 0 ) failed" class="issue-link" data-issue-key="LU-8573"><del>LU-8573</del></a> lnet: Revert <a href="https://jira.whamcloud.com/browse/LU-7650" title="ko2iblnd map_on_demand can&#39;t negotitate when page sizes are different between nodes." class="issue-link" data-issue-key="LU-7650"><del>LU-7650</del></a> patches<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 59fbfa5f3fa925f676636f1b78c986d05cd295bc</p> <p>Landed for 2.9</p> <p>Hongchao Zhang (hongchao@whamcloud.com) uploaded a new patch: <a href="http://review.whamcloud.com/33167" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/33167</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8573" title="IOR: niobuf.c:319:ptlrpc_register_bulk()) ASSERTION( desc-&gt;bd_md_count == 0 ) failed" class="issue-link" data-issue-key="LU-8573"><del>LU-8573</del></a> ptlrpc: race with reply_in_callback<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: ac6b543a08033ddb5879706c0b3c941d3cdfbef0</p> <p>Closing this ticket again, and using <a href="https://jira.whamcloud.com/browse/LU-11647" title="niobuf.c:330:ptlrpc_register_bulk()) ASSERTION( desc-&gt;bd_md_count == 0 ) failed:" class="issue-link" data-issue-key="LU-11647"><del>LU-11647</del></a> for the new patch. This avoids confusing the fix version for the patch.</p> Duplicate LU-11647 LU-11692 Related LU-7650 Development Rank 1|hzympj: Rank (Obsolete) 9223372036854775807 Severity