Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 [LU-9983] LBUG llog_osd.c:327:llog_osd_declare_write_rec() - all DNE MDS https://jira.whamcloud.com/browse/LU-9983 Lustre <p>soak is started, jobs begin to run. <br/> soak-8 (MDT0000) reports and error:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Sep 13 15:53:37 soak-8 kernel: LustreError: 4873:0:(out_handler.c:597:out_write()) soaked-MDT0000: empty or wrong size 0 pos: rc = -61 Sep 13 15:53:37 soak-8 kernel: LustreError: 4990:0:(out_handler.c:1000:out_handle()) soaked-MDT0000: invalid update buffer magic 0 expect bdde0002: rc = -71 Sep 13 15:53:37 soak-8 kernel: LustreError: 4873:0:(out_handler.c:597:out_write()) Skipped 2 previous similar messages Sep 13 15:53:44 soak-8 kernel: LustreError: 5117:0:(mdt_lvb.c:163:mdt_lvbo_fill()) soaked-MDT0000: expected 968 actual 344. </pre> </div></div> <p>All other DNE MDTS (01-03) promptly LBUG:<br/> soak-9</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Sep 13 15:53:37 soak-9 kernel: LustreError: 11-0: soaked-MDT0000-osp-MDT0001: operation out_update to node 192.168.1.108@o2ib failed: rc = -71 Sep 13 15:53:37 soak-9 kernel: LustreError: 4734:0:(layout.c:2082:__req_capsule_get()) @@@ Wrong buffer <span class="code-keyword">for</span> field `object_update_reply<span class="code-quote">' (1 of 1) in format `OUT_UPDATE'</span>: 0 vs. 4096 (server)#012 req@ffff880808e79500 x1578438980902080/t0(0) o1000-&gt;soaked-MDT0002-osp-MDT0001@192.168.1.110@o2ib:24/4 lens 392/192 e 0 to 0 dl 1505318024 ref 2 fl Interpret:RM/0/0 rc -71/-71 Sep 13 15:53:37 soak-9 kernel: LustreError: Skipped 1 previous similar message Sep 13 15:53:47 soak-9 kernel: LustreError: 4719:0:(llog_osd.c:327:llog_osd_declare_write_rec()) ASSERTION( rec ) failed: Sep 13 15:53:47 soak-9 kernel: LustreError: 4719:0:(llog_osd.c:327:llog_osd_declare_write_rec()) LBUG Sep 13 15:53:47 soak-9 kernel: Pid: 4719, comm: dist_txn-1 Sep 13 15:53:47 soak-9 kernel: #012Call Trace: Sep 13 15:53:47 soak-9 kernel: [&lt;ffffffffc0e667ae&gt;] libcfs_call_trace+0x4e/0x60 [libcfs] Sep 13 15:53:47 soak-9 kernel: [&lt;ffffffffc0e6683c&gt;] lbug_with_loc+0x4c/0xb0 [libcfs] Sep 13 15:53:47 soak-9 kernel: [&lt;ffffffffc0f658f5&gt;] llog_osd_declare_write_rec+0x3c5/0x3d0 [obdclass] Sep 13 15:53:47 soak-9 kernel: [&lt;ffffffffc0f5a0e4&gt;] llog_declare_write_rec+0x84/0x200 [obdclass] Sep 13 15:53:47 soak-9 kernel: [&lt;ffffffffc0f5a862&gt;] llog_cancel_rec+0xe2/0x870 [obdclass] Sep 13 15:53:47 soak-9 kernel: [&lt;ffffffffc0f60f6a&gt;] llog_cat_cancel_records+0x13a/0x2e0 [obdclass] Sep 13 15:53:47 soak-9 kernel: [&lt;ffffffffc0e71ba7&gt;] ? libcfs_debug_msg+0x57/0x80 [libcfs] Sep 13 15:53:47 soak-9 kernel: [&lt;ffffffffc11f22d9&gt;] distribute_txn_commit_thread+0x599/0xca0 [ptlrpc] Sep 13 15:53:47 soak-9 kernel: [&lt;ffffffff810c12c8&gt;] ? check_preempt_curr+0x78/0xa0 Sep 13 15:53:47 soak-9 kernel: [&lt;ffffffff810c4810&gt;] ? default_wake_function+0x0/0x20 Sep 13 15:53:47 soak-9 kernel: [&lt;ffffffffc11f1d40&gt;] ? distribute_txn_commit_thread+0x0/0xca0 [ptlrpc] Sep 13 15:53:47 soak-9 kernel: [&lt;ffffffff810b098f&gt;] kthread+0xcf/0xe0 Sep 13 15:53:47 soak-9 kernel: [&lt;ffffffff810b08c0&gt;] ? kthread+0x0/0xe0 Sep 13 15:53:47 soak-9 kernel: [&lt;ffffffff816b4f18&gt;] ret_from_fork+0x58/0x90 Sep 13 15:53:47 soak-9 kernel: [&lt;ffffffff810b08c0&gt;] ? kthread+0x0/0xe0 Sep 13 15:53:47 soak-9 kernel: Sep 13 15:53:47 soak-9 kernel: Kernel panic - not syncing: LBUG Sep 13 15:53:50 soak-9 kernel: CPU: 25 PID: 4719 Comm: dist_txn-1 Tainted: P OE ------------ 3.10.0-693.1.1.el7_lustre.x86_64 #1 Sep 13 15:53:50 soak-9 kernel: Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 Sep 13 15:53:50 soak-9 kernel: ffff880419612f00 000000001ae378cc ffff880418eafb90 ffffffff816a3d6d Sep 13 15:53:50 soak-9 kernel: ffff880418eafc10 ffffffff8169dc54 ffffffff00000008 ffff880418eafc20 Sep 13 15:53:50 soak-9 kernel: ffff880418eafbc0 000000001ae378cc 000000001ae378cc ffff88082da4f8b8 Sep 13 15:53:50 soak-9 kernel: Call Trace: Sep 13 15:53:50 soak-9 kernel: [&lt;ffffffff816a3d6d&gt;] dump_stack+0x19/0x1b Sep 13 15:53:50 soak-9 kernel: [&lt;ffffffff8169dc54&gt;] panic+0xe8/0x20d Sep 13 15:53:50 soak-9 kernel: [&lt;ffffffffc0e66854&gt;] lbug_with_loc+0x64/0xb0 [libcfs] Sep 13 15:53:50 soak-9 kernel: [&lt;ffffffffc0f658f5&gt;] llog_osd_declare_write_rec+0x3c5/0x3d0 [obdclass] Sep 13 15:53:50 soak-9 kernel: [&lt;ffffffffc0f5a0e4&gt;] llog_declare_write_rec+0x84/0x200 [obdclass] Sep 13 15:53:50 soak-9 kernel: [&lt;ffffffffc0f5a862&gt;] llog_cancel_rec+0xe2/0x870 [obdclass] Sep 13 15:53:50 soak-9 kernel: [&lt;ffffffffc0f60f6a&gt;] llog_cat_cancel_records+0x13a/0x2e0 [obdclass] Sep 13 15:53:50 soak-9 kernel: [&lt;ffffffffc0e71ba7&gt;] ? libcfs_debug_msg+0x57/0x80 [libcfs] Sep 13 15:53:50 soak-9 kernel: [&lt;ffffffffc11f22d9&gt;] distribute_txn_commit_thread+0x599/0xca0 [ptlrpc] Sep 13 15:53:50 soak-9 kernel: [&lt;ffffffff810c12c8&gt;] ? check_preempt_curr+0x78/0xa0 Sep 13 15:53:50 soak-9 kernel: [&lt;ffffffff810c4810&gt;] ? wake_up_state+0x20/0x20 Sep 13 15:53:50 soak-9 kernel: [&lt;ffffffffc11f1d40&gt;] ? sub_trans_commit_cb+0x20/0x20 [ptlrpc] Sep 13 15:53:50 soak-9 kernel: [&lt;ffffffff810b098f&gt;] kthread+0xcf/0xe0 Sep 13 15:53:50 soak-9 kernel: [&lt;ffffffff810b08c0&gt;] ? insert_kthread_work+0x40/0x40 Sep 13 15:53:50 soak-9 kernel: [&lt;ffffffff816b4f18&gt;] ret_from_fork+0x58/0x90 Sep 13 15:53:50 soak-9 kernel: [&lt;ffffffff810b08c0&gt;] ? insert_kthread_work+0x40/0x40 Sep 13 15:53:50 soak-9 kernel: Kernel Offset: disabled Sep 13 15:53:50 soak-9 kernel: ------------[ cut here ]------------ </pre> </div></div> <p>soak-10</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Sep 13 15:53:37 soak-10 kernel: LustreError: 11-0: soaked-MDT0000-osp-MDT0002: operation out_update to node 192.168.1.108@o2ib failed: rc = -71 Sep 13 15:53:37 soak-10 kernel: LustreError: 14955:0:(layout.c:2082:__req_capsule_get()) @@@ Wrong buffer <span class="code-keyword">for</span> field `object_update_reply<span class="code-quote">' (1 of 1) in format `OUT_UPDATE'</span>: 0 vs. 4096 (server)#012 req@ffff880818bb5400 x1578438986144240/t0(0) o1000-&gt;soaked-MDT0000-osp-MDT0002@192.168.1.108@o2ib:24/4 lens 392/192 e 0 to 0 dl 1505318024 ref 2 fl Interpret:RM/0/0 rc -71/-71 Sep 13 15:53:37 soak-10 kernel: LustreError: 14939:0:(out_handler.c:597:out_write()) soaked-MDT0002: empty or wrong size 0 pos: rc = -61 Sep 13 15:53:37 soak-10 kernel: LustreError: 14939:0:(out_handler.c:1000:out_handle()) soaked-MDT0002: invalid update buffer magic 0 expect bdde0002: rc = -71 Sep 13 15:53:44 soak-10 kernel: LustreError: 14961:0:(llog_osd.c:327:llog_osd_declare_write_rec()) ASSERTION( rec ) failed: Sep 13 15:53:44 soak-10 kernel: LustreError: 14961:0:(llog_osd.c:327:llog_osd_declare_write_rec()) LBUG Sep 13 15:53:44 soak-10 kernel: Pid: 14961, comm: dist_txn-2 Sep 13 15:53:44 soak-10 kernel: #012Call Trace: Sep 13 15:53:44 soak-10 kernel: [&lt;ffffffffc0e087ae&gt;] libcfs_call_trace+0x4e/0x60 [libcfs] Sep 13 15:53:44 soak-10 kernel: [&lt;ffffffffc0e0883c&gt;] lbug_with_loc+0x4c/0xb0 [libcfs] Sep 13 15:53:44 soak-10 kernel: [&lt;ffffffffc0f208f5&gt;] llog_osd_declare_write_rec+0x3c5/0x3d0 [obdclass] Sep 13 15:53:44 soak-10 kernel: [&lt;ffffffffc0f150e4&gt;] llog_declare_write_rec+0x84/0x200 [obdclass] Sep 13 15:53:44 soak-10 kernel: [&lt;ffffffffc0f15862&gt;] llog_cancel_rec+0xe2/0x870 [obdclass] Sep 13 15:53:44 soak-10 kernel: [&lt;ffffffffc0f1bf6a&gt;] llog_cat_cancel_records+0x13a/0x2e0 [obdclass] Sep 13 15:53:44 soak-10 kernel: [&lt;ffffffffc0e13ba7&gt;] ? libcfs_debug_msg+0x57/0x80 [libcfs] Sep 13 15:53:44 soak-10 kernel: [&lt;ffffffffc11fc2d9&gt;] distribute_txn_commit_thread+0x599/0xca0 [ptlrpc] Sep 13 15:53:44 soak-10 kernel: [&lt;ffffffff810c4810&gt;] ? default_wake_function+0x0/0x20 Sep 13 15:53:44 soak-10 kernel: [&lt;ffffffffc11fbd40&gt;] ? distribute_txn_commit_thread+0x0/0xca0 [ptlrpc] Sep 13 15:53:44 soak-10 kernel: [&lt;ffffffff810b098f&gt;] kthread+0xcf/0xe0 Sep 13 15:53:44 soak-10 kernel: [&lt;ffffffff810b08c0&gt;] ? kthread+0x0/0xe0 Sep 13 15:53:44 soak-10 kernel: [&lt;ffffffff816b4f18&gt;] ret_from_fork+0x58/0x90 Sep 13 15:53:44 soak-10 kernel: [&lt;ffffffff810b08c0&gt;] ? kthread+0x0/0xe0 Sep 13 15:53:44 soak-10 kernel: Sep 13 15:53:44 soak-10 kernel: Kernel panic - not syncing: LBUG Sep 13 15:53:45 soak-10 kernel: CPU: 10 PID: 14961 Comm: dist_txn-2 Tainted: P OE ------------ 3.10.0-693.1.1.el7_lustre.x86_64 #1 Sep 13 15:53:45 soak-10 kernel: Hardware name: Intel Corporation S2600GZ ........../S2600GZ, BIOS SE5C600.86B.01.08.0003.022620131521 02/26/2013 Sep 13 15:53:45 soak-10 kernel: ffff8803f8e7bf00 000000009b9e8911 ffff8803f8ea7b90 ffffffff816a3d6d Sep 13 15:53:45 soak-10 kernel: ffff8803f8ea7c10 ffffffff8169dc54 ffffffff00000008 ffff8803f8ea7c20 Sep 13 15:53:45 soak-10 kernel: ffff8803f8ea7bc0 000000009b9e8911 000000009b9e8911 ffff88082d88f8b8 Sep 13 15:53:45 soak-10 kernel: Call Trace: Sep 13 15:53:45 soak-10 kernel: [&lt;ffffffff816a3d6d&gt;] dump_stack+0x19/0x1b Sep 13 15:53:45 soak-10 kernel: [&lt;ffffffff8169dc54&gt;] panic+0xe8/0x20d Sep 13 15:53:45 soak-10 kernel: [&lt;ffffffffc0e08854&gt;] lbug_with_loc+0x64/0xb0 [libcfs] Sep 13 15:53:45 soak-10 kernel: [&lt;ffffffffc0f208f5&gt;] llog_osd_declare_write_rec+0x3c5/0x3d0 [obdclass] Sep 13 15:53:45 soak-10 kernel: [&lt;ffffffffc0f150e4&gt;] llog_declare_write_rec+0x84/0x200 [obdclass] Sep 13 15:53:45 soak-10 kernel: [&lt;ffffffffc0f15862&gt;] llog_cancel_rec+0xe2/0x870 [obdclass] Sep 13 15:53:45 soak-10 kernel: [&lt;ffffffffc0f1bf6a&gt;] llog_cat_cancel_records+0x13a/0x2e0 [obdclass] Sep 13 15:53:45 soak-10 kernel: [&lt;ffffffffc0e13ba7&gt;] ? libcfs_debug_msg+0x57/0x80 [libcfs] Sep 13 15:53:45 soak-10 kernel: [&lt;ffffffffc11fc2d9&gt;] distribute_txn_commit_thread+0x599/0xca0 [ptlrpc] Sep 13 15:53:45 soak-10 kernel: [&lt;ffffffff810c4810&gt;] ? wake_up_state+0x20/0x20 Sep 13 15:53:45 soak-10 kernel: [&lt;ffffffffc11fbd40&gt;] ? sub_trans_commit_cb+0x20/0x20 [ptlrpc] Sep 13 15:53:45 soak-10 kernel: [&lt;ffffffff810b098f&gt;] kthread+0xcf/0xe0 Sep 13 15:53:45 soak-10 kernel: [&lt;ffffffff810b08c0&gt;] ? insert_kthread_work+0x40/0x40 Sep 13 15:53:45 soak-10 kernel: [&lt;ffffffff816b4f18&gt;] ret_from_fork+0x58/0x90 Sep 13 15:53:45 soak-10 kernel: [&lt;ffffffff810b08c0&gt;] ? insert_kthread_work+0x40/0x40 Sep 13 15:53:45 soak-10 kernel: Kernel Offset: disabled Sep 13 15:53:45 soak-10 kernel: ------------[ cut here ]------------ </pre> </div></div> <p>soak-11</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Sep 13 15:53:43 soak-11 kernel: LustreError: 4755:0:(llog_osd.c:327:llog_osd_declare_write_rec()) ASSERTION( rec ) failed: Sep 13 15:53:43 soak-11 kernel: LustreError: 4755:0:(llog_osd.c:327:llog_osd_declare_write_rec()) LBUG Sep 13 15:53:43 soak-11 kernel: Pid: 4755, comm: dist_txn-3 Sep 13 15:53:43 soak-11 kernel: #012Call Trace: Sep 13 15:53:43 soak-11 kernel: [&lt;ffffffffc0d6f7ae&gt;] libcfs_call_trace+0x4e/0x60 [libcfs] Sep 13 15:53:43 soak-11 kernel: [&lt;ffffffffc0d6f83c&gt;] lbug_with_loc+0x4c/0xb0 [libcfs] Sep 13 15:53:44 soak-11 kernel: [&lt;ffffffffc0e878f5&gt;] llog_osd_declare_write_rec+0x3c5/0x3d0 [obdclass] Sep 13 15:53:44 soak-11 kernel: [&lt;ffffffffc0e7c0e4&gt;] llog_declare_write_rec+0x84/0x200 [obdclass] Sep 13 15:53:44 soak-11 kernel: [&lt;ffffffffc0e7c862&gt;] llog_cancel_rec+0xe2/0x870 [obdclass] Sep 13 15:53:44 soak-11 kernel: [&lt;ffffffffc0e82f6a&gt;] llog_cat_cancel_records+0x13a/0x2e0 [obdclass] Sep 13 15:53:44 soak-11 kernel: [&lt;ffffffffc0d7aba7&gt;] ? libcfs_debug_msg+0x57/0x80 [libcfs] Sep 13 15:53:44 soak-11 kernel: [&lt;ffffffffc11632d9&gt;] distribute_txn_commit_thread+0x599/0xca0 [ptlrpc] Sep 13 15:53:44 soak-11 kernel: [&lt;ffffffff810c4810&gt;] ? default_wake_function+0x0/0x20 Sep 13 15:53:44 soak-11 kernel: [&lt;ffffffffc1162d40&gt;] ? distribute_txn_commit_thread+0x0/0xca0 [ptlrpc] Sep 13 15:53:44 soak-11 kernel: [&lt;ffffffff810b098f&gt;] kthread+0xcf/0xe0 Sep 13 15:53:44 soak-11 kernel: [&lt;ffffffff810b08c0&gt;] ? kthread+0x0/0xe0 Sep 13 15:53:44 soak-11 kernel: [&lt;ffffffff816b4f18&gt;] ret_from_fork+0x58/0x90 Sep 13 15:53:44 soak-11 kernel: [&lt;ffffffff810b08c0&gt;] ? kthread+0x0/0xe0 Sep 13 15:53:44 soak-11 kernel: Sep 13 15:53:44 soak-11 kernel: Kernel panic - not syncing: LBUG </pre> </div></div> version=2.10.52_83_g1fc4ed3 lustre-master build 3637 - RHEL 7.3 distro with 7.4 kernels LU-9983 LBUG llog_osd.c:327:llog_osd_declare_write_rec() - all DNE MDS Bug Critical Resolved Fixed WC Triage Cliff White Wed, 13 Sep 2017 16:07:50 +0000 Wed, 2 Jan 2019 20:44:07 +0000 Sun, 17 Dec 2017 15:56:19 +0000 Lustre 2.10.1 Lustre 2.11.0 Lustre 2.11.0 Lustre 2.10.6 0 15 <p>Hit this again on two MDS over the weekend</p> <p>Hit this again after the RHEL 7.4 update - soak really is not useable with this bug.</p> <p>John L. Hammond (john.hammond@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/29177" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29177</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9983" title="LBUG llog_osd.c:327:llog_osd_declare_write_rec() - all DNE MDS" class="issue-link" data-issue-key="LU-9983"><del>LU-9983</del></a> llog: revert <a href="https://jira.whamcloud.com/browse/LU-9848" title="LBUG: ASSERTION( len &gt;= (24) &amp;&amp; (len &amp; 0x7) == 0 ) failed" class="issue-link" data-issue-key="LU-9848"><del>LU-9848</del></a><br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: 1c11b55cc8d600db09bd7502b01bf8515821a9a3</p> <p>Loaded that build, immediately hit LBUG.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Sep 23 00:09:42 soak-9 kernel: LustreError: 2294:0:(layout.c:2085:__req_capsule_get()) @@@ Wrong buffer <span class="code-keyword">for</span> field `object_update_reply<span class="code-quote">' (1 of 1) in format `OUT_UPDATE'</span>: 0 vs. 4096 (server)#012 req@ffff8808103d1800 x1579283258584816/t0(0) o1000-&gt;soaked-MDT0000-osp-MDT0001@192.168.1.108@o2ib:24/4 lens 392/192 e 0 to 0 dl 1506125390 ref 2 fl Interpret:RM/0/0 rc -71/-71 Sep 23 00:09:53 soak-9 kernel: LustreError: 2351:0:(llog_osd.c:327:llog_osd_declare_write_rec()) ASSERTION( rec ) failed: Sep 23 00:09:53 soak-9 kernel: LustreError: 2351:0:(llog_osd.c:327:llog_osd_declare_write_rec()) LBUG Sep 23 00:09:53 soak-9 kernel: Pid: 2351, comm: dist_txn-1 Sep 23 00:09:53 soak-9 kernel: #012Call Trace: Sep 23 00:09:53 soak-9 kernel: [&lt;ffffffffc08907ae&gt;] libcfs_call_trace+0x4e/0x60 [libcfs] Sep 23 00:09:53 soak-9 kernel: [&lt;ffffffffc089083c&gt;] lbug_with_loc+0x4c/0xb0 [libcfs] Sep 23 00:09:53 soak-9 kernel: [&lt;ffffffffc0a16b75&gt;] llog_osd_declare_write_rec+0x3c5/0x3d0 [obdclass] Sep 23 00:09:53 soak-9 kernel: [&lt;ffffffffc0a0b0e4&gt;] llog_declare_write_rec+0x84/0x200 [obdclass] Sep 23 00:09:53 soak-9 kernel: [&lt;ffffffffc0a0b862&gt;] llog_cancel_rec+0xe2/0x870 [obdclass] Sep 23 00:09:53 soak-9 kernel: [&lt;ffffffffc0a1233a&gt;] llog_cat_cancel_records+0x13a/0x2e0 [obdclass] Sep 23 00:09:53 soak-9 kernel: [&lt;ffffffffc089bbc7&gt;] ? libcfs_debug_msg+0x57/0x80 [libcfs] Sep 23 00:09:54 soak-9 kernel: [&lt;ffffffffc0cf2139&gt;] distribute_txn_commit_thread+0x599/0xca0 [ptlrpc] Sep 23 00:09:54 soak-9 kernel: [&lt;ffffffff810c4810&gt;] ? default_wake_function+0x0/0x20 Sep 23 00:09:54 soak-9 kernel: [&lt;ffffffffc0cf1ba0&gt;] ? distribute_txn_commit_thread+0x0/0xca0 [ptlrpc] Sep 23 00:09:54 soak-9 kernel: [&lt;ffffffff810b098f&gt;] kthread+0xcf/0xe0 Sep 23 00:09:54 soak-9 kernel: [&lt;ffffffff810b08c0&gt;] ? kthread+0x0/0xe0 Sep 23 00:09:54 soak-9 kernel: [&lt;ffffffff816b4f58&gt;] ret_from_fork+0x58/0x90 Sep 23 00:09:54 soak-9 kernel: [&lt;ffffffff810b08c0&gt;] ? kthread+0x0/0xe0 Sep 23 00:09:54 soak-9 kernel: </pre> </div></div> <p>Will restart with panic_on_lbug=0</p> <p>This actually include 2 bugs.</p> <p>1. Bulk transfer between MDTs somehow is screwed up, so those update request can not pass the audit. </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Sep 13 15:53:37 soak-8 kernel: LustreError: 4873:0:(out_handler.c:597:out_write()) soaked-MDT0000: empty or wrong size 0 pos: rc = -61 Sep 13 15:53:37 soak-8 kernel: LustreError: 4990:0:(out_handler.c:1000:out_handle()) soaked-MDT0000: invalid update buffer magic 0 expect bdde0002: rc = -71 </pre> </div></div> <p>2. when the master MDT get failure of remote request caused by 1, there seems a race between invalidate the request and cancel log. </p> <p>2 seems easy to fix, but 1 is the major issue here. Did we land sth recently which might break the bulk transfer between MDTs?</p> <p>wangdi (di.wang@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/29178" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29178</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9983" title="LBUG llog_osd.c:327:llog_osd_declare_write_rec() - all DNE MDS" class="issue-link" data-issue-key="LU-9983"><del>LU-9983</del></a> llog: skip non-exist log cancellation<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: 7e54ccd5d638cddd77d1aee76aeeae0e26361974</p> <p>wangdi (di.wang@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/29179" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29179</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9983" title="LBUG llog_osd.c:327:llog_osd_declare_write_rec() - all DNE MDS" class="issue-link" data-issue-key="LU-9983"><del>LU-9983</del></a> kernel: revert kernel upgrade<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: 144226402da2a74b1ab5391393ba4ff64962d11a</p> <p> <a href="https://review.whamcloud.com/29178" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29178</a> is the fix for issue 2.</p> <p>Unfortunately, I am still not sure what cause the issue1. And it is also very interesting, according to the debug log, the update requests still valid when packing the RPC, but somehow on the server side, those request magic number is reset to 0.</p> <p>On the source MDT, see those magic</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000004:00080000:15.0:1506127961.257426:0:3000:0:(osp_trans.c:1330:osp_get_next_request()) ou ffff88042ece15a0 version 3 rpc_version 3 00000004:00000001:15.0:1506127961.257444:0:3000:0:(osp_trans.c:1119:osp_send_update_req()) Process entered 00000004:00000001:15.0:1506127961.257446:0:3000:0:(osp_trans.c:361:osp_prep_update_req()) Process entered 00000004:00000040:15.0:1506127961.257448:0:3000:0:(osp_trans.c:272:object_update_request_dump()) i = 0 fid = [0x2c0003ab2:0x2:0x0] op = create params = 1 batchid = 0 size = 256 repsize 0 00000004:00000040:15.0:1506127961.257453:0:3000:0:(osp_trans.c:272:object_update_request_dump()) i = 1 fid = [0x2c0003ab2:0x2:0x0] op = ref_add params = 0 batchid = 0 size = 40 repsize 0 00000004:00000040:15.0:1506127961.257456:0:3000:0:(osp_trans.c:272:object_update_request_dump()) i = 2 fid = [0x2c0003ab2:0x2:0x0] op = insert params = 3 batchid = 0 size = 96 repsize 0 00000004:00000040:15.0:1506127961.257458:0:3000:0:(osp_trans.c:272:object_update_request_dump()) i = 3 fid = [0x2c0003ab2:0x2:0x0] op = insert params = 3 batchid = 0 size = 96 repsize 0 00000004:00000040:15.0:1506127961.257461:0:3000:0:(osp_trans.c:272:object_update_request_dump()) i = 4 fid = [0x2c0003ab2:0x2:0x0] op = xattr_set params = 3 batchid = 0 size = 144 repsize 0 00000004:00000040:15.0:1506127961.257464:0:3000:0:(osp_trans.c:272:object_update_request_dump()) i = 5 fid = [0x2c0003ab2:0x2:0x0] op = xattr_set params = 3 batchid = 0 size = 160 repsize 0 00000004:00000040:15.0:1506127961.257467:0:3000:0:(osp_trans.c:272:object_update_request_dump()) i = 6 fid = [0x2c0003ab2:0x2:0x0] op = xattr_set params = 3 batchid = 0 size = 96 repsize 0 00000004:00000040:15.0:1506127961.257469:0:3000:0:(osp_trans.c:272:object_update_request_dump()) i = 7 fid = [0x2c0003ab2:0x3:0x0] op = create params = 1 batchid = 0 size = 256 repsize 0 00000004:00000040:15.0:1506127961.257472:0:3000:0:(osp_trans.c:272:object_update_request_dump()) i = 8 fid = [0x20000000a:0x3:0x0] op = insert params = 3 batchid = 0 size = 112 repsize 0 00000004:00000040:15.0:1506127961.257475:0:3000:0:(osp_trans.c:278:object_update_request_dump()) updates = ffff8804b92e0000 magic = bdde0002 count = 9 size = 1256 00000004:00000040:15.0:1506127961.257478:0:3000:0:(osp_trans.c:272:object_update_request_dump()) i = 0 fid = [0x2c0003ab2:0x1:0x0] op = write params = 2 batchid = 0 size = 32832 repsize 0 00000004:00000040:15.0:1506127961.257482:0:3000:0:(osp_trans.c:278:object_update_request_dump()) updates = ffff880806030000 magic = bdde0002 count = 1 size = 32832 00000004:00000040:15.0:1506127961.257484:0:3000:0:(osp_trans.c:272:object_update_request_dump()) i = 0 fid = [0x2c0003ab2:0x1:0x0] op = write params = 2 batchid = 0 size = 128 repsize 0 00000004:00000040:15.0:1506127961.257487:0:3000:0:(osp_trans.c:278:object_update_request_dump()) updates = ffff8808067f6000 magic = bdde0002 count = 1 size = 128 00000004:00000040:15.0:1506127961.257489:0:3000:0:(osp_trans.c:272:object_update_request_dump()) i = 0 fid = [0x2c0003ab2:0x3:0x0] op = write params = 2 batchid = 0 size = 32832 repsize 0 00000004:00000040:15.0:1506127961.257492:0:3000:0:(osp_trans.c:278:object_update_request_dump()) updates = ffff880806040000 magic = bdde0002 count = 1 size = 32832 00000004:00000040:15.0:1506127961.257494:0:3000:0:(osp_trans.c:272:object_update_request_dump()) i = 0 fid = [0x2c0003ab2:0x3:0x0] op = write params = 2 batchid = 0 size = 2696 repsize 0 00000004:00000040:15.0:1506127961.257497:0:3000:0:(osp_trans.c:278:object_update_request_dump()) updates = ffff8808067f7000 magic = bdde0002 count = 1 size = 2696 </pre> </div></div> <p>But on the receiver, the magic has been changed to zero.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000800:00000200:10.0:1506127961.258292:0:3044:0:(o2iblnd_cb.c:1386:kiblnd_launch_tx()) conn[ffff88082dfb7200] (20)++ 00000800:00000200:10.0:1506127961.258294:0:3044:0:(o2iblnd_cb.c:1156:kiblnd_queue_tx_locked()) conn[ffff88082dfb7200] (21)++ 00000800:00000200:10.0:1506127961.258299:0:3044:0:(o2iblnd_cb.c:1392:kiblnd_launch_tx()) conn[ffff88082dfb7200] (22)-- 00000100:00000200:10.0:1506127961.258302:0:3044:0:(niobuf.c:262:ptlrpc_start_bulk_transfer()) Transferring 5 pages 77970 bytes via portal 14 id 12345-192.168.1.109@o2ib mbits 0x59c5acbf08b80-0x59c5acbf08b80 00000100:00000001:10.0:1506127961.258306:0:3044:0:(niobuf.c:264:ptlrpc_start_bulk_transfer()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:10.0:1506127961.258595:0:3044:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=0 : 0 : 0) 00000020:00020000:10.0:1506127961.258600:0:3044:0:(out_handler.c:1000:out_handle()) soaked-MDT0003: invalid update buffer magic 0 expect bdde0002: rc = -71 00000020:00000001:10.0:1506127961.279044:0:3044:0:(out_handler.c:1001:out_handle()) Process leaving via out_free (rc=18446744073709547449 : -4167 : 0xffffffffffffefb9) 00000020:00000010:10.0:1506127961.279048:0:3044:0:(out_handler.c:1165:out_handle()) kfreed 'update_bufs[i]': 4096 at ffff88083fcb1000. 00000020:00000010:10.0:1506127961.279066:0:3044:0:(out_handler.c:1165:out_handle()) kfreed 'update_bufs[i]': 32841 at ffff88080f180000. 00000020:00000010:10.0:1506127961.279086:0:3044:0:(out_handler.c:1165:out_handle()) kfreed 'update_bufs[i]': 4096 at ffff88080f54f000. 00000020:00000010:10.0:1506127961.279090:0:3044:0:(out_handler.c:1165:out_handle()) kfreed 'update_bufs[i]': 32841 at ffff88080f190000. 00000020:00000010:10.0:1506127961.279115:0:3044:0:(out_handler.c:1165:out_handle()) kfreed 'update_bufs[i]': 4096 at ffff880817a78000. 00000020:00000010:10.0:1506127961.279117:0:3044:0:(out_handler.c:1169:out_handle()) kfreed 'update_bufs': 40 at ffff88042b7fc340. </pre> </div></div> <p>Sarah told me this failure did not happen for build 18, whose top is<br/> <a href="https://jira.whamcloud.com/browse/LU-9903" title="kernel update [RHEL6.9 2.6.32-696.10.1.el6]" class="issue-link" data-issue-key="LU-9903"><del>LU-9903</del></a> kernel: kernel update RHEL6.9 <span class="error">&#91;2.6.32-696.10.1.el6&#93;</span> </p> <p>So I checked all of commit since <a href="https://jira.whamcloud.com/browse/LU-9725" title="Mount commands don&#39;t return for targets in LFS with DNE and 3 MDTs " class="issue-link" data-issue-key="LU-9725"><del>LU-9725</del></a>, the most suspicious patches are kernel upgrade, so I revert them <a href="https://review.whamcloud.com/#/c/29179/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/29179/</a> . Cliff: could you please try this? Thanks.</p> <p>That patch produced the old kernel, (3.10.0-514.26.2) which doesn't work with CENTOS 7.4, can you re-base on a current kernel? (3.10.0-693.2.2)<br/> I just finished up-grading soak to 7.4, really don't want to downgrade. </p> <p>wangdi (di.wang@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/29185" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29185</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9983" title="LBUG llog_osd.c:327:llog_osd_declare_write_rec() - all DNE MDS" class="issue-link" data-issue-key="LU-9983"><del>LU-9983</del></a> lustre: performance v1<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 16343af007dc503f38926fae3a7ebd4e7d66e5a0</p> <p>wangdi (di.wang@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/29186" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29186</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9983" title="LBUG llog_osd.c:327:llog_osd_declare_write_rec() - all DNE MDS" class="issue-link" data-issue-key="LU-9983"><del>LU-9983</del></a> lustre: 9983 try v2<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: d7a390d2f9bb337476c8a1a2b20b524ebc11e7c9</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> can you re-base on a current kernel? (3.10.0-693.2.2) I just finished up-grading soak to 7.4, really don't want to downgrade. </pre> </div></div> <p>Ok, then let's use bisect to find out which patch caused this issue. Could you please try patch 29185 and 29186? And please remove the update log before re-run thanks.</p> <p>Sunday, I re-loaded build 26 and reformatted without DNE. Soak has run for 14 hours without any issues. </p> <p>Cliff tried 29185 and it also hit this LBUG. Build 29185 is basically build 18 + 2 kernel upgrade patches, <br/> <a href="https://review.whamcloud.com/29002" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29002</a><br/> <a href="https://review.whamcloud.com/28532" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28532</a></p> <p>Since we did not hit this problem for build 18. so it seems the kernel upgrade is the culprit here? (somehow it break the bulk transfer between MDTs).</p> <p>I will upload the client/server debug log here, could some network guys have a look?</p> <p>Cliff will also try running IOR with data verification (which we should probably be running on soak all the time), so we can verify if bulk transfers of data are working properly or not. This would help isolate if there are problems with the core bulk data transfer mechanism (which would point more in the direction of LNet or OFED changes), or if this is isolated to OUT.</p> <p>Di, it is also bad if the MDS is LASSERTing on data that it got over the network. It should only report an error in this case.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Di, it is also bad if the MDS is LASSERTing on data that it got over the network. It should only report an error in this case. </pre> </div></div> <p>Oh, I posted a patch to fix this LASSERT <a href="https://review.whamcloud.com/29178" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29178</a> , please check. thanks.</p> <p>Loaded patch 29185 - immediate LBUG. loading next patch</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Sep 25 16:22:16 soak-10 kernel: LustreError: 13509:0:(llog_osd.c:327:llog_osd_declare_write_rec()) ASSERTION( rec ) failed: Sep 25 16:22:16 soak-10 kernel: LustreError: 13509:0:(llog_osd.c:327:llog_osd_declare_write_rec()) LBUG Sep 25 16:22:16 soak-10 kernel: Pid: 13509, comm: dist_txn-2 Sep 25 16:22:16 soak-10 kernel: #012Call Trace: Sep 25 16:22:16 soak-10 kernel: [&lt;ffffffffc09187ae&gt;] libcfs_call_trace+0x4e/0x60 [libcfs] Sep 25 16:22:16 soak-10 kernel: [&lt;ffffffffc091883c&gt;] lbug_with_loc+0x4c/0xb0 [libcfs] Sep 25 16:22:16 soak-10 kernel: [&lt;ffffffffc0a3bac5&gt;] llog_osd_declare_write_rec+0x3c5/0x3d0 [obdclass] Sep 25 16:22:16 soak-10 kernel: [&lt;ffffffffc0a300e4&gt;] llog_declare_write_rec+0x84/0x200 [obdclass] Sep 25 16:22:16 soak-10 kernel: [&lt;ffffffffc0a30862&gt;] llog_cancel_rec+0xe2/0x870 [obdclass] Sep 25 16:22:16 soak-10 kernel: [&lt;ffffffffc0a3728a&gt;] llog_cat_cancel_records+0x13a/0x2e0 [obdclass] Sep 25 16:22:16 soak-10 kernel: [&lt;ffffffffc0923bc7&gt;] ? libcfs_debug_msg+0x57/0x80 [libcfs] Sep 25 16:22:17 soak-10 kernel: [&lt;ffffffffc0d170e9&gt;] distribute_txn_commit_thread+0x599/0xca0 [ptlrpc] Sep 25 16:22:17 soak-10 kernel: [&lt;ffffffff810c4810&gt;] ? default_wake_function+0x0/0x20 Sep 25 16:22:17 soak-10 kernel: [&lt;ffffffffc0d16b50&gt;] ? distribute_txn_commit_thread+0x0/0xca0 [ptlrpc] Sep 25 16:22:17 soak-10 kernel: [&lt;ffffffff810b098f&gt;] kthread+0xcf/0xe0 Sep 25 16:22:17 soak-10 kernel: [&lt;ffffffff810b08c0&gt;] ? kthread+0x0/0xe0 Sep 25 16:22:17 soak-10 kernel: [&lt;ffffffff816b4f18&gt;] ret_from_fork+0x58/0x90 Sep 25 16:22:17 soak-10 kernel: [&lt;ffffffff810b08c0&gt;] ? kthread+0x0/0xe0 Sep 25 16:22:17 soak-10 kernel: Sep 25 16:22:17 soak-10 kernel: Kernel panic - not syncing: LBUG </pre> </div></div> <p>Errors from soak-8</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Sep 25 16:21:41 soak-8 systemd: Removed slice User Slice of root. Sep 25 16:21:41 soak-8 systemd: Stopping User Slice of root. Sep 25 16:22:05 soak-8 kernel: LustreError: 13225:0:(out_handler.c:597:out_write()) soaked-MDT0000: empty or wrong size 0 pos: rc = -61 Sep 25 16:22:05 soak-8 kernel: LustreError: 13706:0:(out_handler.c:1000:out_handle()) soaked-MDT0000: invalid update buffer magic 0 expect bdde0002: rc = -71 Sep 25 16:22:47 soak-8 sshd[13757]: error: Could not load host key: /etc/ssh/ssh_host_dsa_key Sep 25 16:22:47 soak-8 sshd[13757]: Accepted publickey <span class="code-keyword">for</span> root from 10.10.1.116 port 50002 ssh2: RSA SHA256:VGwjPuk53LIsLKjhGizbClh9X4HNRiAOs+XaQdKAWxM Sep 25 16:22:47 soak-8 systemd: Created slice User Slice of root. Sep 25 16:22:47 soak-8 systemd: Starting User Slice of root. Sep 25 16:22:47 soak-8 systemd-logind: New session 26 of user root. Sep 25 16:22:47 soak-8 systemd: Started Session 26 of user root. Sep 25 16:22:47 soak-8 systemd: Starting Session 26 of user root. Sep 25 16:22:47 soak-8 sshd[13757]: pam_unix(sshd:session): session opened <span class="code-keyword">for</span> user root by (uid=0) Sep 25 16:22:47 soak-8 sshd[13757]: pam_unix(sshd:session): session closed <span class="code-keyword">for</span> user root Sep 25 16:22:47 soak-8 systemd-logind: Removed session 26. Sep 25 16:22:47 soak-8 systemd: Removed slice User Slice of root. Sep 25 16:22:47 soak-8 systemd: Stopping User Slice of root. Sep 25 16:22:49 soak-8 kernel: Lustre: 12799:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out <span class="code-keyword">for</span> sent delay: [sent 1506356561/real 0] req@ffff8803f3a70f00 x1579526007953840/t0(0) o400-&gt;soaked-MDT0002-osp-MDT0000@192.168.1.110@o2ib:24/4 lens 224/224 e 0 to 1 dl 1506356568 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Sep 25 16:22:49 soak-8 kernel: Lustre: soaked-MDT0003-osp-MDT0000: Connection to soaked-MDT0003 (at 192.168.1.111@o2ib) was lost; in progress operations using <span class="code-keyword">this</span> service will wait <span class="code-keyword">for</span> recovery to complete Sep 25 16:22:49 soak-8 kernel: Lustre: 12799:0:(client.c:2114:ptlrpc_expire_one_request()) Skipped 1 previous similar message Sep 25 16:22:50 soak-8 sshd[13780]: error: Could not load host key: /etc/ssh/ssh_host_dsa_key Sep 25 16:22:50 soak-8 sshd[13780]: Accepted publickey <span class="code-keyword">for</span> root from 10.10.1.116 port 50026 ssh2: RSA SHA256:VGwjPuk53LIsLKjhGizbClh9X4HNRiAOs+XaQdKAWxM </pre> </div></div> <p>Attempt to run the second patch with IOR and data verification. removed all old update_logs from MDTs. Soak started 2017-09-25 18:02:05 One IOR single-shared-file test completed without error. soak-9 and soak-11 immediately crashed: &lt;15 seconds after start of test. </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">soak-9.log:Sep 25 18:02:17 soak-9 kernel: LustreError: 2686:0:(llog_osd.c:327:llog_osd_declare_write_rec()) LBUG Sep 25 18:01:01 soak-9 systemd: Stopping User Slice of root. Sep 25 18:02:06 soak-9 kernel: LustreError: 2615:0:(out_handler.c:597:out_write()) soaked-MDT0001: empty or wrong size 0 pos: rc = -61 Sep 25 18:02:06 soak-9 kernel: LustreError: 11-0: soaked-MDT0000-osp-MDT0001: operation out_update to node 192.168.1.108@o2ib failed: rc = -71 Sep 25 18:02:06 soak-9 kernel: LustreError: 2630:0:(layout.c:2085:__req_capsule_get()) @@@ Wrong buffer <span class="code-keyword">for</span> field `object_update_reply<span class="code-quote">' (1 of 1) in format `OUT_UPDATE'</span>: 0 vs. 4096 (server)#012 req@ffff88080daa3900 x1579533157282240/t0(0) o1000-&gt;soaked-MDT0000-osp-MDT0001@192.168.1.108@o2ib:24/4 lens 392/192 e 0 to 0 dl 1506362537 ref 2 fl Interpret:RM/0/0 rc -71/-71 Sep 25 18:02:17 soak-9 kernel: LustreError: 2686:0:(llog_osd.c:327:llog_osd_declare_write_rec()) ASSERTION( rec ) failed: Sep 25 18:02:17 soak-9 kernel: LustreError: 2686:0:(llog_osd.c:327:llog_osd_declare_write_rec()) LBUG Sep 25 18:02:17 soak-9 kernel: Pid: 2686, comm: dist_txn-1 Sep 25 18:02:17 soak-9 kernel: #012Call Trace: Sep 25 18:02:17 soak-9 kernel: [&lt;ffffffffc09097ae&gt;] libcfs_call_trace+0x4e/0x60 [libcfs] Sep 25 18:02:17 soak-9 kernel: [&lt;ffffffffc090983c&gt;] lbug_with_loc+0x4c/0xb0 [libcfs] Sep 25 18:02:17 soak-9 kernel: [&lt;ffffffffc0a93ac5&gt;] llog_osd_declare_write_rec+0x3c5/0x3d0 [obdclass] Sep 25 18:02:17 soak-9 kernel: [&lt;ffffffffc0a880e4&gt;] llog_declare_write_rec+0x84/0x200 [obdclass] Sep 25 18:02:17 soak-9 kernel: [&lt;ffffffffc0a88862&gt;] llog_cancel_rec+0xe2/0x870 [obdclass] Sep 25 18:02:17 soak-9 kernel: [&lt;ffffffffc0a8f28a&gt;] llog_cat_cancel_records+0x13a/0x2e0 [obdclass] Sep 25 18:02:17 soak-9 kernel: [&lt;ffffffffc0914bc7&gt;] ? libcfs_debug_msg+0x57/0x80 [libcfs] Sep 25 18:02:17 soak-9 kernel: [&lt;ffffffffc0d20139&gt;] distribute_txn_commit_thread+0x599/0xca0 [ptlrpc] Sep 25 18:02:17 soak-9 kernel: [&lt;ffffffff810c12c8&gt;] ? check_preempt_curr+0x78/0xa0 Sep 25 18:02:17 soak-9 kernel: [&lt;ffffffff810c4810&gt;] ? default_wake_function+0x0/0x20 Sep 25 18:02:17 soak-9 kernel: [&lt;ffffffffc0d1fba0&gt;] ? distribute_txn_commit_thread+0x0/0xca0 [ptlrpc] Sep 25 18:02:17 soak-9 kernel: [&lt;ffffffff810b098f&gt;] kthread+0xcf/0xe0 Sep 25 18:02:17 soak-9 kernel: [&lt;ffffffff810b08c0&gt;] ? kthread+0x0/0xe0 Sep 25 18:02:17 soak-9 kernel: [&lt;ffffffff816b4f58&gt;] ret_from_fork+0x58/0x90 Sep 25 18:02:17 soak-9 kernel: [&lt;ffffffff810b08c0&gt;] ? kthread+0x0/0xe0 Sep 25 18:02:17 soak-9 kernel: Sep 25 18:02:17 soak-9 kernel: Kernel panic - not syncing: LBUG </pre> </div></div> <p>Next steps?</p> <p>Per request, lustre-log from soak-8 attached. </p> <p>Ah, so this seems to isolated to OUT. hmm, let me add more debug log.</p> <p>wangdi (di.wang@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/29202" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29202</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9983" title="LBUG llog_osd.c:327:llog_osd_declare_write_rec() - all DNE MDS" class="issue-link" data-issue-key="LU-9983"><del>LU-9983</del></a> out: try 3<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: 4e13d3da5beffabe3cb69268349c366acd36a825</p> <p>wangdi (di.wang@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/29207" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29207</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9983" title="LBUG llog_osd.c:327:llog_osd_declare_write_rec() - all DNE MDS" class="issue-link" data-issue-key="LU-9983"><del>LU-9983</del></a> out: try 3<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 6fcce248254b265f7521deaacb300420445e826b</p> <p>Please ignore this</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Cliff will also try running IOR with data verification (which we should probably be running on soak all the time), so we can verify if bulk transfers of data are working properly or not. This would help isolate if there are problems with the core bulk data transfer mechanism (which would point more in the direction of LNet or OFED changes), or if this is isolated to OUT. </pre> </div></div> <p>Note: bulk between MDTs are using different iovec ops than OSC&lt;--&gt; OST. See <a href="https://jira.whamcloud.com/browse/LU-5835" title="Introduce the IOVEC in ptlrpc" class="issue-link" data-issue-key="LU-5835"><del>LU-5835</del></a> <a href="http://review.whamcloud.com/12525/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12525/</a> </p> <p>If bulk transferring between OSC and OST are proved to be correct, then maybe we should check those iovec ops for MDTs.</p> <p>wangdi (di.wang@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/29208" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29208</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9983" title="LBUG llog_osd.c:327:llog_osd_declare_write_rec() - all DNE MDS" class="issue-link" data-issue-key="LU-9983"><del>LU-9983</del></a> osp: align the OSP request size by 4k<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: d741a50ea589969c92685b0fff63e22488b44bb4</p> <p>Although this patch can avoid the LBUG, but the real reason still needs to be understood. It looks like PTLRPC (or Lnet with RHEL 7.4 kernel) can not handle non-pagesize aligned buffer on the receiver side. </p> <p>Previous patch (build 50791) ran 4 hours in soak without LBUG. Switching to <a href="https://review.whamcloud.com/#/c/29208/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/29208/</a> - build 50793</p> <p>wangdi (di.wang@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/29218" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29218</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9983" title="LBUG llog_osd.c:327:llog_osd_declare_write_rec() - all DNE MDS" class="issue-link" data-issue-key="LU-9983"><del>LU-9983</del></a> llog: skip non-exist log cancellation<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 4df15f1acd5ccdb0d5628a2ce620d03d90ea9171</p> <p>Amir Shehata (amir.shehata@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/29222" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29222</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9983" title="LBUG llog_osd.c:327:llog_osd_declare_write_rec() - all DNE MDS" class="issue-link" data-issue-key="LU-9983"><del>LU-9983</del></a> lnet: test to see if this is the cause<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 6f109b6b5d3fadd077df0414292c52f33659c6e0</p> <p>Finally, I got two nodes with IB in Onyx, and can reproduce the issue easily. And also it works fine with tcp lnd. So it is definitely O2ib issue.</p> <p>If needed, I can ask Sarah to test it in RHEL7.3 to see if it works there.</p> <p>wangdi (di.wang@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/29238" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29238</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9983" title="LBUG llog_osd.c:327:llog_osd_declare_write_rec() - all DNE MDS" class="issue-link" data-issue-key="LU-9983"><del>LU-9983</del></a> llog: skip non-exist log cancellation<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: 27fe551a1c18e8d90c2c04ebb5a2dee315b26321</p> <p>wangdi (di.wang@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/29240" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29240</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9983" title="LBUG llog_osd.c:327:llog_osd_declare_write_rec() - all DNE MDS" class="issue-link" data-issue-key="LU-9983"><del>LU-9983</del></a> osp: align the OSP request size by 4k<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: d6bc57faa857d2fb5353476afbf416cf3fde255f</p> <p>Running <a href="https://build.hpdd.intel.com/job/lustre-reviews/50862/" class="external-link" target="_blank" rel="nofollow noopener">https://build.hpdd.intel.com/job/lustre-reviews/50862/</a>, </p> <ul class="alternate" type="square"> <li>connection drops have stopped</li> <li>random job failures have stopped.</li> <li>so far no LBUGS.<br/> This build is way better than the previous tests. </li> </ul> <p>John L. Hammond (john.hammond@intel.com) merged in patch <a href="https://review.whamcloud.com/29208/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29208/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9983" title="LBUG llog_osd.c:327:llog_osd_declare_write_rec() - all DNE MDS" class="issue-link" data-issue-key="LU-9983"><del>LU-9983</del></a> osp: align the OSP request size by 4k<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: <br/> Commit: c535d7a4021e3c520246bdde118be97414938266</p> <p>Bob Glossman (bob.glossman@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/29270" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29270</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9983" title="LBUG llog_osd.c:327:llog_osd_declare_write_rec() - all DNE MDS" class="issue-link" data-issue-key="LU-9983"><del>LU-9983</del></a> osp: align the OSP request size by 4k<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 9994585785eeb46fdb42b6b409764c03ea5a7651</p> <p>Running the (almost) latest patch, seeing quite a few of these:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">/scratch/logs/syslog/soak-8.log:Oct 1 22:37:25 soak-8 kernel: LustreError: 8097:0:(mdt_lvb.c:163:mdt_lvbo_fill()) Skipped 4 previous similar messages /scratch/logs/syslog/soak-8.log:Oct 1 22:37:25 soak-8 kernel: LustreError: 8097:0:(mdt_lvb.c:163:mdt_lvbo_fill()) soaked-MDT0000: expected 944 actual 416. /scratch/logs/syslog/soak-9.log:Oct 1 22:42:25 soak-9 kernel: LustreError: 2165:0:(mdt_lvb.c:163:mdt_lvbo_fill()) Skipped 6 previous similar messages /scratch/logs/syslog/soak-9.log:Oct 1 22:42:25 soak-9 kernel: LustreError: 2165:0:(mdt_lvb.c:163:mdt_lvbo_fill()) soaked-MDT0001: expected 872 actual 416. /scratch/logs/syslog/soak-10.log:Oct 1 22:42:26 soak-10 kernel: LustreError: 2401:0:(mdt_lvb.c:163:mdt_lvbo_fill()) Skipped 10 previous similar messages /scratch/logs/syslog/soak-10.log:Oct 1 22:42:26 soak-10 kernel: LustreError: 2401:0:(mdt_lvb.c:163:mdt_lvbo_fill()) soaked-MDT0002: expected 872 actual 416. /scratch/logs/syslog/soak-10.log:Oct 1 22:42:26 soak-10 kernel: LustreError: 4181:0:(mdt_lvb.c:163:mdt_lvbo_fill()) soaked-MDT0002: expected 872 actual 416. /scratch/logs/syslog/soak-10.log:Oct 1 22:44:04 soak-10 kernel: LustreError: 2351:0:(mdt_lvb.c:163:mdt_lvbo_fill()) soaked-MDT0002: expected 872 actual 416. /scratch/logs/syslog/soak-9.log:Oct 1 22:44:04 soak-9 kernel: LustreError: 2351:0:(mdt_lvb.c:163:mdt_lvbo_fill()) soaked-MDT0001: expected 848 actual 416. /scratch/logs/syslog/soak-10.log:Oct 1 22:57:27 soak-10 kernel: LustreError: 4296:0:(mdt_lvb.c:163:mdt_lvbo_fill()) Skipped 8 previous similar messages /scratch/logs/syslog/soak-10.log:Oct 1 22:57:27 soak-10 kernel: LustreError: 4296:0:(mdt_lvb.c:163:mdt_lvbo_fill()) soaked-MDT0002: expected 872 actual 416. /scratch/logs/syslog/soak-9.log:Oct 1 22:57:27 soak-9 kernel: LustreError: 2329:0:(mdt_lvb.c:163:mdt_lvbo_fill()) Skipped 9 previous similar messages /scratch/logs/syslog/soak-9.log:Oct 1 22:57:27 soak-9 kernel: LustreError: 2329:0:(mdt_lvb.c:163:mdt_lvbo_fill()) soaked-MDT0001: expected 800 actual 416. /scratch/logs/syslog/soak-9.log:Oct 1 22:59:06 soak-9 kernel: LustreError: 2357:0:(mdt_lvb.c:163:mdt_lvbo_fill()) soaked-MDT0001: expected 776 actual 416. </pre> </div></div> <p>We see some timeouts/lustre-log dumps, but no LBUGs. Further details tomorrow, timeouts may be failover-related.</p> <p>I think the mdt_lvbo_fill() error is fallout from PFL and has been seen on master (<a href="https://jira.whamcloud.com/browse/LU-9825" title="Multiple errors on OST/MDS " class="issue-link" data-issue-key="LU-9825">LU-9825</a>). That said, it is so commonly printed as to make it very annoying, and IMHO something that should be fixed. Lai, could you please take a look.</p> <p>Lai will be out this week so is there someone else who could confirm that this is benign enough to live with in 2.10.1 or not?</p> <p>John L. Hammond (john.hammond@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/29290" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29290</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9983" title="LBUG llog_osd.c:327:llog_osd_declare_write_rec() - all DNE MDS" class="issue-link" data-issue-key="LU-9983"><del>LU-9983</del></a> ko2iblnd: allow for discontiguous fragments<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: cfce4303780fb6a12d2b08b784e333a2de7de07b</p> <p>Should I move soak to this patch? </p> <p>Yes, please test the viability of John's patch as a fix for this issue. It has been validated already by Amir/John on their side. After a successful run at the soak scale, we can create a tmp patch to revert Di's fix and have only John's patch present for a final verification.</p> <p>I don't think we need to revert Di's patch, even if John's patch works. </p> <p>I'll do this on Monday.</p> <p>Thanks Cliff.</p> <p>Di - do you agree with Andreas that your fix is useful independent of John's proposed fix?</p> <p>Yes, Joe. I agree with Andreas.</p> <p>With this patch, multiple errors when attempting client mounts. Servers mount.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">[ 4377.206482] Lustre: 10866:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out <span class="code-keyword">for</span> slow reply: [sent 1507572416/real 1507572416] req@ffff880825ec0300 x1580800666501216/t0(0) o503-&gt;MGC192.168.1.108@o2ib@192.168.1.108@o2ib:26/25 lens 272/8416 e 0 to 1 dl 1507572460 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 [ 4377.238308] LustreError: 166-1: MGC192.168.1.108@o2ib: Connection to MGS (at 192.168.1.108@o2ib) was lost; in progress operations using <span class="code-keyword">this</span> service will fail [ 4377.254213] LustreError: 15c-8: MGC192.168.1.108@o2ib: The configuration from log <span class="code-quote">'soaked-client'</span> failed (-5). This may be the result of communication errors between <span class="code-keyword">this</span> node and the MGS, a bad configuration, or other errors. See the syslog <span class="code-keyword">for</span> more information. [ 4377.254668] Lustre: MGC192.168.1.108@o2ib: Connection restored to MGC192.168.1.108@o2ib_0 (at 192.168.1.108@o2ib) </pre> </div></div> <p>Clients do not mount with this patch - set -1 debug, dumped lustre-log from MDS and client after failure. Attached. </p> <p>John L. Hammond (john.hammond@intel.com) merged in patch <a href="https://review.whamcloud.com/29238/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29238/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9983" title="LBUG llog_osd.c:327:llog_osd_declare_write_rec() - all DNE MDS" class="issue-link" data-issue-key="LU-9983"><del>LU-9983</del></a> llog: skip non-exist log cancellation<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: <br/> Commit: 858f0ee15eb0ba2eca525297d510213ba6c5ec5e</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/29290/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29290/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9983" title="LBUG llog_osd.c:327:llog_osd_declare_write_rec() - all DNE MDS" class="issue-link" data-issue-key="LU-9983"><del>LU-9983</del></a> ko2iblnd: allow for discontiguous fragments<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 7954a520428002de8629916d4ae7b6660818dc6c</p> <p>Cray is seeing this issue in our 2.10 testing. I read through the comments in this ticket but I did not see a clear description of the root cause of this issue. Could someone provide a description? I see a ko2iblnd change was made in association with this patch, and I'm concerned that gnilnd may be vulnerable as well.</p> <p>Chris, the root cause is the bug addressed by <a href="https://review.whamcloud.com/29290/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29290</a></p> <p>See the commit message for an explanation. This in turn triggers a second bug which is addressed by <a href="https://review.whamcloud.com/29238/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29238</a>/.</p> <p>You can also see the failed assertion in some OOM cases on an MDS.</p> <p>Got it. Thanks, John.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/29218/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29218/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9983" title="LBUG llog_osd.c:327:llog_osd_declare_write_rec() - all DNE MDS" class="issue-link" data-issue-key="LU-9983"><del>LU-9983</del></a> llog: skip non-exist log cancellation<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 16d7737894c0a8a111f64f04d7c6bdf0ddc86167</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/29270/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29270/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9983" title="LBUG llog_osd.c:327:llog_osd_declare_write_rec() - all DNE MDS" class="issue-link" data-issue-key="LU-9983"><del>LU-9983</del></a> osp: align the OSP request size by 4k<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 1f50b1e494ff1b4988508c6d6398ee6769467931</p> <p>It looks like all the parts of this have finally landed to master</p> <p>Minh Diep (minh.diep@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/30580" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30580</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9983" title="LBUG llog_osd.c:327:llog_osd_declare_write_rec() - all DNE MDS" class="issue-link" data-issue-key="LU-9983"><del>LU-9983</del></a> ko2iblnd: allow for discontiguous fragments<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: 0d10af5408ef98801920c04daf86d1d21c8d6dda</p> Related LU-9500 LU-9026 LU-9810 LU-10089 LU-10068 Development Rank 1|hzzk4f: Rank (Obsolete) 9223372036854775807 Severity