https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4183 Lustre <p>Running three instances of fsx writing to an NFS mounted Lustre file system with a single NFS server, the MDS crashes and reboots. From the MDS console:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Message from syslogd@c03 at Oct 29 13:04:27 ... kernel:LustreError: 16448:0:(osd_internal.h:963:osd_trans_exec_op()) ASSERTION( rb &lt; OSD_OT_MAX ) failed: rb = 12 Message from syslogd@c03 at Oct 29 13:04:27 ... kernel:LustreError: 16448:0:(osd_internal.h:963:osd_trans_exec_op()) LBUG </pre> </div></div> <p>From the crash dump:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>crash&gt; bt PID: 16448 TASK: ffff880738300040 CPU: 4 COMMAND: "mdt00_005" #0 [ffff8806565c1628] machine_kexec at ffffffff81035d6b #1 [ffff8806565c1688] crash_kexec at ffffffff810c0e22 #2 [ffff8806565c1758] panic at ffffffff8150de5f #3 [ffff8806565c17d8] lbug_with_loc at ffffffffa04f0eeb [libcfs] #4 [ffff8806565c17f8] osd_punch at ffffffffa0d12243 [osd_ldiskfs] #5 [ffff8806565c1848] llog_osd_write_blob at ffffffffa0639050 [obdclass] #6 [ffff8806565c18b8] llog_osd_write_rec at ffffffffa063d04e [obdclass] #7 [ffff8806565c1968] llog_write_rec at ffffffffa0610428 [obdclass] #8 [ffff8806565c19c8] llog_cat_add_rec at ffffffffa0619299 [obdclass] #9 [ffff8806565c1a38] llog_add at ffffffffa0610221 [obdclass] #10 [ffff8806565c1a88] mdd_changelog_store at ffffffffa0f54463 [mdd] #11 [ffff8806565c1af8] mdd_changelog_data_store at ffffffffa0f465c6 [mdd] #12 [ffff8806565c1b48] mdd_attr_set at ffffffffa0f4c9b1 [mdd] #13 [ffff8806565c1bc8] mdt_attr_set at ffffffffa0e1cc28 [mdt] #14 [ffff8806565c1c18] mdt_reint_setattr at ffffffffa0e1d4dd [mdt] #15 [ffff8806565c1c88] mdt_reint_rec at ffffffffa0e16eb1 [mdt] #16 [ffff8806565c1ca8] mdt_reint_internal at ffffffffa0dfec93 [mdt] #17 [ffff8806565c1ce8] mdt_reint at ffffffffa0dfef94 [mdt] #18 [ffff8806565c1d08] mdt_handle_common at ffffffffa0e01a8a [mdt] #19 [ffff8806565c1d58] mds_regular_handle at ffffffffa0e3bc55 [mdt] #20 [ffff8806565c1d68] ptlrpc_server_handle_request at ffffffffa07f2e25 [ptlrpc] #21 [ffff8806565c1e48] ptlrpc_main at ffffffffa07f418d [ptlrpc] #22 [ffff8806565c1ee8] kthread at ffffffff81096a36 #23 [ffff8806565c1f48] kernel_thread at ffffffff8100c0ca </pre> </div></div> <p>On the OSTs, osd-ldiskfs.track_declares_assert=1</p> <p>From the crash dump, here is the system information:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>crash&gt; sys KERNEL: /home/build/kernel/rpmbuild/BUILD/kernel-2.6.32.358.18.1.el6_lustre/vmlinux DUMPFILE: /var/crash/127.0.0.1-2013-10-29-13:05:16/vmcore [PARTIAL DUMP] CPUS: 24 DATE: Tue Oct 29 13:04:30 2013 UPTIME: 3 days, 22:31:58 LOAD AVERAGE: 0.17, 0.06, 0.02 TASKS: 362 NODENAME: c03 RELEASE: 2.6.32-358.18.1.el6_lustre.x86_64 VERSION: #1 SMP Fri Oct 11 16:41:53 PDT 2013 MACHINE: x86_64 (2399 Mhz) MEMORY: 32 GB PANIC: "Kernel panic - not syncing: LBUG" </pre> </div></div> <p>Two of the three fsx commands were:<br/> ./fsx -c 5 /misc/export/nfs_x_1<br/> ./fsx /misc/export/nfs_x_2</p> Lustre 2.5.0-RC1, RHEL 6, OpenSFS cluster with one MDS/MGS, two OSSs with two OSTs each, one node with Lustre mounted/NFS server exporting Lustre, three Lustre clients NFS mounting the file system LU-4183

MDS crash when running fsx with NFS export

Bug Minor Resolved Duplicate WC Triage James Nunez Tue, 29 Oct 2013 21:04:44 +0000 Thu, 14 Aug 2014 17:24:20 +0000 Thu, 14 Aug 2014 17:24:20 +0000 Lustre 2.4.1 Lustre 2.5.0 0 8 <p>Looking at the log in the crash dump:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>crash&gt; log ... LustreError: 16448:0:(osd_io.c:1065:osd_ldiskfs_write_record()) sda3: error reading offset 1249280 (block 305): rc = -28 LustreError: 16448:0:(llog_osd.c:156:llog_osd_write_blob()) scratch-MDT0000-osd: error writing log record: rc = -28 LustreError: 16448:0:(osd_io.c:1065:osd_ldiskfs_write_record()) sda3: error reading offset 1249280 (block 305): rc = -28 LustreError: 16448:0:(llog_osd.c:156:llog_osd_write_blob()) scratch-MDT0000-osd: error writing log record: rc = -28 LustreError: 16448:0:(osd_internal.h:963:osd_trans_exec_op()) ASSERTION( rb &lt; OSD_OT_MAX ) failed: rb = 12 LustreError: 16448:0:(osd_internal.h:963:osd_trans_exec_op()) LBUG Pid: 16448, comm: mdt00_005 Call Trace: [&lt;ffffffffa04f0895&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] [&lt;ffffffffa04f0e97&gt;] lbug_with_loc+0x47/0xb0 [libcfs] [&lt;ffffffffa0d12243&gt;] osd_punch+0x613/0x620 [osd_ldiskfs] [&lt;ffffffffa0639050&gt;] llog_osd_write_blob+0x1e0/0x850 [obdclass] [&lt;ffffffffa063d04e&gt;] llog_osd_write_rec+0xb5e/0x1370 [obdclass] [&lt;ffffffffa0610428&gt;] llog_write_rec+0xc8/0x290 [obdclass] [&lt;ffffffffa0619299&gt;] llog_cat_add_rec+0x2b9/0x480 [obdclass] [&lt;ffffffffa0610221&gt;] llog_add+0x91/0x1d0 [obdclass] [&lt;ffffffffa0f54463&gt;] mdd_changelog_store+0xf3/0x260 [mdd] [&lt;ffffffffa0f465c6&gt;] mdd_changelog_data_store+0x1a6/0x280 [mdd] [&lt;ffffffffa0f4c9b1&gt;] mdd_attr_set+0xeb1/0x1470 [mdd] [&lt;ffffffffa0e1cc28&gt;] mdt_attr_set+0x268/0x560 [mdt] [&lt;ffffffffa0e1d4dd&gt;] mdt_reint_setattr+0x5bd/0xcf0 [mdt] [&lt;ffffffffa07e2f9e&gt;] ? lustre_pack_reply_flags+0xae/0x1f0 [ptlrpc] [&lt;ffffffffa0e16eb1&gt;] mdt_reint_rec+0x41/0xe0 [mdt] [&lt;ffffffffa0dfec93&gt;] mdt_reint_internal+0x4c3/0x780 [mdt] [&lt;ffffffffa0dfef94&gt;] mdt_reint+0x44/0xe0 [mdt] [&lt;ffffffffa0e01a8a&gt;] mdt_handle_common+0x52a/0x1470 [mdt] [&lt;ffffffffa0e3bc55&gt;] mds_regular_handle+0x15/0x20 [mdt] [&lt;ffffffffa07f2e25&gt;] ptlrpc_server_handle_request+0x385/0xc00 [ptlrpc] [&lt;ffffffffa050227f&gt;] ? lc_watchdog_touch+0x6f/0x170 [libcfs] [&lt;ffffffffa07ea4c9&gt;] ? ptlrpc_wait_event+0xa9/0x2d0 [ptlrpc] [&lt;ffffffffa07f418d&gt;] ptlrpc_main+0xaed/0x1740 [ptlrpc] [&lt;ffffffffa07f36a0&gt;] ? ptlrpc_main+0x0/0x1740 [ptlrpc] [&lt;ffffffff81096a36&gt;] kthread+0x96/0xa0 [&lt;ffffffff8100c0ca&gt;] child_rip+0xa/0x20 [&lt;ffffffff810969a0&gt;] ? kthread+0x0/0xa0 [&lt;ffffffff8100c0c0&gt;] ? child_rip+0x0/0x20 Kernel panic - not syncing: LBUG Pid: 16448, comm: mdt00_005 Not tainted 2.6.32-358.18.1.el6_lustre.x86_64 #1 Call Trace: [&lt;ffffffff8150de58&gt;] ? panic+0xa7/0x16f [&lt;ffffffffa04f0eeb&gt;] ? lbug_with_loc+0x9b/0xb0 [libcfs] [&lt;ffffffffa0d12243&gt;] ? osd_punch+0x613/0x620 [osd_ldiskfs] [&lt;ffffffffa0639050&gt;] ? llog_osd_write_blob+0x1e0/0x850 [obdclass] [&lt;ffffffffa063d04e&gt;] ? llog_osd_write_rec+0xb5e/0x1370 [obdclass] [&lt;ffffffffa0610428&gt;] ? llog_write_rec+0xc8/0x290 [obdclass] [&lt;ffffffffa0619299&gt;] ? llog_cat_add_rec+0x2b9/0x480 [obdclass] [&lt;ffffffffa0610221&gt;] ? llog_add+0x91/0x1d0 [obdclass] [&lt;ffffffffa0f54463&gt;] ? mdd_changelog_store+0xf3/0x260 [mdd] [&lt;ffffffffa0f465c6&gt;] ? mdd_changelog_data_store+0x1a6/0x280 [mdd] [&lt;ffffffffa0f4c9b1&gt;] ? mdd_attr_set+0xeb1/0x1470 [mdd] [&lt;ffffffffa0e1cc28&gt;] ? mdt_attr_set+0x268/0x560 [mdt] [&lt;ffffffffa0e1d4dd&gt;] ? mdt_reint_setattr+0x5bd/0xcf0 [mdt] [&lt;ffffffffa07e2f9e&gt;] ? lustre_pack_reply_flags+0xae/0x1f0 [ptlrpc] [&lt;ffffffffa0e16eb1&gt;] ? mdt_reint_rec+0x41/0xe0 [mdt] [&lt;ffffffffa0dfec93&gt;] ? mdt_reint_internal+0x4c3/0x780 [mdt] [&lt;ffffffffa0dfef94&gt;] ? mdt_reint+0x44/0xe0 [mdt] [&lt;ffffffffa0e01a8a&gt;] ? mdt_handle_common+0x52a/0x1470 [mdt] [&lt;ffffffffa0e3bc55&gt;] ? mds_regular_handle+0x15/0x20 [mdt] [&lt;ffffffffa07f2e25&gt;] ? ptlrpc_server_handle_request+0x385/0xc00 [ptlrpc] [&lt;ffffffffa050227f&gt;] ? lc_watchdog_touch+0x6f/0x170 [libcfs] [&lt;ffffffffa07ea4c9&gt;] ? ptlrpc_wait_event+0xa9/0x2d0 [ptlrpc] [&lt;ffffffffa07f418d&gt;] ? ptlrpc_main+0xaed/0x1740 [ptlrpc] [&lt;ffffffffa07f36a0&gt;] ? ptlrpc_main+0x0/0x1740 [ptlrpc] [&lt;ffffffff81096a36&gt;] ? kthread+0x96/0xa0 [&lt;ffffffff8100c0ca&gt;] ? child_rip+0xa/0x20 [&lt;ffffffff810969a0&gt;] ? kthread+0x0/0xa0 [&lt;ffffffff8100c0c0&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p>It looks like this is a bug in llog_osd_write_blob() calling dt_punch() when a write fails:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> rc = dt_record_write(env, o, &amp;lgi-&gt;lgi_buf, off, th); <span class="code-keyword">if</span> (rc) CERROR(<span class="code-quote">"%s: error writing log tail: rc = %d\n"</span>, o-&gt;do_lu.lo_dev-&gt;ld_obd-&gt;obd_name, rc); out_unlock: dt_write_unlock(env, o); out: <span class="code-comment">/* cleanup the content written above */</span> <span class="code-keyword">if</span> (rc) { dt_punch(env, o, lgi-&gt;lgi_attr.la_size, OBD_OBJECT_EOF, th, BYPASS_CAPA); dt_attr_set(env, o, &amp;lgi-&gt;lgi_attr, th, BYPASS_CAPA); } </pre> </div></div> <p>There are two separate classes of problems that I can see here:</p> <ul class="alternate" type="square"> <li>Because OSD_OT_PUNCH is not declared, osd_trans_exec_op() considers this a rollback operation. The osd_trans_declare_op2rb[] mapping has the rollback operation <span class="error">&#91;OSD_OT_WRITE&#93;</span> = OSD_OT_WRITE instead of OSD_OT_PUNCH.</li> <li>The rollback mapping for <span class="error">&#91;OSD_OT_PUNCH&#93;</span> = OSD_OT_MAX, which means that truncates can never be rolled back, which is what triggers this LASSERT().</li> </ul> <ul class="alternate" type="square"> <li>llog_osd_write_blob() it saves the llog size lgi_attr before the lock is held. That means racing llog writes could invalidate this size before the write is actually done.</li> <li>In llog_osd_write_blob() it drops dt_write_lock() before doing the error handling. This means that another thread could start writing to the llog, and then the truncate erases llog operations that are unrelated to the failure. In this case the failure is due to -ENOSPC on the "non-buf" write, so there is no need to do the truncate at all. The truncate is only be needed in case the "buf" is partially written, and should be done while holding dt_write_lock() on the object.</li> </ul> <p>I hit this too today, Also due to ENOSPC, apparently. nfs was not involved (mainline kernel lustre client).<br/> It was a sanity.sh test run:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>&lt;4&gt;[ 5963.714103] Lustre: DEBUG MARKER: == sanity test 103: acl test ========================================================================= 00:14:02 (1390626842) &lt;0&gt;[ 5964.573079] LustreError: 21998:0:(osd_internal.h:959:osd_trans_exec_op()) ASSERTION( rb &lt; OSD_OT_MAX ) failed: rb = 12 &lt;0&gt;[ 5964.573451] LustreError: 21998:0:(osd_internal.h:959:osd_trans_exec_op()) LBUG &lt;4&gt;[ 5964.573755] Pid: 21998, comm: mdt00_003 &lt;4&gt;[ 5964.573910] &lt;4&gt;[ 5964.573910] Call Trace: &lt;4&gt;[ 5964.574184] [&lt;ffffffffa041a8a5&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] &lt;4&gt;[ 5964.574386] [&lt;ffffffffa041aea7&gt;] lbug_with_loc+0x47/0xb0 [libcfs] &lt;4&gt;[ 5964.574580] [&lt;ffffffffa0978ad3&gt;] osd_punch+0x613/0x620 [osd_ldiskfs] &lt;4&gt;[ 5964.574787] [&lt;ffffffffa0577d80&gt;] llog_osd_write_blob+0x1e0/0x850 [obdclass] &lt;4&gt;[ 5964.574993] [&lt;ffffffffa057bdad&gt;] llog_osd_write_rec+0xadd/0x13a0 [obdclass] &lt;4&gt;[ 5964.575191] [&lt;ffffffffa0365412&gt;] ? do_get_write_access+0x3d2/0x550 [jbd2] &lt;4&gt;[ 5964.575407] [&lt;ffffffffa056a4f8&gt;] llog_write_rec+0xc8/0x290 [obdclass] &lt;4&gt;[ 5964.575602] [&lt;ffffffffa05729e9&gt;] llog_cat_add_rec+0x2b9/0x480 [obdclass] &lt;4&gt;[ 5964.575800] [&lt;ffffffffa056a2f1&gt;] llog_add+0x91/0x1d0 [obdclass] &lt;4&gt;[ 5964.575984] [&lt;ffffffffa0e8c946&gt;] osp_sync_add_rec+0x246/0xa10 [osp] &lt;4&gt;[ 5964.576173] [&lt;ffffffffa0cdb8e8&gt;] ? __ldiskfs_journal_stop+0x68/0xa0 [ldiskfs] &lt;4&gt;[ 5964.576482] [&lt;ffffffffa0e8d1bb&gt;] osp_sync_add+0x7b/0x80 [osp] &lt;4&gt;[ 5964.576679] [&lt;ffffffffa0e4fe20&gt;] ? lod_xattr_set+0xc0/0x7e0 [lod] &lt;4&gt;[ 5964.576876] [&lt;ffffffffa0e7d493&gt;] osp_attr_set+0x193/0x1d0 [osp] &lt;4&gt;[ 5964.577057] [&lt;ffffffffa0e4c837&gt;] lod_attr_set+0x247/0x450 [lod] &lt;4&gt;[ 5964.577241] [&lt;ffffffffa0d410d8&gt;] mdd_acl_set+0x668/0x8d0 [mdd] &lt;4&gt;[ 5964.577850] [&lt;ffffffffa0d28372&gt;] mdd_xattr_set+0x192/0xbd0 [mdd] &lt;4&gt;[ 5964.578059] [&lt;ffffffffa07f0304&gt;] ? lustre_msg_get_versions+0xa4/0x120 [ptlrpc] &lt;4&gt;[ 5964.578372] [&lt;ffffffffa0d88dca&gt;] ? mdt_version_save+0x8a/0x1a0 [mdt] &lt;4&gt;[ 5964.578561] [&lt;ffffffffa0d8ffaf&gt;] mdt_reint_setxattr+0x73f/0x1850 [mdt] &lt;4&gt;[ 5964.578763] [&lt;ffffffffa05dbc20&gt;] ? lu_ucred+0x20/0x30 [obdclass] &lt;4&gt;[ 5964.578947] [&lt;ffffffffa0d6a405&gt;] ? mdt_ucred+0x15/0x20 [mdt] &lt;4&gt;[ 5964.579127] [&lt;ffffffffa0d8457c&gt;] ? mdt_root_squash+0x2c/0x410 [mdt] &lt;4&gt;[ 5964.579317] [&lt;ffffffffa0d883b1&gt;] mdt_reint_rec+0x41/0xe0 [mdt] &lt;4&gt;[ 5964.579499] [&lt;ffffffffa0d6ed33&gt;] mdt_reint_internal+0x4c3/0x780 [mdt] &lt;4&gt;[ 5964.579687] [&lt;ffffffffa0d6f57b&gt;] mdt_reint+0x6b/0x120 [mdt] &lt;4&gt;[ 5964.579883] [&lt;ffffffffa084ebec&gt;] tgt_request_handle+0x23c/0xac0 [ptlrpc] &lt;4&gt;[ 5964.580089] [&lt;ffffffffa07ffb18&gt;] ptlrpc_main+0xcc8/0x1940 [ptlrpc] &lt;4&gt;[ 5964.580288] [&lt;ffffffffa07fee50&gt;] ? ptlrpc_main+0x0/0x1940 [ptlrpc] &lt;4&gt;[ 5964.580476] [&lt;ffffffff81094726&gt;] kthread+0x96/0xa0 &lt;4&gt;[ 5964.580657] [&lt;ffffffff8100c10a&gt;] child_rip+0xa/0x20 &lt;4&gt;[ 5964.580837] [&lt;ffffffff81094690&gt;] ? kthread+0x0/0xa0 &lt;4&gt;[ 5964.581002] [&lt;ffffffff8100c100&gt;] ? child_rip+0x0/0x20 &lt;4&gt;[ 5964.581168] &lt;0&gt;[ 5964.581889] Kernel panic - not syncing: LBUG </pre> </div></div> <p>This will likely be fixed with <a href="https://jira.whamcloud.com/browse/LU-4901" title="Remove triple buffer update from llog_osd_write_blob()" class="issue-link" data-issue-key="LU-4901"><del>LU-4901</del></a>.</p> <p>The patch is actually being developed under <a href="https://jira.whamcloud.com/browse/LU-4528" title="osd_trans_exec_op()) ASSERTION( oti-&gt;oti_declare_ops_rb[rb] &gt; 0 ) failed: rb = 0" class="issue-link" data-issue-key="LU-4528"><del>LU-4528</del></a> instead of <a href="https://jira.whamcloud.com/browse/LU-4901" title="Remove triple buffer update from llog_osd_write_blob()" class="issue-link" data-issue-key="LU-4901"><del>LU-4901</del></a>.</p> <p>Reopening to remove fix version as this is a duplicate</p> Related LU-4528 LU-4901 Development Rank 1|hzw79b: Rank (Obsolete) 11320 Severity