https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7409 Lustre <p>The typical stack trace is as follows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Call Trace: [&lt;ffffffff8156bf13&gt;] ? panic+0xac/0x179 [&lt;ffffffffa0f4e5cc&gt;] ? zio_wait+0x21c/0x3e0 [zfs] [&lt;ffffffffa0e7ef87&gt;] ? dmu_tx_dirty_buf+0x247/0x3d0 [zfs] [&lt;ffffffffa0f4e2f3&gt;] ? zio_destroy+0xb3/0x170 [zfs] [&lt;ffffffffa0e5e55f&gt;] ? dbuf_dirty+0x5f/0x16d0 [zfs] [&lt;ffffffff8157156b&gt;] ? _spin_unlock+0x2b/0x40 [&lt;ffffffffa0e848ea&gt;] ? dnode_rele+0x5a/0xa0 [zfs] [&lt;ffffffffa0e61501&gt;] ? dmu_buf_will_dirty+0x91/0x100 [zfs] [&lt;ffffffffa0e6cc70&gt;] ? dmu_write+0xa0/0x230 [zfs] [&lt;ffffffffa08444c1&gt;] ? osd_write+0x1d1/0x3a0 [osd_zfs] [&lt;ffffffffa06b9bdd&gt;] ? dt_record_write+0x3d/0x130 [obdclass] [&lt;ffffffffa067955a&gt;] ? llog_osd_write_rec+0xd6a/0x1b70 [obdclass] [&lt;ffffffffa06673f6&gt;] ? llog_write_rec+0xb6/0x270 [obdclass] [&lt;ffffffffa066c1b8&gt;] ? llog_write+0x298/0x430 [obdclass] [&lt;ffffffffa066c1cf&gt;] ? llog_write+0x2af/0x430 [obdclass] [&lt;ffffffffa14780a1&gt;] ? record_marker+0x1c1/0x1e0 [mgs] [&lt;ffffffffa14779ea&gt;] ? record_start_log+0x38a/0x4a0 [mgs] [&lt;ffffffffa14787cf&gt;] ? mgs_write_log_lov+0x38f/0x6b0 [mgs] [&lt;ffffffffa148a5c6&gt;] ? mgs_write_log_mdt+0x326/0x1630 [mgs] [&lt;ffffffff810c156d&gt;] ? trace_hardirqs_on+0xd/0x10 [&lt;ffffffffa148d475&gt;] ? mgs_write_log_target+0xb55/0x1980 [mgs] [&lt;ffffffff810c156d&gt;] ? trace_hardirqs_on+0xd/0x10 [&lt;ffffffffa057cc11&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] [&lt;ffffffffa1471d89&gt;] ? mgs_target_reg+0xa19/0xe50 [mgs] [&lt;ffffffffa0940b3f&gt;] ? tgt_request_handle+0x8cf/0x1300 [ptlrpc] [&lt;ffffffffa08eb85a&gt;] ? ptlrpc_main+0xdaa/0x18b0 [ptlrpc] [&lt;ffffffffa08eaab0&gt;] ? ptlrpc_main+0x0/0x18b0 [ptlrpc] [&lt;ffffffff810a728e&gt;] ? kthread+0x9e/0xc0 [&lt;ffffffff8100c38a&gt;] ? child_rip+0xa/0x20 [&lt;ffffffff815714b0&gt;] ? _spin_unlock_irq+0x30/0x40 [&lt;ffffffff8100bb90&gt;] ? restore_args+0x0/0x30 [&lt;ffffffff810a71f0&gt;] ? kthread+0x0/0xc0 [&lt;ffffffff8100c380&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p>Patch will be submitted shortly</p> LU-7409

llog declares write region that don't match actually write region later for osd_zfs

Bug Minor Closed Won't Fix Alex Zhuravlev Jinshan Xiong zfs Sat, 7 Nov 2015 23:04:33 +0000 Thu, 16 Apr 2020 08:04:47 +0000 Thu, 16 Apr 2020 08:04:47 +0000 0 4 <p>Jinshan Xiong (jinshan.xiong@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/17085" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17085</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7409" title="llog declares write region that don&#39;t match actually write region later for osd_zfs" class="issue-link" data-issue-key="LU-7409"><del>LDEV-239</del></a> osd-zfs: declare exact object region to write<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: d0ee25c74f33cb99a0a7c2564e6112ca267ff06e</p> <p>LLOG can't declare exact region as actual size is choosed after dmu_tx_assign() - it's append, essentially.</p> <p>Ricardo developed a patch to allow append declaration, but I have no idea where that patch is. is your zfs build configured with debug enabled?</p> <p>Yes, this is with debug enabled, and it failed at file system mount time. I saw there is a discussion about llog append therefore this patch is not a final solution, but simply an attempt to pass the check so that I can do some real test. </p> <p>well, then what's the purpose of the ticket? looks like a duplication of <a href="https://jira.whamcloud.com/browse/LU-2160" title="Implement ZFS dmu_tx_hold_append() declarations for llog " class="issue-link" data-issue-key="LU-2160">LU-2160</a> ?</p> <p>it can allow me to mount my MDS after debug enabled.</p> <p>Alex, I recall that the actual patch to the ZFS declare code is not very complex? How hard would it be to recreate that patch?</p> <p>I didn't see the patch, but in theory - yes. it should be consisting of two pieces:<br/> 1) recognize a special (like -1) offset and reserve slightly more credits like we're going to write at huge offset resulting in a deep tree<br/> 2) changes to the debugging code so that such a "undefined" write can be satisfied with that special declaration at (1)</p> <p>my original point was that we can't land this patch.</p> <p>Does declaring/holding a large range of the file cause ZFS to serialize IO to that region, or is this just accounting and serialization happens elsewhere? I'm just recalling the case of file creates where the TXG is serialized because (IIRC) you cannot modify a dnode in the same TXG as it is created in.</p> <p>declaration is just accounting and actual serializations happens against specific dbufs at actual write.</p> <p>This patch can work reasonably well on normal llog because I reserved large enough buffer for cushion, but it has problems with cat log for unlink case. Right now cat log is being used for unlink, changelog, and HSM, but I think the change to these logs are predictable at declare phase?</p> <p>nope, we can't really predict this as the transactions can take different time and while one thread has been executing a single transaction, another is able to produce many more (each adding a new record). and we may have lots of threads like these. if we try to declare large ranges, then this can result in huge credits. the credits can be multiplied by N depending on the pool's configuration. ZFS reserves memory for all the credits promised. so that a transaction going to modify few llogs may want to reserve 1,5GB memory. and again we've got to run few hundred threads usually. IOW, this approach doesn't work at scale and we need a proper support in ZFS's debugging code to understand append case. as a short term solution, we don't use debugging.</p> <p>thanks for explanation, I see it now. It looks like ZFS temporarily reserves arc buffer for the data to be written. However, in our case for append, the reserved space are counted multiple times in the same txg, is it possible for us to add some code in dmu_tx_try_assign() and count it more accurately?</p> <p>I think it's better and simpler to modify ZFS instead. especially given Brian B. wasn't against.</p> <p>we don't plan to configure ZFS with debugging enabled, so this shouldn't be an issue</p> Duplicate LU-2160 Related LU-7408 Development Rank 1|hzxsjb: Rank (Obsolete) 9223372036854775807 Severity