https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-13536 Lustre <p>Hi Folks,</p> <p>We recently upgraded our Lustre ZFS servers at SUT and have been experiencing an issues with the ZFS filesytem crashing. Last week we upgraded from Lustre 2.10.5 (plus a dozen patches) &amp; ZFS 0.7.9, over to Lustre 2.12.4 &amp; ZFS 0.7.13</p> <p>Now if we import and mount our main zfs/lustre filesystem, then and resume Slurm jobs and move onto starting the Slurm partitions we'll hit a kernel panic on the MDS shortly after the partitions are up:</p> <p> </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>May 8 20:12:37 warble2 kernel: VERIFY(dnode_add_ref(dn, (void *)(uintptr_t)tx-&gt;tx_txg)) failed May 8 20:12:37 warble2 kernel: PANIC at dnode.c:1635:dnode_setdirty() May 8 20:12:37 warble2 kernel: Showing stack for process 45209 May 8 20:12:37 warble2 kernel: CPU: 7 PID: 45209 Comm: mdt01_123 Tainted: P OE ------------ 3.10.0-1127.el7.x86_64 #1 May 8 20:12:37 warble2 kernel: Hardware name: Dell Inc. PowerEdge R740/0JM3W2, BIOS 2.5.4 01/13/2020 May 8 20:12:37 warble2 kernel: Call Trace: May 8 20:12:37 warble2 kernel: [&lt;ffffffff9077ff85&gt;] dump_stack+0x19/0x1b May 8 20:12:37 warble2 kernel: [&lt;ffffffffc04d4f24&gt;] spl_dumpstack+0x44/0x50 [spl] May 8 20:12:37 warble2 kernel: [&lt;ffffffffc04d4ff9&gt;] spl_panic+0xc9/0x110 [spl] May 8 20:12:37 warble2 kernel: [&lt;ffffffff900c7780&gt;] ? wake_up_atomic_t+0x30/0x30 May 8 20:12:37 warble2 kernel: [&lt;ffffffffc0c21073&gt;] ? dbuf_rele_and_unlock+0x283/0x4c0 [zfs] May 8 20:12:37 warble2 kernel: [&lt;ffffffffc04d0238&gt;] ? spl_kmem_zalloc+0xd8/0x180 [spl] May 8 20:12:37 warble2 kernel: [&lt;ffffffff90784002&gt;] ? mutex_lock+0x12/0x2f May 8 20:12:37 warble2 kernel: [&lt;ffffffffc0c31a2c&gt;] ? dmu_objset_userquota_get_ids+0x23c/0x440 [zfs] May 8 20:12:37 warble2 kernel: [&lt;ffffffffc0c40f39&gt;] dnode_setdirty+0xe9/0xf0 [zfs] May 8 20:12:37 warble2 kernel: [&lt;ffffffffc0c4120c&gt;] dnode_allocate+0x18c/0x230 [zfs] May 8 20:12:37 warble2 kernel: [&lt;ffffffffc0c2dd2b&gt;] dmu_object_alloc_dnsize+0x34b/0x3e0 [zfs] May 8 20:12:37 warble2 kernel: [&lt;ffffffffc1630032&gt;] __osd_object_create+0x82/0x170 [osd_zfs] May 8 20:12:37 warble2 kernel: [&lt;ffffffffc163027b&gt;] osd_mksym+0x6b/0x110 [osd_zfs] May 8 20:12:37 warble2 kernel: [&lt;ffffffff907850c2&gt;] ? down_write+0x12/0x3d May 8 20:12:37 warble2 kernel: [&lt;ffffffffc162b966&gt;] osd_create+0x316/0xaf0 [osd_zfs] May 8 20:12:37 warble2 kernel: [&lt;ffffffffc18ed9c5&gt;] lod_sub_create+0x1f5/0x480 [lod] May 8 20:12:37 warble2 kernel: [&lt;ffffffffc18de179&gt;] lod_create+0x69/0x340 [lod] May 8 20:12:37 warble2 kernel: [&lt;ffffffffc1622690&gt;] ? osd_trans_create+0x410/0x410 [osd_zfs] May 8 20:12:37 warble2 kernel: [&lt;ffffffffc1958173&gt;] mdd_create_object_internal+0xc3/0x300 [mdd] May 8 20:12:37 warble2 kernel: [&lt;ffffffffc194122b&gt;] mdd_create_object+0x7b/0x820 [mdd] May 8 20:12:37 warble2 kernel: [&lt;ffffffffc194b7b8&gt;] mdd_create+0xdd8/0x14a0 [mdd] May 8 20:12:37 warble2 kernel: [&lt;ffffffffc17d96d4&gt;] mdt_create+0xb54/0x1090 [mdt] May 8 20:12:37 warble2 kernel: [&lt;ffffffffc119ae94&gt;] ? lprocfs_stats_lock+0x24/0xd0 [obdclass] May 8 20:12:37 warble2 kernel: [&lt;ffffffffc17d9d7b&gt;] mdt_reint_create+0x16b/0x360 [mdt] May 8 20:12:37 warble2 kernel: [&lt;ffffffffc17dc963&gt;] mdt_reint_rec+0x83/0x210 [mdt] May 8 20:12:37 warble2 kernel: [&lt;ffffffffc17b9273&gt;] mdt_reint_internal+0x6e3/0xaf0 [mdt] May 8 20:12:37 warble2 kernel: [&lt;ffffffffc17c46e7&gt;] mdt_reint+0x67/0x140 [mdt] May 8 20:12:37 warble2 kernel: [&lt;ffffffffc14af64a&gt;] tgt_request_handle+0xada/0x1570 [ptlrpc] May 8 20:12:37 warble2 kernel: [&lt;ffffffffc1488d91&gt;] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] May 8 20:12:37 warble2 kernel: [&lt;ffffffffc07dcbde&gt;] ? ktime_get_real_seconds+0xe/0x10 [libcfs] May 8 20:12:37 warble2 kernel: [&lt;ffffffffc145447b&gt;] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] May 8 20:12:37 warble2 kernel: [&lt;ffffffffc1451295&gt;] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] May 8 20:12:37 warble2 kernel: [&lt;ffffffff900d3dc3&gt;] ? __wake_up+0x13/0x20 May 8 20:12:37 warble2 kernel: [&lt;ffffffffc1457de4&gt;] ptlrpc_main+0xb34/0x1470 [ptlrpc] May 8 20:12:37 warble2 kernel: [&lt;ffffffffc14572b0&gt;] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] May 8 20:12:37 warble2 kernel: [&lt;ffffffff900c6691&gt;] kthread+0xd1/0xe0 May 8 20:12:37 warble2 kernel: [&lt;ffffffff900c65c0&gt;] ? insert_kthread_work+0x40/0x40 May 8 20:12:37 warble2 kernel: [&lt;ffffffff90792d1d&gt;] ret_from_fork_nospec_begin+0x7/0x21 May 8 20:12:37 warble2 kernel: [&lt;ffffffff900c65c0&gt;] ? insert_kthread_work+0x40/0x40</pre> </div></div> <p>This issue has come up once last week, and twice tonight. We note there's a little bit of chatter over at: <a href="https://github.com/openzfs/zfs/issues/8705" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/openzfs/zfs/issues/8705</a> but no real feedback yet, and it's been open for some time now. Are there any recommendations from the experience of Lustre developers on how we might mitigate this particular problem?</p> <p>Right now we're cloning our server image to include ZFS 0.8.3 to see if that will help.</p> <p> </p> <p>Cheers,</p> <p>Simon</p> <p> </p> Dell R740. Centos 7.8. Kernel: 3.10.0-1127.el7.x86_64, lustre-2.12.4-1.el7.x86_64, zfs-0.7.13-1.el7.x86_64, spl-0.7.13-1.el7.x86_64 LU-13536

Lustre ZFS dnode kernel panic

Bug Minor Resolved Fixed Alex Zhuravlev SC Admin Fri, 8 May 2020 11:38:01 +0000 Tue, 20 Apr 2021 15:46:14 +0000 Mon, 21 Sep 2020 11:04:44 +0000 Lustre 2.12.4 0 9 <p>Have you tried disabling the ZFS <tt>dnodesize=auto</tt> property? That may avoid the frequent crashes while this issue is investigated. </p> <p>Alex</p> <p>Can you please advise here?</p> <p>Thanks</p> <p>Peter</p> <p>Thanks Andreas,</p> <p>We have not changed the <tt>dnodesize</tt> setting on our datasets. If we were to do so, how would we best determine the most suitable fixed value? It looks like the most common sizes for <tt>dnsize</tt> on our MDT are 512 &amp; 1K, but that's just skimming through a small selection - not the entire FS.{{}}</p> <p>The change to ZFS 0.8.3 went OK. No issues with importing and mounting ZFS &amp; Lustre. Starting the partition in Slurm this time did not cause a crash. That's not to say it was the cause, it just happened to be one loose theory based only on a few occurances.</p> <p> </p> <p>Cheers,</p> <p>Simon</p> <p>The original behavior (before dnodesize was an option) can be had with <tt>dnodesize=512</tt>, and would be the de-facto safest option to use. You could try using a static <tt>dnodesize=1024</tt> to still give you better xattr performance, possibly avoiding whatever problem that <tt>dnodesize=auto</tt> is causing, but it <em>may</em> be that <tt>=1024</tt> is itself the problem.</p> <p>It is definitely also possible that 0.8.3 has fixed issues in this area that were not backported to 0.7.13.</p> <p>I could be mistaken but wouldn't moving to ZFS 0.8.3 require also switching to RHEL 8.x servers? If so, I would caution that this is still in the early stages of support so would need some careful testing against your workloads before considering it ready for production. Work is active in this area but, at the time of writing, this is still somewhat of a WIP.</p> <p>&gt; The change to ZFS 0.8.3 went OK. No issues with importing and mounting ZFS &amp; Lustre</p> <p>Hmm I was working backwards through my email so had not seen that when I posted the above. Clearly my memory is faulty on this occasion.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=scadmin" class="user-hover" rel="scadmin">scadmin</a> just to check in this old ticket to confirm - is no news good news and the move to ZFS 0.8.x resolved this issue for you?</p> <p>Thanks guys. </p> <p>Yes, it's possible the issue has now been resolved in the later ZFS release. That problem doesn't seem to have come back to bite us yet! Let's archive this case.</p> <p>Cheers,<br/> simon</p> <p>0k - thanks!</p> <p>For reference, we could not reproduce this crash after applying these 2 patches from zfs-0.8.0-rc3 on top of zfs-0.7.13</p> <ul> <li>78e213946 Fix dnode_hold() freeing dnode behavior</li> <li>58769a4eb <font color="#1d1c1d">Don’t allow dnode allocation if dn_holds != 0</font></li> </ul> <p>Thanks for finding those patches @degremoa</p> <p>@pjones We are hitting this issue with Lustre 2.12.6 and I think there have been a few other reports. Anecdotally, I think we hit this bug when we have "badly behaving" jobs that heavily stress the MDS.</p> <p>I installed the "zfs-dkms-0.7.13.rpm" from the  Whamcloud download site along with the other Lustre required software. What do you think about Whamcloud applying the above two patches and re-rolling this rpm specifically for Lustre?</p> <p>thanks</p> <p>-k</p> Related Development Rank 1|i00zwf: Rank (Obsolete) 9223372036854775807 Severity