https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6923 Lustre <p>I saw an iozone writing process hung at waiting transaction to open. The backtrace is as follows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[10459.403319] iozone D ffff88082f353440 0 6714 1 0x00000004 [10459.403321] ffff880085e83b28 0000000000000082 ffff88055b805180 ffff880085e83fd8 [10459.403323] 0000000000013440 0000000000013440 ffff880803713d20 ffff8807f3701368 [10459.403326] ffff8807f3701220 ffff8807f3701370 ffff8807f3701248 0000000000000000 [10459.403328] Call Trace: [10459.403330] [&lt;ffffffff81766c59&gt;] schedule+0x29/0x70 [10459.403334] [&lt;ffffffffc09b4b45&gt;] cv_wait_common+0xe5/0x120 [spl] [10459.403337] [&lt;ffffffff810b4cd0&gt;] ? prepare_to_wait_event+0x100/0x100 [10459.403341] [&lt;ffffffffc09b4b95&gt;] __cv_wait+0x15/0x20 [spl] [10459.403360] [&lt;ffffffffc0ab2f43&gt;] txg_wait_open+0x83/0xd0 [zfs] [10459.403379] [&lt;ffffffffc0a6e760&gt;] dmu_tx_wait+0x380/0x390 [zfs] [10459.403387] [&lt;ffffffff817690b2&gt;] ? mutex_lock+0x12/0x2f [10459.403406] [&lt;ffffffffc0a6e80a&gt;] dmu_tx_assign+0x9a/0x510 [zfs] [10459.403423] [&lt;ffffffffc0a5f69c&gt;] dmu_free_long_range+0x18c/0x240 [zfs] [10459.403449] [&lt;ffffffffc0ad266d&gt;] zfs_rmnode+0x5d/0x340 [zfs] [10459.403473] [&lt;ffffffffc0af2bb8&gt;] zfs_zinactive+0x168/0x180 [zfs] [10459.403494] [&lt;ffffffffc0aecc30&gt;] zfs_inactive+0x60/0x200 [zfs] [10459.403518] [&lt;ffffffffc0b034c3&gt;] zpl_evict_inode+0x43/0x60 [zfs] [10459.403521] [&lt;ffffffff811ee864&gt;] evict+0xb4/0x180 [10459.403523] [&lt;ffffffff811ef045&gt;] iput+0xf5/0x180 [10459.403525] [&lt;ffffffff811e3a83&gt;] do_unlinkat+0x193/0x2c0 [10459.403528] [&lt;ffffffff811d8cbf&gt;] ? SYSC_newstat+0x2f/0x40 [10459.403530] [&lt;ffffffff811e4b06&gt;] SyS_unlink+0x16/0x20 [10459.403532] [&lt;ffffffff8176aced&gt;] system_call_fastpath+0x1a/0x1f </pre> </div></div> <p>I will attach the backtrace of all running processes soon</p> ZFS LU-6923

writing process hung at txg_wait_open

Bug Minor Open Unresolved WC Triage Jinshan Xiong zfs Tue, 28 Jul 2015 19:44:13 +0000 Sat, 15 Dec 2018 18:07:06 +0000 0 5 <p>probably this will be fixed with <a href="http://review.whamcloud.com/#/c/15481/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/15481/</a></p> <p>Hi Alex,</p> <p>No, I was benchmarking ZFS master and there is no Lustre code involved.</p> <p>txg_sync has been running for ages with the following backtrace:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[16932.253158] txg_sync R running task 0 26338 2 0x00000000 [16932.253166] ffff8807b1c77c38 ffff8807f659a3e8 ffffc900086ad7f0 000000000ccd7dca [16932.253177] ffffffffc09adb20 ffffffffc09adb20 0000000000000000 000000000000c210 [16932.253185] 0000000000001000 0000000000000000 0000000000000000 ffff8807b1c77c68 [16932.253187] Call Trace: [16932.253190] [&lt;ffffffff811b6cf9&gt;] ? __kmalloc_node+0x1c9/0x2a0 [16932.253193] [&lt;ffffffffc09adb20&gt;] ? spl_kmem_zalloc+0xc0/0x170 [spl] [16932.253197] [&lt;ffffffffc09adb20&gt;] ? spl_kmem_zalloc+0xc0/0x170 [spl] [16932.253201] [&lt;ffffffffc09b4bd8&gt;] ? __cv_wait_io+0x18/0x20 [spl] [16932.253226] [&lt;ffffffffc0afbe83&gt;] ? zio_wait+0x123/0x210 [zfs] [16932.253240] [&lt;ffffffffc0a5d9da&gt;] ? ddt_get_dedup_stats+0x3a/0x60 [zfs] [16932.253243] [&lt;ffffffff8107c2ca&gt;] ? mod_timer+0x12a/0x1e0 [16932.253260] [&lt;ffffffffc0a875b1&gt;] ? dsl_pool_sync+0xb1/0x470 [zfs] [16932.253267] [&lt;ffffffffc09ad76a&gt;] ? spl_kmem_free+0x2a/0x40 [spl] [16932.253292] [&lt;ffffffffc0aae346&gt;] ? spa_update_dspace+0x26/0x40 [zfs] [16932.253315] [&lt;ffffffffc0aa1d32&gt;] ? spa_sync+0x3a2/0xb20 [zfs] [16932.253321] [&lt;ffffffff810b4ce2&gt;] ? autoremove_wake_function+0x12/0x40 [16932.253327] [&lt;ffffffff8101bdc9&gt;] ? read_tsc+0x9/0x20 [16932.253350] [&lt;ffffffffc0ab3d8b&gt;] ? txg_sync_thread+0x36b/0x630 [zfs] [16932.253357] [&lt;ffffffff8101c3b9&gt;] ? sched_clock+0x9/0x10 [16932.253380] [&lt;ffffffffc0ab3a20&gt;] ? txg_quiesce_thread+0x380/0x380 [zfs] [16932.253388] [&lt;ffffffffc09aff61&gt;] ? thread_generic_wrapper+0x71/0x80 [spl] [16932.253395] [&lt;ffffffffc09afef0&gt;] ? __thread_exit+0x20/0x20 [spl] [16932.253402] [&lt;ffffffff810912c2&gt;] ? kthread+0xd2/0xf0 [16932.253408] [&lt;ffffffff810911f0&gt;] ? kthread_create_on_node+0x1c0/0x1c0 [16932.253413] [&lt;ffffffff8176ac3c&gt;] ? ret_from_fork+0x7c/0xb0 [16932.253415] [&lt;ffffffff810911f0&gt;] ? kthread_create_on_node+0x1c0/0x1c0 </pre> </div></div> <p>filed a ticket at: <a href="https://github.com/zfsonlinux/zfs/issues/3645" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/zfsonlinux/zfs/issues/3645</a></p> <p>Could you supply the config (zdb -C &lt;poolname&gt;) and any recent pool events (zpool events -v)? Thanks.</p> <p>sorry, my wording wasn't good enough. see the patch, it switches Lustre to use 0.6.4.2, which:</p> <ul> <li>Fix hard lockup due to infinite loop in zfs_zget() (zfsonlinux/zfs#3349)</li> </ul> <p>Hi Alex,</p> <p>If you're referring to commit 6102d037 for issue #3349, it has already been included in my code base for testing.</p> <p>Don,</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>jxiong@titan:~/work/zfs/zfs$ sudo zdb -C zfstest MOS Configuration: version: 5000 name: 'zfstest' state: 0 txg: 101 pool_guid: 4724987114355670347 errata: 0 hostname: 'titan' vdev_children: 1 vdev_tree: type: 'root' id: 0 guid: 4724987114355670347 children[0]: type: 'disk' id: 0 guid: 4902936745656769906 path: '/dev/nvme0n1p1' whole_disk: 0 metaslab_array: 34 metaslab_shift: 29 ashift: 9 asize: 99993780224 is_log: 0 create_txg: 4 features_for_read: com.delphix:hole_birth com.delphix:embedded_data </pre> </div></div> <p>We hit similar trouble on lola as described in <a href="https://jira.whamcloud.com/browse/LU-7221" title="replay-ost-single test_3: ASSERTION( __v &gt; 0 &amp;&amp; __v &lt; ((int)0x5a5a5a5a5a5a5a5a) ) failed: value: 0" class="issue-link" data-issue-key="LU-7221"><del>LU-7221</del></a>.</p> Duplicate LU-7721 Development Rank 1|hzxj8f: Rank (Obsolete) 9223372036854775807 Severity