https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-12167 Lustre <p>This issue was created by maloo for wangshilong &lt;wshilong@ddn.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.whamcloud.com/test_sets/30cc74a4-5923-11e9-8e92-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/30cc74a4-5923-11e9-8e92-52540065bddc</a></p> <p>test_90c failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Timeout occurred after 379 mins, last suite running was conf-sanity, restarting cluster to continue tests </pre> </div></div> <p>Looks ZFS transaction blocked which hang on OST umount</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> 19270.120645] Pid: 1384, comm: ll_ost00_006 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Mon Mar 25 21:14:10 UTC 2019 [19270.122330] Call Trace: [19270.122837] [&lt;ffffffffc042f2d5&gt;] cv_wait_common+0x125/0x150 [spl] [19270.123988] [&lt;ffffffffc042f315&gt;] __cv_wait+0x15/0x20 [spl] [19270.124988] [&lt;ffffffffc05a62bf&gt;] txg_wait_synced+0xef/0x140 [zfs] [19270.126335] [&lt;ffffffffc11d39db&gt;] osd_trans_stop+0x53b/0x5e0 [osd_zfs] [19270.127563] [&lt;ffffffffc132ed45&gt;] ofd_trans_stop+0x25/0x60 [ofd] [19270.128692] [&lt;ffffffffc1333335&gt;] ofd_destroy+0x2c5/0x960 [ofd] [19270.129759] [&lt;ffffffffc132b5a4&gt;] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd] [19270.130919] [&lt;ffffffffc1321677&gt;] ofd_destroy_hdl+0x267/0x970 [ofd] [19270.132027] [&lt;ffffffffc0fed06a&gt;] tgt_request_handle+0x91a/0x15c0 [ptlrpc] [19270.133698] [&lt;ffffffffc0f906ae&gt;] ptlrpc_server_handle_request+0x24e/0xab0 [ptlrpc] [19270.135099] [&lt;ffffffffc0f941ac&gt;] ptlrpc_main+0xbbc/0x2090 [ptlrpc] [19270.136246] [&lt;ffffffff9d4c1c31&gt;] kthread+0xd1/0xe0 [19270.137169] [&lt;ffffffff9db74c37&gt;] ret_from_fork_nospec_end+0x0/0x39 [19270.138304] [&lt;ffffffffffffffff&gt;] 0xffffffffffffffff [19270.139282] Pid: 1381, comm: ll_ost00_005 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Mon Mar 25 21:14:10 UTC 2019 [19270.141084] Call Trace: [19270.141612] [&lt;ffffffffc042f2d5&gt;] cv_wait_common+0x125/0x150 [spl] [19270.143092] [&lt;ffffffffc042f315&gt;] __cv_wait+0x15/0x20 [spl] [19270.144111] [&lt;ffffffffc05a62bf&gt;] txg_wait_synced+0xef/0x140 [zfs] [19270.145275] [&lt;ffffffffc11d39db&gt;] osd_trans_stop+0x53b/0x5e0 [osd_zfs] [19270.146454] [&lt;ffffffffc132ed45&gt;] ofd_trans_stop+0x25/0x60 [ofd] [19270.147545] [&lt;ffffffffc1333335&gt;] ofd_destroy+0x2c5/0x960 [ofd] [19270.148616] [&lt;ffffffffc132b5a4&gt;] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd] [19270.149771] [&lt;ffffffffc1321677&gt;] ofd_destroy_hdl+0x267/0x970 [ofd] [19270.150910] [&lt;ffffffffc0fed06a&gt;] tgt_request_handle+0x91a/0x15c0 [ptlrpc] [19270.152179] [&lt;ffffffffc0f906ae&gt;] ptlrpc_server_handle_request+0x24e/0xab0 [ptlrpc] [19270.153593] [&lt;ffffffffc0f941ac&gt;] ptlrpc_main+0xbbc/0x2090 [ptlrpc] [19270.154744] [&lt;ffffffff9d4c1c31&gt;] kthread+0xd1/0xe0 [19270.155633] [&lt;ffffffff9db74c37&gt;] ret_from_fork_nospec_end+0x0/0x39 [19270.156767] [&lt;ffffffffffffffff&gt;] 0xffffffffffffffff [19270.157685] Pid: 1393, comm: ll_ost00_009 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Mon Mar 25 21:14:10 UTC 2019 [19270.159379] Call Trace: [19270.159829] [&lt;ffffffffc042f2d5&gt;] cv_wait_common+0x125/0x150 [spl] [19270.160932] [&lt;ffffffffc042f315&gt;] __cv_wait+0x15/0x20 [spl] [19270.161933] [&lt;ffffffffc05a62bf&gt;] txg_wait_synced+0xef/0x140 [zfs] [19270.163056] [&lt;ffffffffc11d39db&gt;] osd_trans_stop+0x53b/0x5e0 [osd_zfs] [19270.164205] [&lt;ffffffffc132ed45&gt;] ofd_trans_stop+0x25/0x60 [ofd] [19270.165278] [&lt;ffffffffc1333335&gt;] ofd_destroy+0x2c5/0x960 [ofd] [19270.166331] [&lt;ffffffffc132b5a4&gt;] ofd_destroy_by_fid+0x1f4/0x4a0 [ofd] [19270.167492] [&lt;ffffffffc1321677&gt;] ofd_destroy_hdl+0x267/0x970 [ofd] [19270.168600] [&lt;ffffffffc0fed06a&gt;] tgt_request_handle+0x91a/0x15c0 [ptlrpc] [19270.169846] [&lt;ffffffffc0f906ae&gt;] ptlrpc_server_handle_request+0x24e/0xab0 [ptlrpc] [19270.171223] [&lt;ffffffffc0f941ac&gt;] ptlrpc_main+0xbbc/0x2090 [ptlrpc] [19270.172365] [&lt;ffffffff9d4c1c31&gt;] kthread+0xd1/0xe0 [19270.173243] [&lt;ffffffff9db74c37&gt;] ret_from_fork_nospec_end+0x0/0x39 [19270.174349] [&lt;ffffffffffffffff&gt;] 0xffffffffffffffff .... </pre> </div></div> <p>VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV<br/> conf-sanity test_90c - Timeout occurred after 379 mins, last suite running was conf-sanity, restarting cluster to continue tests</p> LU-12167

conf-sanity test_90c: transaction blocked hang on OST umount

Bug Minor Open Unresolved Alex Zhuravlev Maloo Sun, 7 Apr 2019 13:15:11 +0000 Sat, 4 Jul 2020 07:51:43 +0000 0 6 <p>Check following OST backtrace.</p> <p><a href="https://testing.whamcloud.com/test_logs/3373a93e-5923-11e9-8e92-52540065bddc/show_text" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_logs/3373a93e-5923-11e9-8e92-52540065bddc/show_text</a></p> <p>Looks dmu objection allocation blocked on mutex acquired:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[22795.946424] Call Trace: [22795.946868] [&lt;ffffffff9d4d31b6&gt;] __cond_resched+0x26/0x30 [22795.947820] [&lt;ffffffff9db67eea&gt;] _cond_resched+0x3a/0x50 [22795.948754] [&lt;ffffffff9db65e92&gt;] mutex_lock+0x12/0x2f [22795.949668] [&lt;ffffffffc055d5b3&gt;] dnode_check_slots_free+0x63/0x100 [zfs] [22795.950844] [&lt;ffffffffc055e4b6&gt;] dnode_hold_impl+0x4e6/0xc40 [zfs] [22795.951938] [&lt;ffffffffc054b905&gt;] ? dmu_object_next+0x95/0x140 [zfs] [22795.953063] [&lt;ffffffffc054bb9f&gt;] dmu_object_alloc_dnsize+0x1ef/0x3e0 [zfs] [22795.954262] [&lt;ffffffffc11ddba2&gt;] __osd_object_create+0x82/0x170 [osd_zfs] [22795.955441] [&lt;ffffffffc11ddf0d&gt;] osd_mkreg+0x7d/0x210 [osd_zfs] [22795.956484] [&lt;ffffffffc11d9506&gt;] osd_create+0x316/0xaf0 [osd_zfs] [22795.957577] [&lt;ffffffffc133028c&gt;] ofd_precreate_objects+0xeec/0x1aa0 [ofd] [22795.958765] [&lt;ffffffffc13221f4&gt;] ofd_create_hdl+0x474/0x2100 [ofd] [22795.959861] [&lt;ffffffffc0c0a8d4&gt;] ? libcfs_log_return+0x24/0x30 [libcfs] [22795.961032] [&lt;ffffffffc0f84613&gt;] ? lustre_pack_reply_v2+0x183/0x280 [ptlrpc] [22795.962281] [&lt;ffffffffc0fed06a&gt;] tgt_request_handle+0x91a/0x15c0 [ptlrpc] [22795.963464] [&lt;ffffffffc0c0df17&gt;] ? libcfs_debug_msg+0x57/0x80 [libcfs] [22795.964627] [&lt;ffffffffc0f906ae&gt;] ptlrpc_server_handle_request+0x24e/0xab0 [ptlrpc] [22795.965966] [&lt;ffffffffc0f941ac&gt;] ptlrpc_main+0xbbc/0x2090 [ptlrpc] [22795.967048] [&lt;ffffffff9d4d0880&gt;] ? finish_task_switch+0x50/0x1c0 [22795.968125] [&lt;ffffffffc0f935f0&gt;] ? ptlrpc_register_service+0xfa0/0xfa0 [ptlrpc] [22795.969395] [&lt;ffffffff9d4c1c31&gt;] kthread+0xd1/0xe0 [22795.970237] [&lt;ffffffff9d4c1b60&gt;] ? insert_kthread_work+0x40/0x40 [22795.971279] [&lt;ffffffff9db74c37&gt;] ret_from_fork_nospec_begin+0x21/0x21 [22795.972400] [&lt;ffffffff9d4c1b60&gt;] ? insert_kthread_work+0x40/0x40 [22795.973453] ll_ost00_004 D ffff9f40243a30c0 0 1372 2 0x00000080 </pre> </div></div> <p>this looks like <a href="https://github.com/zfsonlinux/zfs/issues/8426" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/zfsonlinux/zfs/issues/8426</a></p> <p>So problem will be solved once ZFS is upgraded to 0.8</p> <p>Probably the same problem hit in <a href="https://testing.whamcloud.com/test_sets/6ccd175e-0a5a-11ea-8e77-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/6ccd175e-0a5a-11ea-8e77-52540065bddc</a>, but during conf-sanity test_76c and with a slightly different stack trace:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[22615.838841] Pid: 9453, comm: ll_ost00_006 3.10.0-957.27.2.el7_lustre.x86_64 #1 SMP Fri Nov 8 17:23:54 UTC 2019 [22615.838842] Call Trace: [22615.838866] [&lt;ffffffffc05272d5&gt;] cv_wait_common+0x125/0x150 [spl] [22615.838870] [&lt;ffffffffc0527315&gt;] __cv_wait+0x15/0x20 [spl] [22615.838996] [&lt;ffffffffc07d84d3&gt;] txg_wait_open+0xc3/0x110 [zfs] [22615.839014] [&lt;ffffffffc078ddca&gt;] dmu_tx_wait+0x3aa/0x3c0 [zfs] [22615.839031] [&lt;ffffffffc078de72&gt;] dmu_tx_assign+0x92/0x490 [zfs] [22615.839075] [&lt;ffffffffc137dfd9&gt;] osd_trans_start+0x199/0x440 [osd_zfs] [22615.839136] [&lt;ffffffffc14c906c&gt;] ofd_precreate_objects+0xa1c/0x1df0 [ofd] [22615.839140] [&lt;ffffffffc14bc024&gt;] ofd_create_hdl+0x474/0x2100 [ofd] [22615.839518] [&lt;ffffffffc11afeea&gt;] tgt_request_handle+0x97a/0x1620 [ptlrpc] [22615.839556] [&lt;ffffffffc1156a66&gt;] ptlrpc_server_handle_request+0x256/0xb10 [ptlrpc] [22615.839590] [&lt;ffffffffc115a59c&gt;] ptlrpc_main+0xbac/0x1540 [ptlrpc] [22615.839594] [&lt;ffffffffb7ac2e81&gt;] kthread+0xd1/0xe0 [22615.839597] [&lt;ffffffffb8177c37&gt;] ret_from_fork_nospec_end+0x0/0x39 [22615.839605] [&lt;ffffffffffffffff&gt;] 0xffffffffffffffff [22615.862159] Pid: 30965, comm: ll_ost00_004 3.10.0-957.27.2.el7_lustre.x86_64 #1 SMP Fri Nov 8 17:23:54 UTC 2019 [22615.863851] Call Trace: [22615.864279] [&lt;ffffffffc05272d5&gt;] cv_wait_common+0x125/0x150 [spl] [22615.865584] [&lt;ffffffffc0527315&gt;] __cv_wait+0x15/0x20 [spl] [22615.866771] [&lt;ffffffffc07d84d3&gt;] txg_wait_open+0xc3/0x110 [zfs] [22615.867885] [&lt;ffffffffc078ddca&gt;] dmu_tx_wait+0x3aa/0x3c0 [zfs] [22615.868958] [&lt;ffffffffc078de72&gt;] dmu_tx_assign+0x92/0x490 [zfs] [22615.870026] [&lt;ffffffffc137dfd9&gt;] osd_trans_start+0x199/0x440 [osd_zfs] [22615.871199] [&lt;ffffffffc14c906c&gt;] ofd_precreate_objects+0xa1c/0x1df0 [ofd] [22615.872413] [&lt;ffffffffc14bc024&gt;] ofd_create_hdl+0x474/0x2100 [ofd] [22615.873538] [&lt;ffffffffc11afeea&gt;] tgt_request_handle+0x97a/0x1620 [ptlrpc] [22615.874798] [&lt;ffffffffc1156a66&gt;] ptlrpc_server_handle_request+0x256/0xb10 [ptlrpc] [22615.876165] [&lt;ffffffffc115a59c&gt;] ptlrpc_main+0xbac/0x1540 [ptlrpc] [22615.877303] [&lt;ffffffffb7ac2e81&gt;] kthread+0xd1/0xe0 [22615.878183] [&lt;ffffffffb8177c37&gt;] ret_from_fork_nospec_end+0x0/0x39 [22615.879298] [&lt;ffffffffffffffff&gt;] 0xffffffffffffffff </pre> </div></div> <p>+1 on master with the same stack as Sebastien:<br/> <a href="https://testing.whamcloud.com/test_sets/ba979f34-0a79-406f-931d-202a6419d48e" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/ba979f34-0a79-406f-931d-202a6419d48e</a></p> Development Rank 1|i00elj: Rank (Obsolete) 9223372036854775807 Severity