https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11235 Lustre <p>This issue was created by maloo for Lai Siyao &lt;lai.siyao@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.whamcloud.com/test_sets/b3a1452a-9c82-11e8-8ee3-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/b3a1452a-9c82-11e8-8ee3-52540065bddc</a></p> <p>MDS backtrace <a href="https://testing.whamcloud.com/test_logs/b50461b8-9c82-11e8-8ee3-52540065bddc/show_text" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_logs/b50461b8-9c82-11e8-8ee3-52540065bddc/show_text</a> shows it's stuck in txg_wait_synced().</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[10285.072535] ldlm_cn00_002 D ffff9825b67adee0 0 32218 2 0x00000080 [10285.073583] Call Trace: [10285.073963] [&lt;ffffffff8b914029&gt;] schedule+0x29/0x70 [10285.074659] [&lt;ffffffffc05da205&gt;] cv_wait_common+0x125/0x150 [spl] [10285.075360] [&lt;ffffffff8b2bc610&gt;] ? wake_up_atomic_t+0x30/0x30 [10285.076201] [&lt;ffffffffc05da245&gt;] __cv_wait+0x15/0x20 [spl] [10285.076939] [&lt;ffffffffc0704bdf&gt;] txg_wait_synced+0xef/0x140 [zfs] [10285.077663] [&lt;ffffffffc1348a43&gt;] osd_sync+0xc3/0x140 [osd_zfs] [10285.078507] [&lt;ffffffffc1163a5b&gt;] tgt_sync+0x14b/0x270 [ptlrpc] [10285.079287] [&lt;ffffffffc1165f32&gt;] tgt_blocking_ast+0x302/0x630 [ptlrpc] [10285.080043] [&lt;ffffffffc10b6e5a&gt;] ldlm_cancel_callback+0x8a/0x330 [ptlrpc] [10285.080787] [&lt;ffffffffc10b7156&gt;] ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [10285.081492] [&lt;ffffffffc10dc06b&gt;] ldlm_request_cancel+0x18b/0x730 [ptlrpc] [10285.082266] [&lt;ffffffffc10df5ea&gt;] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [10285.082964] [&lt;ffffffffc10df8d8&gt;] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [10285.083688] [&lt;ffffffffc111040b&gt;] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [10285.084449] [&lt;ffffffff8b2c52ab&gt;] ? __wake_up_common+0x5b/0x90 [10285.085095] [&lt;ffffffffc1113c44&gt;] ptlrpc_main+0xb14/0x1fb0 [ptlrpc] [10285.085752] [&lt;ffffffffc1113130&gt;] ? ptlrpc_register_service+0xe90/0xe90 [ptlrpc] [10285.086464] [&lt;ffffffff8b2bb621&gt;] kthread+0xd1/0xe0 [10285.087009] [&lt;ffffffff8b2bb550&gt;] ? insert_kthread_work+0x40/0x40 [10285.087626] [&lt;ffffffff8b9205f7&gt;] ret_from_fork_nospec_begin+0x21/0x21 [10285.088259] [&lt;ffffffff8b2bb550&gt;] ? insert_kthread_work+0x40/0x40 </pre> </div></div> LU-11235

sanity: test_230d timeout on ZFS backends

Bug Minor Open Unresolved WC Triage Maloo always_except zfs Mon, 13 Aug 2018 12:33:03 +0000 Wed, 9 Dec 2020 22:12:18 +0000 0 6 <p>Alex, do you have any clue?</p> <p>Lai, in many other cases that was just a sign of large (and usually slow) I/O</p> <p>The test log shows it was stuck there over 170mins.</p> <p>Given that this test is "<tt>check migrate big directory</tt>" it seems probable that this relates to the patches themselves. The <tt>txg_wait_synced()</tt> call is the equivalent of being stuck in <tt>jbd2_journal_stop()</tt> - this is waiting for the transaction to commit, but something is keeping the transaction open for a long time. Normally ZFS will commit a TXG every 1s, so I suspect some kind of reference leak.</p> <p>I see some other stuck threads, and in particular <tt>txg_sync</tt> is waiting on IO to complete, and a second thread is also stuck in the same place (I'm not sure what the <tt>in:imjournal</tt> thread is for):</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[10284.337042] in:imjournal D ffff9825fafbdee0 0 1304 1 0x00000080 [10284.337732] Call Trace: [10284.338475] [&lt;ffffffff8b914029&gt;] schedule+0x29/0x70 [10284.338958] [&lt;ffffffff8b911999&gt;] schedule_timeout+0x239/0x2c0 [10284.341130] [&lt;ffffffff8b91353d&gt;] io_schedule_timeout+0xad/0x130 [10284.341696] [&lt;ffffffff8b9135d8&gt;] io_schedule+0x18/0x20 [10284.342205] [&lt;ffffffff8b911fc1&gt;] bit_wait_io+0x11/0x50 [10284.342703] [&lt;ffffffff8b911ae7&gt;] __wait_on_bit+0x67/0x90 [10284.343789] [&lt;ffffffff8b3936d1&gt;] wait_on_page_bit+0x81/0xa0 [10284.344939] [&lt;ffffffff8b3a4e7b&gt;] truncate_inode_pages_range+0x42b/0x700 [10284.348657] [&lt;ffffffff8b3a51bf&gt;] truncate_inode_pages_final+0x4f/0x60 [10284.349288] [&lt;ffffffffc047610f&gt;] ext4_evict_inode+0x11f/0x4c0 [ext4] [10284.349899] [&lt;ffffffff8b4387d4&gt;] evict+0xb4/0x180 [10284.350372] [&lt;ffffffff8b4390dc&gt;] iput+0xfc/0x190 [10284.350816] [&lt;ffffffff8b4339a0&gt;] __dentry_kill+0x120/0x180 [10284.351361] [&lt;ffffffff8b433aa9&gt;] dput+0xa9/0x160 [10284.351811] [&lt;ffffffff8b42db08&gt;] SYSC_renameat2+0x518/0x5a0 [10284.355455] [&lt;ffffffff8b42ea0e&gt;] SyS_renameat2+0xe/0x10 [10284.355977] [&lt;ffffffff8b42ea4e&gt;] SyS_rename+0x1e/0x20 [10285.072535] ldlm_cn00_002 D ffff9825b67adee0 0 32218 2 0x00000080 [10285.073583] Call Trace: [10285.073963] [&lt;ffffffff8b914029&gt;] schedule+0x29/0x70 [10285.074659] [&lt;ffffffffc05da205&gt;] cv_wait_common+0x125/0x150 [spl] [10285.076201] [&lt;ffffffffc05da245&gt;] __cv_wait+0x15/0x20 [spl] [10285.076939] [&lt;ffffffffc0704bdf&gt;] txg_wait_synced+0xef/0x140 [zfs] [10285.077663] [&lt;ffffffffc1348a43&gt;] osd_sync+0xc3/0x140 [osd_zfs] [10285.078507] [&lt;ffffffffc1163a5b&gt;] tgt_sync+0x14b/0x270 [ptlrpc] [10285.079287] [&lt;ffffffffc1165f32&gt;] tgt_blocking_ast+0x302/0x630 [ptlrpc] [10285.080043] [&lt;ffffffffc10b6e5a&gt;] ldlm_cancel_callback+0x8a/0x330 [ptlrpc] [10285.080787] [&lt;ffffffffc10b7156&gt;] ldlm_lock_cancel+0x56/0x1f0 [ptlrpc] [10285.081492] [&lt;ffffffffc10dc06b&gt;] ldlm_request_cancel+0x18b/0x730 [ptlrpc] [10285.082266] [&lt;ffffffffc10df5ea&gt;] ldlm_handle_cancel+0xba/0x250 [ptlrpc] [10285.082964] [&lt;ffffffffc10df8d8&gt;] ldlm_cancel_handler+0x158/0x590 [ptlrpc] [10285.083688] [&lt;ffffffffc111040b&gt;] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [10285.085095] [&lt;ffffffffc1113c44&gt;] ptlrpc_main+0xb14/0x1fb0 [ptlrpc] [10285.086464] [&lt;ffffffff8b2bb621&gt;] kthread+0xd1/0xe0 [10285.745845] txg_sync D ffff9825d9792f70 0 1592 2 0x00000080 [10285.746553] Call Trace: [10285.747373] [&lt;ffffffff8b914029&gt;] schedule+0x29/0x70 [10285.747843] [&lt;ffffffff8b911999&gt;] schedule_timeout+0x239/0x2c0 [10285.749545] [&lt;ffffffff8b91353d&gt;] io_schedule_timeout+0xad/0x130 [10285.750737] [&lt;ffffffff8b9135d8&gt;] io_schedule+0x18/0x20 [10285.751252] [&lt;ffffffffc05da192&gt;] cv_wait_common+0xb2/0x150 [spl] [10285.752390] [&lt;ffffffffc05da268&gt;] __cv_wait_io+0x18/0x20 [spl] [10285.752994] [&lt;ffffffffc075d023&gt;] zio_wait+0x113/0x1c0 [zfs] [10285.753560] [&lt;ffffffffc0711bc1&gt;] vdev_config_sync+0xf1/0x180 [zfs] [10285.754199] [&lt;ffffffffc06f1a9c&gt;] spa_sync+0xa1c/0xd90 [zfs] [10285.755372] [&lt;ffffffffc0705c41&gt;] txg_sync_thread+0x301/0x510 [zfs] [10285.756565] [&lt;ffffffffc05d5013&gt;] thread_generic_wrapper+0x73/0x80 [spl] [10285.757786] [&lt;ffffffff8b2bb621&gt;] kthread+0xd1/0xe0 </pre> </div></div> <p>There is a second <tt>txg_sync</tt> thread that doesn't appear to be doing anything, I'm not sure if that is for a different MDT zpool?:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[10285.004345] txg_sync S ffff9825e40b9fa0 0 31971 2 0x00000080 [10285.005189] Call Trace: [10285.005442] [&lt;ffffffff8b914029&gt;] schedule+0x29/0x70 [10285.005912] [&lt;ffffffff8b9118d4&gt;] schedule_timeout+0x174/0x2c0 [10285.007064] [&lt;ffffffffc05da362&gt;] __cv_timedwait_common+0xd2/0x190 [spl] [10285.008255] [&lt;ffffffffc05da476&gt;] __cv_timedwait_sig+0x16/0x20 [spl] [10285.008890] [&lt;ffffffffc0705add&gt;] txg_sync_thread+0x19d/0x510 [zfs] [10285.010105] [&lt;ffffffffc05d5013&gt;] thread_generic_wrapper+0x73/0x80 [spl] [10285.011345] [&lt;ffffffff8b2bb621&gt;] kthread+0xd1/0xe0 </pre> </div></div> <p>This could conceivably be stuck waiting for IO from the VM host to complete, I'm not sure yet, so I've resubmitted this test session to re-run to see if the problem hits again. I wouldn't want to just to just accept a 1/2 failure rate for landing, but it is possible this is a freak failure. Resubmitting the patch with several lines of <tt>Test-Parameters: mdtbackfstype=zfs ostbackfstype=zfs mdscount=2 mdtcount=2 testlist=sanity,sanity,sanity,sanity</tt> would tell us whether this was a freak accident or a regular problem.</p> <p>This may be an old issue, because the same symptom appears on 2.10 also, see LDEV-309 and <a href="https://jira.whamcloud.com/browse/LU-8601" title="sanity test_230d: Timeout on ZFS backed MDSs" class="issue-link" data-issue-key="LU-8601"><del>LU-8601</del></a>.</p> <p>I've enabled full debug and checked the logs, each time it's stuck in osd_sync() --&gt; txg_wait_synced(), and with <a href="https://jira.whamcloud.com/browse/LU-4684" title="DNE3: allow migrating DNE striped directory" class="issue-link" data-issue-key="LU-4684"><del>LU-4684</del></a> patches this happens more often is because the new dir migration implementation triggers more lock revoke which triggers Sync-on-Lock-Cancel more often. Per Alex's suggestion in <a href="https://jira.whamcloud.com/browse/LU-8601" title="sanity test_230d: Timeout on ZFS backed MDSs" class="issue-link" data-issue-key="LU-8601"><del>LU-8601</del></a>, I'd suggest to disable this test for ZFS system, otherwise the patch for <a href="https://jira.whamcloud.com/browse/LU-4684" title="DNE3: allow migrating DNE striped directory" class="issue-link" data-issue-key="LU-4684"><del>LU-4684</del></a> can't pass this test.</p> <p>Andreas, what's your opinion?</p> <p>I guess the important question is not whether the test is failing , but whether the test represents something that will happen in the real world when directory restripe is being used? If yes, then it doesn't help us to disable the test, because it just means that users will get similar hangs on their production systems. </p> <p>I agree, there are two possible issues about ZFS which needs investigation:<br/> 1. transaction in sync mode may not work.<br/> 2. txg_wait_synced() may hung if called frequently.</p> <p>I looked though the latest commits in ZFS code, bug didn't find related fix. BTW which version of ZFS is deployed in autotest?</p> <p>another theoretical possibility would be a missing I/O completion callback or some race in that area.</p> <p>&gt;  BTW which version of ZFS is deployed in autotest?</p> <p>Will be 0.7.9 IIUC</p> <p>Occurred again on Lustre b2_10 branch:<br/> <a href="https://testing.whamcloud.com/test_sets/ff7e4cd0-e698-11e8-bfe1-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/ff7e4cd0-e698-11e8-bfe1-52540065bddc</a></p> <p>+1 on master review-dne-selinux-ssk:<br/> <a href="https://testing.whamcloud.com/test_sets/dd21c09d-9d88-44e4-9e8e-2f7e93045034" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/dd21c09d-9d88-44e4-9e8e-2f7e93045034</a></p> <p>there are no errors in any logs, it just looks like it is taking a long time?</p> Duplicate LU-8601 Development Rank 1|i000l3: Rank (Obsolete) 9223372036854775807 Severity