https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-14300 Lustre <p>sanity-sec test_18 hangs. It looks like this issue started on 15 MAY 2020 with <a href="https://testing.whamcloud.com/test_sets/8514ff93-a5ac-45f0-9584-dff4d4e42eae" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/8514ff93-a5ac-45f0-9584-dff4d4e42eae</a> for Lustre 2.13.53.163. </p> <p>Looking at the suite_log for <a href="https://testing.whamcloud.com/test_sets/6875529f-ecc2-432e-9f20-e8810a08448c" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/6875529f-ecc2-432e-9f20-e8810a08448c</a>, the last thing we see is</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>CMD: trevis-201vm1.trevis.whamcloud.com runas -u60003 -g60003 -G60003 lfs quota -q /mnt/lustre running as uid/gid/euid/egid 60003/60003/60003/60003, groups: 60003 [lfs] [quota] [-q] [/mnt/lustre] CMD: trevis-201vm2 chmod 000 /mnt/lustre/d18.sanity-sec CMD: trevis-201vm2 chmod 001 /mnt/lustre/d18.sanity-sec CMD: trevis-201vm2 chmod 002 /mnt/lustre/d18.sanity-sec CMD: trevis-201vm2 chmod 003 /mnt/lustre/d18.sanity-sec CMD: trevis-201vm2 chmod 004 /mnt/lustre/d18.sanity-sec CMD: trevis-201vm2 chmod 005 /mnt/lustre/d18.sanity-sec CMD: trevis-201vm2 chmod 006 /mnt/lustre/d18.sanity-sec </pre> </div></div> <p>Looking at the MDS (vm4) console log, we see the hang</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[32726.826140] Lustre: mdt00_003: service thread pid 819155 was inactive for 41.957 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [32726.826328] Pid: 816983, comm: mdt_rdpg00_000 4.18.0-240.1.1.el8_lustre.x86_64 #1 SMP Wed Dec 9 21:09:51 UTC 2020 [32726.829004] Lustre: Skipped 1 previous similar message [32726.830561] Call Trace TBD: [32726.831789] Pid: 819155, comm: mdt00_003 4.18.0-240.1.1.el8_lustre.x86_64 #1 SMP Wed Dec 9 21:09:51 UTC 2020 [32726.833277] Call Trace TBD: [32808.745683] INFO: task txg_quiesce:816666 blocked for more than 120 seconds. [32808.746826] Tainted: P OE --------- - - 4.18.0-240.1.1.el8_lustre.x86_64 #1 [32808.748143] "echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. [32808.749328] txg_quiesce D 0 816666 2 0x80004080 [32808.750184] Call Trace: [32808.750601] __schedule+0x2a6/0x700 [32808.751160] schedule+0x38/0xa0 [32808.751671] cv_wait_common+0xfb/0x130 [spl] [32808.752325] ? finish_wait+0x80/0x80 [32808.752958] txg_quiesce_thread+0x2ac/0x3a0 [zfs] [32808.753741] ? txg_sync_thread+0x480/0x480 [zfs] [32808.754470] ? __thread_exit+0x20/0x20 [spl] [32808.755136] thread_generic_wrapper+0x6f/0x80 [spl] [32808.755892] kthread+0x112/0x130 [32808.756397] ? kthread_flush_work_fn+0x10/0x10 [32808.757084] ret_from_fork+0x35/0x40 [32808.757657] INFO: task mdt_rdpg00_000:816983 blocked for more than 120 seconds. [32808.758764] Tainted: P OE --------- - - 4.18.0-240.1.1.el8_lustre.x86_64 #1 [32808.760077] "echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. [32808.761260] mdt_rdpg00_000 D 0 816983 2 0x80004080 [32808.762114] Call Trace: [32808.762502] __schedule+0x2a6/0x700 [32808.763046] schedule+0x38/0xa0 [32808.763536] rwsem_down_write_slowpath+0x32d/0x4e0 [32808.764303] ? dmu_tx_callback_register+0x57/0x80 [zfs] [32808.765124] qti_lqes_write_lock+0x87/0x190 [lquota] [32808.765896] qmt_dqacq0+0x4eb/0x2380 [lquota] [32808.766566] ? lqe_locate_find+0x546/0x7c0 [lquota] [32808.767316] ? qmt_dqacq+0x668/0x790 [lquota] [32808.767996] qmt_dqacq+0x668/0x790 [lquota] [32808.768669] mdt_quota_dqacq+0x59/0x120 [mdt] [32808.769391] tgt_request_handle+0xc78/0x1910 [ptlrpc] [32808.770205] ptlrpc_server_handle_request+0x31a/0xba0 [ptlrpc] [32808.771130] ptlrpc_main+0xba4/0x14a0 [ptlrpc] [32808.771910] ? __schedule+0x2ae/0x700 [32808.772528] ? ptlrpc_register_service+0xfb0/0xfb0 [ptlrpc] [32808.773381] kthread+0x112/0x130 [32808.773908] ? kthread_flush_work_fn+0x10/0x10 [32808.774601] ret_from_fork+0x35/0x40 [32808.775157] INFO: task qmt_reba_lustre:817021 blocked for more than 120 seconds. [32808.776291] Tainted: P OE --------- - - 4.18.0-240.1.1.el8_lustre.x86_64 #1 [32808.777627] "echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. [32808.778806] qmt_reba_lustre D 0 817021 2 0x80004080 [32808.779652] Call Trace: [32808.780033] __schedule+0x2a6/0x700 [32808.780586] schedule+0x38/0xa0 [32808.781080] rwsem_down_write_slowpath+0x32d/0x4e0 [32808.781827] ? cfs_hash_bd_lookup_intent+0x2a/0xc0 [libcfs] [32808.782688] qmt_reba_thread+0x6ca/0x9b0 [lquota] [32808.783407] ? qmt_glimpse_lock.isra.19+0xfb0/0xfb0 [lquota] [32808.784269] kthread+0x112/0x130 [32808.784777] ? kthread_flush_work_fn+0x10/0x10 [32808.785452] ret_from_fork+0x35/0x40 [32808.786011] INFO: task mdt00_003:819155 blocked for more than 120 seconds. [32808.787045] Tainted: P OE --------- - - 4.18.0-240.1.1.el8_lustre.x86_64 #1 [32808.788345] "echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. [32808.789522] mdt00_003 D 0 819155 2 0x80004080 [32808.790365] Call Trace: [32808.790760] __schedule+0x2a6/0x700 [32808.791290] ? __wake_up_common+0x7a/0x190 [32808.791922] schedule+0x38/0xa0 [32808.792406] io_schedule+0x12/0x40 [32808.792941] cv_wait_common+0xaf/0x130 [spl] [32808.793592] ? finish_wait+0x80/0x80 [32808.794184] txg_wait_synced_impl+0xc2/0x110 [zfs] [32808.794967] txg_wait_synced+0xc/0x40 [zfs] [32808.795619] osd_trans_stop+0x51f/0x560 [osd_zfs] [32808.796374] top_trans_stop+0x9f/0xd30 [ptlrpc] [32808.797089] ? mdd_acl_chmod+0x185/0x4b0 [mdd] [32808.797791] lod_trans_stop+0x9b/0x340 [lod] [32808.798453] ? mdd_attr_set+0x54a/0xc20 [mdd] [32808.799129] mdd_trans_stop+0x29/0x17f [mdd] [32808.799808] mdt_reint_setattr+0xc7a/0x12a0 [mdt] [32808.800565] ? nodemap_get_from_exp+0xea/0x270 [ptlrpc] [32808.801372] mdt_reint_rec+0x11f/0x250 [mdt] [32808.802044] mdt_reint_internal+0x498/0x780 [mdt] [32808.802778] mdt_reint+0x5e/0x100 [mdt] [32808.803400] tgt_request_handle+0xc78/0x1910 [ptlrpc] [32808.804212] ptlrpc_server_handle_request+0x31a/0xba0 [ptlrpc] [32808.805135] ptlrpc_main+0xba4/0x14a0 [ptlrpc] [32808.805824] ? __schedule+0x2ae/0x700 [32808.806412] ? ptlrpc_register_service+0xfb0/0xfb0 [ptlrpc] [32808.808415] kthread+0x112/0x130 [32808.808950] ? kthread_flush_work_fn+0x10/0x10 [32808.809648] ret_from_fork+0x35/0x40 </pre> </div></div> <p>Logs for more hangs are at<br/> <a href="https://testing.whamcloud.com/test_sets/a8822b08-9e99-4320-8750-fbfffad657a5" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/a8822b08-9e99-4320-8750-fbfffad657a5</a><br/> <a href="https://testing.whamcloud.com/test_sets/713a62aa-3393-4b43-8178-87398949b6c0" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/713a62aa-3393-4b43-8178-87398949b6c0</a><br/> <a href="https://testing.whamcloud.com/test_sets/a801aef1-a4d8-4470-a54e-49c953f2c246" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/a801aef1-a4d8-4470-a54e-49c953f2c246</a></p> ZFS and single MDS LU-14300

sanity-sec test 18 hangs in txg_quiesce on MDS

Bug Minor Resolved Fixed Sergey Cheremencev James Nunez Wed, 6 Jan 2021 16:39:55 +0000 Fri, 15 Oct 2021 18:10:16 +0000 Mon, 11 Oct 2021 13:32:43 +0000 Lustre 2.14.0 Lustre 2.15.0 0 6 <p>Sergey - <br/> These hangs started around the time of the OST pool quotas patch landed. Would you please review this failure and could this issue be due to that patch? Does this test need to change based on the OST pool quota patch?</p> <p>Thanks</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=sergey" class="user-hover" rel="sergey">sergey</a> how is your investigation progressing?</p> <p>It seems qmt is stuck due to problems at ZFS layer:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[25302.786844] Pid: 15606, comm: mdt_rdpg00_004 3.10.0-1127.19.1.el7.x86_64 #1 SMP Tue Aug 25 17:23:54 UTC 2020 [25302.788312] Call Trace: [25302.788742] [&lt;ffffffffc065d325&gt;] cv_wait_common+0x125/0x150 [spl] [25302.789802] [&lt;ffffffffc065d365&gt;] __cv_wait+0x15/0x20 [spl] [25302.790749] [&lt;ffffffffc07afa5b&gt;] dmu_tx_wait+0x20b/0x3b0 [zfs] [25302.791765] [&lt;ffffffffc07afc91&gt;] dmu_tx_assign+0x91/0x490 [zfs] [25302.792780] [&lt;ffffffffc11a0109&gt;] osd_trans_start+0x199/0x440 [osd_zfs] [25302.793914] [&lt;ffffffffc1159351&gt;] qmt_trans_start_with_slv+0x331/0x7f0 [lquota] [25302.795114] [&lt;ffffffffc1150c83&gt;] qmt_dqacq0+0x2d3/0x1ac0 [lquota] [25302.796090] [&lt;ffffffffc1152d77&gt;] qmt_dqacq+0x707/0x810 [lquota] [25302.797089] [&lt;ffffffffc1332fd2&gt;] mdt_quota_dqacq+0x62/0x150 [mdt] [25302.798125] [&lt;ffffffffc0fbf6ba&gt;] tgt_request_handle+0x7ea/0x1750 [ptlrpc] [25302.799602] [&lt;ffffffffc0f5f1c6&gt;] ptlrpc_server_handle_request+0x256/0xb10 [ptlrpc] [25302.800862] [&lt;ffffffffc0f63d1c&gt;] ptlrpc_main+0xb3c/0x14e0 [ptlrpc] [25302.801919] [&lt;ffffffffa94c6691&gt;] kthread+0xd1/0xe0 [25302.802755] [&lt;ffffffffa9b92d37&gt;] ret_from_fork_nospec_end+0x0/0x39 [25302.803791] [&lt;ffffffffffffffff&gt;] 0xffffffffffffffff </pre> </div></div> <p>As qmt_dqacq0 takes write semaphore some of quota threads that also need this semaphore may also hung:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[23041.077700] INFO: task qmt_reba_lustre:31100 blocked for more than 120 seconds. [23041.078865] "echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. [23041.080041] qmt_reba_lustre D ffff9606d327d230 0 31100 2 0x00000080 [23041.081222] Call Trace: [23041.081617] [&lt;ffffffffa9b85da9&gt;] schedule+0x29/0x70 [23041.082386] [&lt;ffffffffa9b874e5&gt;] rwsem_down_write_failed+0x215/0x3c0 [23041.083442] [&lt;ffffffffa9796777&gt;] call_rwsem_down_write_failed+0x17/0x30 [23041.084458] [&lt;ffffffffa9b850fd&gt;] down_write+0x2d/0x3d [23041.085322] [&lt;ffffffffc1157b24&gt;] qmt_reba_thread+0xa54/0xa70 [lquota] [23041.086332] [&lt;ffffffffc11570d0&gt;] ? qmt_glimpse_lock.isra.15+0xaa0/0xaa0 [lquota] [23041.087526] [&lt;ffffffffa94c6691&gt;] kthread+0xd1/0xe0 [23041.088287] [&lt;ffffffffa94c65c0&gt;] ? insert_kthread_work+0x40/0x40 [23041.089278] [&lt;ffffffffa9b92d37&gt;] ret_from_fork_nospec_begin+0x21/0x21 [23041.090278] [&lt;ffffffffa94c65c0&gt;] ? insert_kthread_work+0x40/0x40 </pre> </div></div> <p>There are also several hung tasks not related to quota, but related to ZFS:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[22920.963274] INFO: task txg_quiesce:30858 blocked for more than 120 seconds. [22920.964413] "echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. [22920.965608] txg_quiesce D ffff9606ffc1acc0 0 30858 2 0x00000080 [22920.966876] Call Trace: [22920.967286] [&lt;ffffffffa94d3820&gt;] ? task_rq_unlock+0x20/0x20 [22920.968202] [&lt;ffffffffa9b85da9&gt;] schedule+0x29/0x70 [22920.968981] [&lt;ffffffffc065d325&gt;] cv_wait_common+0x125/0x150 [spl] [22920.969995] [&lt;ffffffffa94c7780&gt;] ? wake_up_atomic_t+0x30/0x30 [22920.970890] [&lt;ffffffffc065d365&gt;] __cv_wait+0x15/0x20 [spl] [22920.971852] [&lt;ffffffffc081165b&gt;] txg_quiesce_thread+0x2cb/0x3c0 [zfs] [22920.972878] [&lt;ffffffffc0811390&gt;] ? txg_init+0x2b0/0x2b0 [zfs] [22920.973845] [&lt;ffffffffc0664e03&gt;] thread_generic_wrapper+0x73/0x80 [spl] [22920.974875] [&lt;ffffffffc0664d90&gt;] ? __thread_exit+0x20/0x20 [spl] [22920.975863] [&lt;ffffffffa94c6691&gt;] kthread+0xd1/0xe0 [22920.976638] [&lt;ffffffffa94c65c0&gt;] ? insert_kthread_work+0x40/0x40 [22920.977638] [&lt;ffffffffa9b92d37&gt;] ret_from_fork_nospec_begin+0x21/0x21 [22920.978633] [&lt;ffffffffa94c65c0&gt;] ? insert_kthread_work+0x40/0x40 [22920.979624] INFO: task qmt_reba_lustre:31100 blocked for more than 120 seconds.</pre> </div></div> <p>And</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[22920.993099] INFO: task mdt00_003:638 blocked for more than 120 seconds. [22920.994173] "echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. [22920.995356] mdt00_003 D ffff9606de0f9070 0 638 2 0x00000080 [22920.996534] Call Trace: [22920.996979] [&lt;ffffffffa9b85da9&gt;] schedule+0x29/0x70 [22920.997749] [&lt;ffffffffa9b838b1&gt;] schedule_timeout+0x221/0x2d0 [22920.998702] [&lt;ffffffffa94db9a2&gt;] ? default_wake_function+0x12/0x20 [22920.999675] [&lt;ffffffffa946d39e&gt;] ? kvm_clock_get_cycles+0x1e/0x20 [22921.000681] [&lt;ffffffffa9b8549d&gt;] io_schedule_timeout+0xad/0x130 [22921.001601] [&lt;ffffffffa94c7306&gt;] ? prepare_to_wait_exclusive+0x56/0x90 [22921.002668] [&lt;ffffffffa9b85538&gt;] io_schedule+0x18/0x20 [22921.003472] [&lt;ffffffffc065d2b2&gt;] cv_wait_common+0xb2/0x150 [spl] [22921.004458] [&lt;ffffffffa94c7780&gt;] ? wake_up_atomic_t+0x30/0x30 [22921.005357] [&lt;ffffffffc065d388&gt;] __cv_wait_io+0x18/0x20 [spl] [22921.006336] [&lt;ffffffffc0810e75&gt;] txg_wait_synced_impl+0xe5/0x130 [zfs] [22921.007375] [&lt;ffffffffc0810ed0&gt;] txg_wait_synced+0x10/0x50 [zfs] [22921.008375] [&lt;ffffffffc11a39bd&gt;] osd_trans_stop+0x54d/0x5f0 [osd_zfs] [22921.009430] [&lt;ffffffffc0fcf9a3&gt;] top_trans_stop+0xa3/0xbf0 [ptlrpc] [22921.010470] [&lt;ffffffffc146f7ff&gt;] ? lod_xattr_get+0x37f/0x890 [lod] [22921.011438] [&lt;ffffffffc1449afc&gt;] lod_trans_stop+0x25c/0x340 [lod] [22921.012452] [&lt;ffffffffc15071f9&gt;] ? mdd_acl_chmod+0x169/0x490 [mdd] [22921.013419] [&lt;ffffffffc150961e&gt;] mdd_trans_stop+0x2e/0x174 [mdd] [22921.014412] [&lt;ffffffffc14fd2df&gt;] mdd_attr_set+0x65f/0xc80 [mdd] [22921.015356] [&lt;ffffffffc1362436&gt;] mdt_attr_set+0x1c6/0x7b0 [mdt] [22921.016349] [&lt;ffffffffc13633aa&gt;] mdt_reint_setattr+0x70a/0xf40 [mdt] [22921.017346] [&lt;ffffffffc13514c1&gt;] ? mdt_root_squash+0x21/0x430 [mdt] [22921.018391] [&lt;ffffffffc1350d3b&gt;] ? ucred_set_audit_enabled.isra.15+0x3b/0x60 [mdt] [22921.019571] [&lt;ffffffffc13663e3&gt;] mdt_reint_rec+0x83/0x210 [mdt] [22921.020566] [&lt;ffffffffc133ea70&gt;] mdt_reint_internal+0x720/0xaf0 [mdt] [22921.021574] [&lt;ffffffffc134a607&gt;] mdt_reint+0x67/0x140 [mdt] [22921.022527] [&lt;ffffffffc0fbf6ba&gt;] tgt_request_handle+0x7ea/0x1750 [ptlrpc] [22921.023591] [&lt;ffffffffc0bccbee&gt;] ? libcfs_nid2str_r+0xfe/0x130 [lnet] [22921.024674] [&lt;ffffffffc0f5f1c6&gt;] ptlrpc_server_handle_request+0x256/0xb10 [ptlrpc] [22921.025863] [&lt;ffffffffc0f63d1c&gt;] ptlrpc_main+0xb3c/0x14e0 [ptlrpc] [22921.026879] [&lt;ffffffffa94d53fe&gt;] ? finish_task_switch+0x4e/0x1c0 [22921.027835] [&lt;ffffffffc0f631e0&gt;] ? ptlrpc_register_service+0xf90/0xf90 [ptlrpc] [22921.029014] [&lt;ffffffffa94c6691&gt;] kthread+0xd1/0xe0 [22921.029772] [&lt;ffffffffa94c65c0&gt;] ? insert_kthread_work+0x40/0x40 [22921.030761] [&lt;ffffffffa9b92d37&gt;] ret_from_fork_nospec_begin+0x21/0x21 [22921.031759] [&lt;ffffffffa94c65c0&gt;] ? insert_kthread_work+0x40/0x40 </pre> </div></div> <p>I believe the problem is in ZFS layer that becomes stuck causing other tasks to hung.<br/> I guess it is better to assign this to somebody who knows ZFS internals.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=jamesanunez" class="user-hover" rel="jamesanunez">jamesanunez</a>, just double-checking here, have you seen any instances of this failure with ldiskfs backends?</p> <p>Cory - No, we do not see this hang with ldiskfs backends. So far, we've only see this for ZFS.</p> <p>lots of threads were waiting for transaction:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [26058.883940] mdt_rdpg00_036 D ffff9606de7be2a0 0 15624 2 0x00000080 [26058.885070] Call Trace: [26058.885515] [&lt;ffffffffa9b85da9&gt;] schedule+0x29/0x70 [26058.886284] [&lt;ffffffffc065d325&gt;] cv_wait_common+0x125/0x150 [spl] [26058.887285] [&lt;ffffffffa94c7780&gt;] ? wake_up_atomic_t+0x30/0x30 [26058.888176] [&lt;ffffffffc065d365&gt;] __cv_wait+0x15/0x20 [spl] [26058.889126] [&lt;ffffffffc07afa5b&gt;] dmu_tx_wait+0x20b/0x3b0 [zfs] [26058.890036] [&lt;ffffffffa9b84022&gt;] ? mutex_lock+0x12/0x2f [26058.890920] [&lt;ffffffffc07afc91&gt;] dmu_tx_assign+0x91/0x490 [zfs] [26058.891845] [&lt;ffffffffc11a0109&gt;] osd_trans_start+0x199/0x440 [osd_zfs] [26058.892911] [&lt;ffffffffc1159351&gt;] qmt_trans_start_with_slv+0x331/0x7f0 [lquota] [26058.894032] [&lt;ffffffffc1150c83&gt;] qmt_dqacq0+0x2d3/0x1ac0 [lquota] [26058.895052] [&lt;ffffffffc115b573&gt;] ? qti_lqes_add+0x1a3/0x5c0 [lquota] [26058.896037] [&lt;ffffffffc1163854&gt;] ? qmt_pool_lqes_lookup+0x2b4/0x8f0 [lquota] [26058.897182] [&lt;ffffffffc1152d77&gt;] qmt_dqacq+0x707/0x810 [lquota] ... [26057.768904] [&lt;ffffffffc0810e75&gt;] txg_wait_synced_impl+0xe5/0x130 [zfs] [26057.769985] [&lt;ffffffffc0810ed0&gt;] txg_wait_synced+0x10/0x50 [zfs] [26057.770930] [&lt;ffffffffc11a39bd&gt;] osd_trans_stop+0x54d/0x5f0 [osd_zfs] [26057.772015] [&lt;ffffffffc0fcf9a3&gt;] top_trans_stop+0xa3/0xbf0 [ptlrpc] [26057.773004] [&lt;ffffffffc146f7ff&gt;] ? lod_xattr_get+0x37f/0x890 [lod] [26057.774018] [&lt;ffffffffc1449afc&gt;] lod_trans_stop+0x25c/0x340 [lod] [26057.774972] [&lt;ffffffffc15071f9&gt;] ? mdd_acl_chmod+0x169/0x490 [mdd] [26057.775985] [&lt;ffffffffc150961e&gt;] mdd_trans_stop+0x2e/0x174 [mdd] [26057.776919] [&lt;ffffffffc14fd2df&gt;] mdd_attr_set+0x65f/0xc80 [mdd] [26057.777886] [&lt;ffffffffc1362436&gt;] mdt_attr_set+0x1c6/0x7b0 [mdt] [26057.778810] [&lt;ffffffffc13633aa&gt;] mdt_reint_setattr+0x70a/0xf40 [mdt] [26057.779852] [&lt;ffffffffc13514c1&gt;] ? mdt_root_squash+0x21/0x430 [mdt] [26057.780825] [&lt;ffffffffc1350d3b&gt;] ? ucred_set_audit_enabled.isra.15+0x3b/0x60 [mdt] [26057.782047] [&lt;ffffffffc13663e3&gt;] mdt_reint_rec+0x83/0x210 [mdt] [26057.782974] [&lt;ffffffffc133ea70&gt;] mdt_reint_internal+0x720/0xaf0 [mdt] [26057.784028] [&lt;ffffffffc134a607&gt;] mdt_reint+0x67/0x140 [mdt] </pre> </div></div> <p>notice none of the threads above hold quota-related mutex/lock.</p> <p>but then another thread, holding a transaction, waits for quota semaphore.<br/> so this thread wss stuck and all another threads were waiting for it to complete transaction.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [26057.805469] mdt_rdpg00_002 D ffff9606f92b6a40 0 7935 2 0x00000080 [26057.806680] Call Trace: [26057.807100] [&lt;ffffffffa9b85da9&gt;] schedule+0x29/0x70 [26057.807954] [&lt;ffffffffa9b874e5&gt;] rwsem_down_write_failed+0x215/0x3c0 [26057.808966] [&lt;ffffffffa9796777&gt;] call_rwsem_down_write_failed+0x17/0x30 [26057.810072] [&lt;ffffffffc065e6e0&gt;] ? spl_kmem_alloc+0xf0/0x150 [spl] [26057.811046] [&lt;ffffffffa9b850fd&gt;] down_write+0x2d/0x3d [26057.811903] [&lt;ffffffffc115d381&gt;] qti_lqes_write_lock+0xb1/0x1b0 [lquota] [26057.813020] [&lt;ffffffffc1150c9e&gt;] qmt_dqacq0+0x2ee/0x1ac0 [lquota] [26057.813998] [&lt;ffffffffa9626702&gt;] ? kmem_cache_free+0x1e2/0x200 [26057.814980] [&lt;ffffffffc1163854&gt;] ? qmt_pool_lqes_lookup+0x2b4/0x8f0 [lquota] [26057.816096] [&lt;ffffffffc1152d77&gt;] qmt_dqacq+0x707/0x810 [lquota] [26057.817117] [&lt;ffffffffc0f52aff&gt;] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] [26057.818231] [&lt;ffffffffc1332fd2&gt;] mdt_quota_dqacq+0x62/0x150 [mdt] </pre> </div></div> <p>James Nunez (jnunez@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/41354" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/41354</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14300" title="sanity-sec test 18 hangs in txg_quiesce on MDS" class="issue-link" data-issue-key="LU-14300"><del>LU-14300</del></a> build: Revert ZFS version to 0.8.4<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: cf6215327af9f224bd4e9a6913541338a4c94de2</p> <p>while I'm not 100$ sure, I haven't seen an evidence/explanation why this is ZFS-related</p> <p>Just a note to document where we've seen this issue and for what configurations. </p> <p>We've seen this issue only on the master branch starting with Lustre 2.13.53.163 for ZFS with a single MDS (not DNE) for full and full-patchless test sessions. </p> <p>We've seen this hang for the following RHEL and ZFS versions:<br/> RHEL 7.7 and ZFS 0.8.3-1 – <a href="https://testing.whamcloud.com/test_sets/366aac3e-bef7-47f7-95de-a22b61a46a98" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/366aac3e-bef7-47f7-95de-a22b61a46a98</a><br/> RHEL 7.7 and ZFS 0.8.4-1 – <a href="https://testing.whamcloud.com/test_sessions/34fca719-e391-4700-9ade-6e1363e749f7" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sessions/34fca719-e391-4700-9ade-6e1363e749f7</a><br/> RHEL 7.8 and ZFS 0.8.3-1 - <a href="https://testing.whamcloud.com/test_sets/d6fe3f68-313f-4214-a290-28d9c39d5601" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/d6fe3f68-313f-4214-a290-28d9c39d5601</a><br/> RHEL 7.8 and ZFS 0.8.4-1 - <a href="https://testing.whamcloud.com/test_sets/02684af6-02e6-4205-a8ec-33a6bd136a17" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/02684af6-02e6-4205-a8ec-33a6bd136a17</a><br/> RHEL 7.9 and ZFS 0.8.4-1 – <a href="https://testing.whamcloud.com/test_sessions/b1e0c45d-dc4c-4b1b-b162-538293429dc0" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sessions/b1e0c45d-dc4c-4b1b-b162-538293429dc0</a><br/> RHEL 8.2 and ZFS 0.8.4-1 - <a href="https://testing.whamcloud.com/test_sets/0d5f60bc-e05b-4059-a1e2-7952b337f00e" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/0d5f60bc-e05b-4059-a1e2-7952b337f00e</a><br/> RHEL 8.3 and ZFS 2.0.0-1 – <a href="https://testing.whamcloud.com/test_sets/94c956a6-8eee-472d-8613-1324746db4a7" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/94c956a6-8eee-472d-8613-1324746db4a7</a></p> <p>Yes, Alex is right in his comment.<br/> qmt_dqacq0 holds quota mutex after starting transaction, so it doesn't wait for transaction.<br/> I went through the code related to handling this mutex but didn't find any obvious places where it can be leaved unlocked.<br/> Is it possible to gather debug logs or trigger a crash dump to continue analysis?</p> <p>James Nunez (jnunez@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/41384" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/41384</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14300" title="sanity-sec test 18 hangs in txg_quiesce on MDS" class="issue-link" data-issue-key="LU-14300"><del>LU-14300</del></a> tests: run testing on OST pools patch<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: b80e2463f6fecad2406b4f2b02f4975976c35a8e</p> <p>Sergey - I don't have a crash dump, but debug logs are collected. Each node has a debug log, look for <b>.debug.</b>; <a href="https://testing.whamcloud.com/test_sets/94c956a6-8eee-472d-8613-1324746db4a7" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/94c956a6-8eee-472d-8613-1324746db4a7</a>. Is that what you're looking for/helpful or do we need to set debug to -1?</p> <p>James Nunez (jnunez@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/41390" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/41390</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14300" title="sanity-sec test 18 hangs in txg_quiesce on MDS" class="issue-link" data-issue-key="LU-14300"><del>LU-14300</del></a> tests: increase debug level sanity-sec 18<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 28ae41581255cb439ef545456f509bf581dce966</p> <p>qti_lqes_write_lock() can be taking few locks. are those ordered somehow? otherwise, I guess, few threads can hit a deadlock.</p> <blockquote><p>qti_lqes_write_lock() can be taking few locks. are those ordered somehow? otherwise, I guess, few threads can hit a deadlock</p></blockquote> <p>No, those are not ordered. I agree this could be one of possible reasons.<br/> Let's look at logs with debug level -1 to be 100% sure in your guess.</p> <p>Sergey Cheremencev (sergey.cheremencev@hpe.com) uploaded a new patch: <a href="https://review.whamcloud.com/43325" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/43325</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14300" title="sanity-sec test 18 hangs in txg_quiesce on MDS" class="issue-link" data-issue-key="LU-14300"><del>LU-14300</del></a> test: default PQ limit for a new user<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: a5981a020b38b052018bed1447b090b206671a1a</p> <p>Sergey Cheremencev (sergey.cheremencev@hpe.com) uploaded a new patch: <a href="https://review.whamcloud.com/43326" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/43326</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14300" title="sanity-sec test 18 hangs in txg_quiesce on MDS" class="issue-link" data-issue-key="LU-14300"><del>LU-14300</del></a> quota: avoid nested lqe lookup<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 73845123b646501cd57886d6e178b65e794a0b0f</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/43326/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/43326/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14300" title="sanity-sec test 18 hangs in txg_quiesce on MDS" class="issue-link" data-issue-key="LU-14300"><del>LU-14300</del></a> quota: avoid nested lqe lookup<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 188112fc806c8c61d536ba3230b8d50f65e4f8fc</p> <p>The issue is solved by <a href="https://review.whamcloud.com/43326/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/43326/</a>.</p> <p>James, do you agree to close the issue? I guess <a href="https://review.whamcloud.com/#/c/41390/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/41390/</a> should be abandoned.</p> <p>It looks like we haven't seen this crash in a while. Thus, I'm fine with closing this ticket.</p> <p>The patch <a href="https://review.whamcloud.com/#/c/43325/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/43325/</a> has not landed. Do we want to land this patch before closing this ticket?</p> <p><a href="https://review.whamcloud.com/#/c/43325/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/43325/</a> is not needed as it has been landed as a part of <a href="https://review.whamcloud.com/43326/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/43326/</a>.</p> Related Development Rank 1|i01iq7: Rank (Obsolete) 9223372036854775807 Severity