https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-15880 Lustre <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[2893639.878072] LustreError: 11121:0:(osp_object.c:594:osp_attr_get()) crex-MDT0001-osp-MDT0000:osp_attr_get update error [0x20000000a:0x1 :0x0]: rc = -5 [2893639.895830] LustreError: 11121:0:(osp_object.c:594:osp_attr_get()) Skipped 172 previous similar messages [2893639.921219] LustreError: 11121:0:(llog_cat.c:458:llog_cat_close()) crex-MDT0001-osp-MDT0000: failure destroying log during cleanup: rc = -5 [2893639.921220] LustreError: 11121:0:(llog_cat.c:458:llog_cat_close()) Skipped 1 previous similar message [2893643.015127] LustreError: 11121:0:(lquota_entry.c:132:lqe_iter_cb()) ASSERTION( lqe-&gt;u.se.lse_pending_write == 0 ) failed: [2893643.029044] LustreError: 11121:0:(lquota_entry.c:132:lqe_iter_cb()) LBUG [2893643.038158] Pid: 11121, comm: umount 3.10.0-1160.49.1.el7_lustre.ddn16.x86_64 #1 SMP Mon Dec 20 11:42:01 PST 2021 [2893643.038159] Call Trace: [2893643.038177] [&lt;0&gt;] libcfs_call_trace+0x90/0xf0 [libcfs] [2893643.038182] [&lt;0&gt;] lbug_with_loc+0x4c/0xa0 [libcfs] [2893643.038190] [&lt;0&gt;] lqe_iter_cb+0x147/0x150 [lquota] [2893643.038196] [&lt;0&gt;] cfs_hash_for_each_tight+0x11e/0x320 [libcfs] [2893643.038201] [&lt;0&gt;] cfs_hash_for_each_safe+0x13/0x20 [libcfs] [2893643.038205] [&lt;0&gt;] lquota_site_free+0x11c/0x310 [lquota] [2893643.038209] [&lt;0&gt;] qsd_qtype_fini+0x94/0x530 [lquota] [2893643.038213] [&lt;0&gt;] qsd_fini+0xd7/0x500 [lquota] [2893643.038221] [&lt;0&gt;] osd_shutdown+0x66/0x2e0 [osd_ldiskfs] [2893643.038229] [&lt;0&gt;] osd_process_config+0x277/0x360 [osd_ldiskfs] [2893643.038238] [&lt;0&gt;] lod_process_config+0x24d/0x1540 [lod] [2893643.038247] [&lt;0&gt;] mdd_process_config+0x146/0x5f0 [mdd] [2893643.038260] [&lt;0&gt;] mdt_stack_fini+0x2c2/0xca0 [mdt] [2893643.038266] [&lt;0&gt;] mdt_device_fini+0x34b/0x930 [mdt] [2893643.038294] [&lt;0&gt;] class_cleanup+0x9b8/0xc50 [obdclass] [2893643.038306] [&lt;0&gt;] class_process_config+0x65c/0x2830 [obdclass] [2893643.038319] [&lt;0&gt;] class_manual_cleanup+0x1c6/0x710 [obdclass] [2893643.038335] [&lt;0&gt;] server_put_super+0xa35/0x1150 [obdclass] [2893643.038338] [&lt;0&gt;] generic_shutdown_super+0x6d/0x100 [2893643.038340] [&lt;0&gt;] kill_anon_super+0x12/0x20 [2893643.038352] [&lt;0&gt;] lustre_kill_super+0x32/0x50 [obdclass] [2893643.038353] [&lt;0&gt;] deactivate_locked_super+0x4e/0x70 [2893643.038355] [&lt;0&gt;] deactivate_super+0x46/0x60 [2893643.038356] [&lt;0&gt;] cleanup_mnt+0x3f/0x80 [2893643.038358] [&lt;0&gt;] __cleanup_mnt+0x12/0x20 [2893643.038360] [&lt;0&gt;] task_work_run+0xbb/0xe0 [2893643.038363] [&lt;0&gt;] do_notify_resume+0xa5/0xc0 [2893643.038365] [&lt;0&gt;] int_signal+0x12/0x17 [2893643.038379] [&lt;0&gt;] 0xfffffffffffffffe </pre> </div></div> <p>the following errors are also related to this issue</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LustreError: 25405:0:(qmt_handler.c:516:qmt_dqacq0()) $$$ Release too much! uuid </pre> </div></div> LU-15880

ASSERTION( lqe->u.se.lse_pending_write == 0 )

Bug Minor Resolved Fixed Hongchao Zhang Hongchao Zhang Sun, 22 May 2022 02:59:52 +0000 Thu, 30 Nov 2023 09:22:41 +0000 Tue, 31 Jan 2023 02:43:34 +0000 Lustre 2.16.0 0 10 <p>"Hongchao Zhang &lt;hongchao@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/47425" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/47425</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15880" title="ASSERTION( lqe-&gt;u.se.lse_pending_write == 0 )" class="issue-link" data-issue-key="LU-15880"><del>LU-15880</del></a> quota: free reserved in case of error<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 7a2cd69247cbefd31ad97b50a4652b0afe9d6bff</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/47425/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/47425/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15880" title="ASSERTION( lqe-&gt;u.se.lse_pending_write == 0 )" class="issue-link" data-issue-key="LU-15880"><del>LU-15880</del></a> quota: fix issues in reserving quota<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 40daa59ac41f450b60b42eb2bb0ff42ebd3c998b</p> <p>Landed for 2.16</p> <p>"Hongchao Zhang &lt;hongchao@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/48856" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/48856</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15880" title="ASSERTION( lqe-&gt;u.se.lse_pending_write == 0 )" class="issue-link" data-issue-key="LU-15880"><del>LU-15880</del></a> quota: fix quota by offline file<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 774e0b4d498f712ba97d09b0ef47cc0231063811</p> <p>"Hongchao Zhang &lt;hongchao@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/48981" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/48981</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15880" title="ASSERTION( lqe-&gt;u.se.lse_pending_write == 0 )" class="issue-link" data-issue-key="LU-15880"><del>LU-15880</del></a> quota: fix insane grant quota<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 1d97cca9265bfa517476c8826b385a1db4a3b188</p> <p>Reopening to track new patches</p> <p>I was thinking how to fix problems causing messsage "Release too much! ..." automatically. There is already a mechanism to release a space at QSD - if usage+qunit is &lt; granted. The problem is that QMT can't release more than stored in lqe_granted. It is not clear what to do in a such case to get correct lqe_granted. Even if the target sends it's actual usage(which is always correct as we got it from the lower level), it is not known about this target contribution in a total granted. To get 100% correct total granted we should recalculate total granted from the beginning. There is already a qmt_pool_recalc that does recalculate for all ids from the pool. We probably can change this to recalculate only certain UID. Now qsd release doesn't fill qb_usage - this also should be changed. I.e. the algorythm at QMT(qmt_dqacq0):<br/> 1. Release too much ? (qb_count &gt; slv_granted || qb_count &gt; lqe_granted)<br/> 2. If yes, update slave index with a new value(qb_usage).<br/> 3. call qmt_pool_recalc for global pool(plus pools that match slave index) for UID.</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/48981/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/48981/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15880" title="ASSERTION( lqe-&gt;u.se.lse_pending_write == 0 )" class="issue-link" data-issue-key="LU-15880"><del>LU-15880</del></a> quota: fix insane grant quota<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: a2fd4d3aee9739dcb23ac3bf46d221a978808463</p> <p>Landed for 2.16</p> <p>Just hit this assertion with 2.15.3 during MDT unmount:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Jul 05 19:37:27 fir-md1-s2 kernel: LustreError: 25778:0:(lquota_entry.c:135:lqe_iter_cb()) ASSERTION( lqe-&gt;u.se.lse_pending_write == 0 ) failed: Jul 05 19:37:27 fir-md1-s2 kernel: LustreError: 25778:0:(lquota_entry.c:135:lqe_iter_cb()) LBUG Jul 05 19:37:27 fir-md1-s2 kernel: Pid: 25778, comm: umount 3.10.0-1160.90.1.el7_lustre.pl1.x86_64 #1 SMP Tue Jun 20 15:47:49 PDT 2023 Message from syslogd@fir-md1-s2 at Jul 5 19:37:27 ... kernel:LustreError: 25778:0:(lquota_entry.c:135:lqe_iter_cb()) ASSERTION( lqe-&gt;u.se.lse_pending_write == 0 ) failed: Message from syslogd@fir-md1-s2 at Jul 5 19:37:27 ... kernel:LustreError: 25778:0:(lquota_entry.c:135:lqe_iter_cb()) LBUG Jul 05 19:37:27 fir-md1-s2 kernel: Call Trace: Jul 05 19:37:27 fir-md1-s2 kernel: [&lt;0&gt;] libcfs_call_trace+0x90/0xf0 [libcfs] Jul 05 19:37:27 fir-md1-s2 kernel: [&lt;0&gt;] lbug_with_loc+0x4c/0xa0 [libcfs] Jul 05 19:37:27 fir-md1-s2 kernel: [&lt;0&gt;] lqe_iter_cb+0x153/0x160 [lquota] Jul 05 19:37:27 fir-md1-s2 kernel: [&lt;0&gt;] cfs_hash_for_each_tight+0x11e/0x340 [libcfs] Jul 05 19:37:27 fir-md1-s2 kernel: [&lt;0&gt;] cfs_hash_for_each_safe+0x13/0x20 [libcfs] Jul 05 19:37:27 fir-md1-s2 kernel: [&lt;0&gt;] lquota_site_free+0x104/0x300 [lquota] Jul 05 19:37:27 fir-md1-s2 kernel: [&lt;0&gt;] qsd_qtype_fini+0x8c/0x510 [lquota] Jul 05 19:37:27 fir-md1-s2 kernel: [&lt;0&gt;] qsd_fini+0x130/0x500 [lquota] Jul 05 19:37:27 fir-md1-s2 kernel: [&lt;0&gt;] osd_shutdown+0x66/0x300 [osd_ldiskfs] Jul 05 19:37:27 fir-md1-s2 kernel: [&lt;0&gt;] osd_process_config+0x277/0x380 [osd_ldiskfs] Jul 05 19:37:27 fir-md1-s2 kernel: [&lt;0&gt;] lod_process_config+0x270/0x1340 [lod] Jul 05 19:37:27 fir-md1-s2 kernel: [&lt;0&gt;] mdd_process_config+0x146/0x640 [mdd] Jul 05 19:37:27 fir-md1-s2 kernel: [&lt;0&gt;] mdt_stack_fini+0x2c2/0xd10 [mdt] Jul 05 19:37:27 fir-md1-s2 kernel: [&lt;0&gt;] mdt_device_fini+0x34b/0x9a0 [mdt] Jul 05 19:37:27 fir-md1-s2 kernel: [&lt;0&gt;] class_cleanup+0xa61/0xd20 [obdclass] Jul 05 19:37:27 fir-md1-s2 kernel: [&lt;0&gt;] class_process_config+0x537/0x2670 [obdclass] Jul 05 19:37:27 fir-md1-s2 kernel: [&lt;0&gt;] class_manual_cleanup+0x1c6/0x760 [obdclass] Jul 05 19:37:27 fir-md1-s2 kernel: [&lt;0&gt;] server_put_super+0xa25/0xf80 [obdclass] Jul 05 19:37:27 fir-md1-s2 kernel: [&lt;0&gt;] generic_shutdown_super+0x6d/0x100 Jul 05 19:37:27 fir-md1-s2 kernel: [&lt;0&gt;] kill_anon_super+0x12/0x20 Jul 05 19:37:27 fir-md1-s2 kernel: [&lt;0&gt;] lustre_kill_super+0x2b/0x40 [lustre] Jul 05 19:37:27 fir-md1-s2 kernel: [&lt;0&gt;] deactivate_locked_super+0x56/0x70 Jul 05 19:37:27 fir-md1-s2 kernel: [&lt;0&gt;] deactivate_super+0x4a/0x70 Jul 05 19:37:27 fir-md1-s2 kernel: [&lt;0&gt;] cleanup_mnt+0x3f/0x90 Jul 05 19:37:27 fir-md1-s2 kernel: [&lt;0&gt;] __cleanup_mnt+0x12/0x20 Jul 05 19:37:27 fir-md1-s2 kernel: [&lt;0&gt;] task_work_run+0xbb/0xe0 Jul 05 19:37:27 fir-md1-s2 kernel: [&lt;0&gt;] do_notify_resume+0xad/0xd0 Jul 05 19:37:27 fir-md1-s2 kernel: [&lt;0&gt;] int_signal+0x12/0x17 Jul 05 19:37:27 fir-md1-s2 kernel: [&lt;0&gt;] 0xfffffffffffffffe Jul 05 19:37:27 fir-md1-s2 kernel: Kernel panic - not syncing: LBUG Message from syslogd@fir-md1-s2 at Jul 5 19:37:27 ... kernel:Kernel panic - not syncing: LBUG </pre> </div></div> <p>"Stephane Thiell &lt;sthiell@stanford.edu&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/51588" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/51588</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15880" title="ASSERTION( lqe-&gt;u.se.lse_pending_write == 0 )" class="issue-link" data-issue-key="LU-15880"><del>LU-15880</del></a> quota: fix issues in reserving quota<br/> Project: fs/lustre-release<br/> Branch: b2_15<br/> Current Patch Set: 1<br/> Commit: a1facc7c92d6fe12960d9b436f87c41234964041</p> <p>"Etienne AUJAMES &lt;eaujames@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/53290" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/53290</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15880" title="ASSERTION( lqe-&gt;u.se.lse_pending_write == 0 )" class="issue-link" data-issue-key="LU-15880"><del>LU-15880</del></a> quota: fix insane grant quota<br/> Project: fs/lustre-release<br/> Branch: b2_15<br/> Current Patch Set: 1<br/> Commit: e65090f72b716d3aff4f45892623b0f38c7baeb0</p> Duplicate LU-17125 Related LU-15927 LU-16529 Development Rank 1|i02qj3: Rank (Obsolete) 9223372036854775807 Severity