https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5331 Lustre <p>Had this crash happen on the tip of master as of yesterday running test 900 of sanity.sh:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>&lt;3&gt;[109552.952152] LustreError: 12574:0:(client.c:1079:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88001bd78be8 x1473227759636904/t0(0) o13-&gt;lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 &lt;3&gt;[109552.954088] LustreError: 12574:0:(client.c:1079:ptlrpc_import_delay_req()) Skipped 1032 previous similar messages &lt;4&gt;[109556.157386] Lustre: server umount lustre-MDT0000 complete &lt;0&gt;[109556.443175] LustreError: 59:0:(qsd_handler.c:1139:qsd_op_adjust()) ASSERTION( qqi ) failed: &lt;0&gt;[109556.443758] LustreError: 59:0:(qsd_handler.c:1139:qsd_op_adjust()) LBUG &lt;4&gt;[109556.444090] Pid: 59, comm: kswapd0 &lt;4&gt;[109556.444347] &lt;4&gt;[109556.444348] Call Trace: &lt;4&gt;[109556.444843] [&lt;ffffffffa0a9c8a5&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] &lt;4&gt;[109556.445208] [&lt;ffffffffa0a9cea7&gt;] lbug_with_loc+0x47/0xb0 [libcfs] &lt;4&gt;[109556.445581] [&lt;ffffffffa08d671c&gt;] qsd_op_adjust+0x4cc/0x5a0 [lquota] &lt;4&gt;[109556.445972] [&lt;ffffffff811a6c7d&gt;] ? generic_drop_inode+0x1d/0x80 &lt;4&gt;[109556.446306] [&lt;ffffffffa09ba8af&gt;] osd_object_delete+0x1ff/0x2d0 [osd_ldiskfs] &lt;4&gt;[109556.446858] [&lt;ffffffffa0c507b1&gt;] lu_object_free+0x81/0x1a0 [obdclass] &lt;4&gt;[109556.447211] [&lt;ffffffffa0ab28e2&gt;] ? cfs_hash_bd_from_key+0x42/0xd0 [libcfs] &lt;4&gt;[109556.447568] [&lt;ffffffffa0c51827&gt;] lu_site_purge+0x2c7/0x4c0 [obdclass] &lt;4&gt;[109556.447915] [&lt;ffffffffa0c51ba8&gt;] lu_cache_shrink+0x188/0x310 [obdclass] &lt;4&gt;[109556.448237] [&lt;ffffffff8113712d&gt;] shrink_slab+0x13d/0x1c0 &lt;4&gt;[109556.448547] [&lt;ffffffff8113a58a&gt;] balance_pgdat+0x5ba/0x830 &lt;4&gt;[109556.448856] [&lt;ffffffff81140676&gt;] ? set_pgdat_percpu_threshold+0xa6/0xd0 &lt;4&gt;[109556.449198] [&lt;ffffffff8113a934&gt;] kswapd+0x134/0x3b0 &lt;4&gt;[109556.449486] [&lt;ffffffff81098f90&gt;] ? autoremove_wake_function+0x0/0x40 &lt;4&gt;[109556.449810] [&lt;ffffffff8113a800&gt;] ? kswapd+0x0/0x3b0 &lt;4&gt;[109556.450097] [&lt;ffffffff81098c06&gt;] kthread+0x96/0xa0 &lt;4&gt;[109556.450423] [&lt;ffffffff8100c24a&gt;] child_rip+0xa/0x20 &lt;4&gt;[109556.450736] [&lt;ffffffff81098b70&gt;] ? kthread+0x0/0xa0 &lt;4&gt;[109556.451027] [&lt;ffffffff8100c240&gt;] ? child_rip+0x0/0x20 &lt;4&gt;[109556.451516] &lt;0&gt;[109556.452221] LustreError: 32249:0:(ofd_dev.c:2296:ofd_fini()) ASSERTION( atomic_read(&amp;d-&gt;ld_ref) == 0 ) failed: </pre> </div></div> <p>tag in my tree: master-20140710<br/> crashdump and modules: /exports/crashdumps/192.168.10.219-2014-07-10-04\:14\:17/</p> LU-5331

qsd_handler.c:1139:qsd_op_adjust()) ASSERTION( qqi ) failed

Bug Major Resolved Fixed Niu Yawei Oleg Drokin Fri, 11 Jul 2014 04:25:59 +0000 Thu, 20 Jan 2022 15:48:32 +0000 Wed, 30 Jul 2014 03:11:12 +0000 Lustre 2.6.0 Lustre 2.6.0 0 5 <p>This looks a general race of umount thread vs. slab shrink thread.</p> <p>1. memory reclaim thread (kswapd, for exmaple) calls lu_site_purge(nr) to purge some lu_objects;<br/> 2. umount process calls lu_site_purge(-1) to try to purge all lu_objects, since some objects have been removed from lu_site by the kswapd, this lu_site_purge(-1) won't process on these objects;<br/> 3. umount process continue and hit ASSERTION( atomic_read(&amp;d-&gt;ld_ref) == 0 ), because some objects are being processed by kswapd now;<br/> 4. kswapd continue and hit ASSERTION( qqi ), because the qsd_instance has been freed by umount process;</p> <p>It looks to me that ofd_stack_fini() and mdt_stack_fini() needs be improved to address such kind of race. I don't quite understand why we always called lu_purge_site(-1) twice in ofd_statck_fini() &amp; mdt_stack_fini().</p> <p>Alex/Tappro, why do we have to call lu_site_purge() twice in ofd_stack_fini() &amp; mdt_statck_fini()? Thanks.</p> <p><a href="http://review.whamcloud.com/11099" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/11099</a></p> <p>patch landed for 2.6</p> <p>The lu_site::ls_purge_mutex added by the patch may hurt osd-zfs object creation rate, see <a href="https://jira.whamcloud.com/browse/LU-5749" title="osd-zfs: object creation may serialize on lu_site::ls_purge_mutex" class="issue-link" data-issue-key="LU-5749"><del>LU-5749</del></a>.</p> Related LU-6089 LU-5749 Development Rank 1|hzwr73: Rank (Obsolete) 14873 Severity