https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6089 Lustre <p>Had this crash happen on the tip of master as of yesterday running test 132 of sanity.sh:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Lustre: 3948:0:(client.c:1942:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1420636700/re al 1420636700] req@ffff8800290ba380 x1489642625567436/t0(0) o250-&gt;MGC192.168.20.154@tcp@0@lo:26/25 lens 400/544 e 0 to 1 dl 1420636706 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 3948:0:(client.c:1942:ptlrpc_expire_one_request()) Skipped 2 previous similar messages LustreError: 39:0:(qsd_handler.c:1139:qsd_op_adjust()) ASSERTION( qqi ) failed: LustreError: 39:0:(qsd_handler.c:1139:qsd_op_adjust()) LBUG Pid: 39, comm: kswapd0 Call Trace: [&lt;ffffffffa0779895&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] [&lt;ffffffffa0779e97&gt;] lbug_with_loc+0x47/0xb0 [libcfs] [&lt;ffffffffa0f25028&gt;] qsd_op_adjust+0x478/0x580 [lquota] [&lt;ffffffffa100a597&gt;] osd_object_delete+0x217/0x2f0 [osd_ldiskfs] [&lt;ffffffffa091e0c1&gt;] lu_object_free+0x81/0x1a0 [obdclass] [&lt;ffffffffa091f167&gt;] lu_site_purge+0x2e7/0x4e0 [obdclass] [&lt;ffffffffa091f4e8&gt;] lu_cache_shrink+0x188/0x310 [obdclass] [&lt;ffffffff81138dba&gt;] shrink_slab+0x12a/0x1a0 [&lt;ffffffff8113c0da&gt;] balance_pgdat+0x59a/0x820 [&lt;ffffffff8113c494&gt;] kswapd+0x134/0x3b0 </pre> </div></div> LU-6089

qsd_handler.c:1139:qsd_op_adjust()) ASSERTION( qqi ) failed

Bug Major Resolved Cannot Reproduce WC Triage Andreas Dilger Wed, 7 Jan 2015 21:18:37 +0000 Sat, 9 Oct 2021 06:42:29 +0000 Sat, 9 Oct 2021 06:42:29 +0000 Lustre 2.7.0 0 3 <p>This has the same symptoms of <a href="https://jira.whamcloud.com/browse/LU-5331" title="qsd_handler.c:1139:qsd_op_adjust()) ASSERTION( qqi ) failed" class="issue-link" data-issue-key="LU-5331"><del>LU-5331</del></a>, but I'm running the latest master (v2_6_92_0-14-g37145b3 + <a href="http://review.whamcloud.com/12515" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12515</a>). I'll try reverting that patch to see if it fixes the problem.</p> <p>Hit a very similar crash again in osd_object_delete-&gt;qsd_op_adjust() when unmounting in sanity.sh test_65j and I am again testing <a href="http://review.whamcloud.com/12515" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12515</a> from <a href="https://jira.whamcloud.com/browse/LU-5242" title="Test hang sanity test_132, test_133: umount ost" class="issue-link" data-issue-key="LU-5242"><del>LU-5242</del></a>. It looks like I restarted another test after my previous run that passed sanity.sh but I don't recall for sure if I reverted the patch at that time. I'll need to test without this patch again.</p> <p>I reverted the 12515 patch, and while I've observed the original <a href="https://jira.whamcloud.com/browse/LU-5242" title="Test hang sanity test_132, test_133: umount ost" class="issue-link" data-issue-key="LU-5242"><del>LU-5242</del></a> problem of not being able to unmount/remount the filesystem quickly, I haven't had any crashes, when I was previously 2-of-2 in sanity.sh while the patch was applied.</p> <p>I've now hit this once while testing <a href="http://review.whamcloud.com/11258" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/11258</a> on master instead of the 12515 patch, in a thread doing memory reclaim during an unmount operation, though it wasn't a thread involved in the unmount process:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Lustre: Failing over testfs-OST0000 Lustre: server umount testfs-OST0000 complete Lustre: Failing over testfs-OST0001 general protection fault: 0000 [#1] SMP Pid: 2170, comm: java Tainted: P--------------- 2.6.32-431.29.2.el6_lustre.g36cd22b.x86_6 RIP: 0010:[&lt;ffffffffa07a4cf6&gt;] [&lt;ffffffffa07a4cf6&gt;] qsd_op_adjust+0xb6/0x580 [lq uota] Process java (pid: 2170, threadinfo ffff8800d00ce000, task ffff880037c61540) Call Trace: osd_object_delete+0x217/0x2f0 [osd_ldiskfs] lu_object_free+0x81/0x1a0 [obdclass] lu_site_purge+0x2e7/0x4e0 [obdclass] lu_cache_shrink+0x188/0x310 [obdclass] shrink_slab+0x12a/0x1a0 do_try_to_free_pages+0x3f7/0x610 try_to_free_pages+0x92/0x120 __alloc_pages_nodemask+0x47e/0x8d0 kmem_getpages+0x62/0x170 fallback_alloc+0x1ba/0x270 ____cache_alloc_node+0x99/0x160 user_path_parent+0x31/0x80 sys_renameat+0xb8/0x3a0 sys_rename+0x1b/0x20 </pre> </div></div> <p>I've also gone back and retested the latest version of 12515 and not hit this problem in sanity as I had twice before.</p> Related LU-5242 LU-5331 Development Rank 1|hzx3dr: Rank (Obsolete) 16950 Severity