https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-10395 Lustre <p>Just had this crash in my master-next testing:</p> <p> </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>  [271899.484182] Lustre: DEBUG MARKER: == replay-single test 26: |X| open(O_CREAT), unlink two, close one, replay, close one (test mds_cleanup_orphans) ====================================================================================================== 05:31:19 (1513333879) [271900.114927] Turning device loop0 (0x700000) read-only [271900.159562] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [271900.197289] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [271900.868045] LustreError: 29112:0:(osd_internal.h:899:osd_fid2oi()) ASSERTION( osd-&gt;od_oi_table != NULL &amp;&amp; osd-&gt;od_oi_count &gt;= 1 ) failed: [0xa:0x8:0x0] [271900.870230] LustreError: 29112:0:(osd_internal.h:899:osd_fid2oi()) LBUG [271900.870897] Pid: 29112, comm: ll_mgs_0002 [271900.871499] Call Trace: [271900.874098] [&lt;ffffffffa02927ce&gt;] libcfs_call_trace+0x4e/0x60 [libcfs] [271900.874904] [&lt;ffffffffa029285c&gt;] lbug_with_loc+0x4c/0xb0 [libcfs] [271900.875989] [&lt;ffffffffa0bd5390&gt;] __osd_oi_lookup+0x2e0/0x390 [osd_ldiskfs] [271900.876716] [&lt;ffffffffa0bd715a&gt;] osd_oi_lookup+0xca/0x190 [osd_ldiskfs] [271900.877452] [&lt;ffffffffa0bd3112&gt;] osd_fid_lookup+0x4a2/0x1b50 [osd_ldiskfs] [271900.878132] [&lt;ffffffff810e3224&gt;] ? lockdep_init_map+0xc4/0x600 [271900.902777] [&lt;ffffffffa0bd4821&gt;] osd_object_init+0x61/0x180 [osd_ldiskfs] [271900.903535] [&lt;ffffffffa03d352f&gt;] lu_object_alloc+0xdf/0x310 [obdclass] [271900.904230] [&lt;ffffffffa03d38cc&gt;] lu_object_find_at+0x16c/0x290 [obdclass] [271900.904930] [&lt;ffffffffa03d4d88&gt;] dt_locate_at+0x18/0xb0 [obdclass] [271900.905594] [&lt;ffffffffa0399140&gt;] llog_osd_open+0x4f0/0xf80 [obdclass] [271900.906616] [&lt;ffffffffa038814a&gt;] llog_open+0x13a/0x3b0 [obdclass] [271900.907360] [&lt;ffffffffa0647953&gt;] llog_origin_handle_read_header+0x1b3/0x630 [ptlrpc] [271900.908617] [&lt;ffffffffa068da13&gt;] tgt_llog_read_header+0x33/0xe0 [ptlrpc] [271900.909364] [&lt;ffffffffa069716b&gt;] tgt_request_handle+0x93b/0x13e0 [ptlrpc] [271900.910077] [&lt;ffffffffa063c091&gt;] ptlrpc_server_handle_request+0x261/0xaf0 [ptlrpc] [271900.911298] [&lt;ffffffffa063fe48&gt;] ptlrpc_main+0xa58/0x1df0 [ptlrpc] [271900.911970] [&lt;ffffffff81706467&gt;] ? _raw_spin_unlock_irq+0x27/0x50 [271900.912646] [&lt;ffffffffa063f3f0&gt;] ? ptlrpc_main+0x0/0x1df0 [ptlrpc] [271900.914668] [&lt;ffffffff810a2eba&gt;] kthread+0xea/0xf0 [271900.915359] [&lt;ffffffff810a2dd0&gt;] ? kthread+0x0/0xf0 [271900.915996] [&lt;ffffffff8170fb98&gt;] ret_from_fork+0x58/0x90 [271900.916627] [&lt;ffffffff810a2dd0&gt;] ? kthread+0x0/0xf0 [271900.917273] [271900.917861] Kernel panic - not syncing: LBUG </pre> </div></div> <p>I have a crashdump.</p> LU-10395

ASSERTION( osd->od_oi_table != NULL && osd->od_oi_count >= 1 ) failed

Bug Minor Resolved Fixed Alex Zhuravlev Oleg Drokin Fri, 15 Dec 2017 15:27:37 +0000 Sat, 11 Jul 2020 15:17:21 +0000 Thu, 16 Apr 2020 20:03:49 +0000 Lustre 2.12.0 Lustre 2.13.0 Lustre 2.14.0 Lustre 2.12.6 0 6 <p>Just had this one happen again.</p> <p>and just once more.</p> <p>On which branch? any special patch(es)?</p> <p>hit it again om the curent master-next (= current master as of the time of this comment):</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[269316.691753] Lustre: DEBUG MARKER: == replay-single test 20c: check that client eviction does not affect file content =================== 23:11:02 (1515039062) [269316.759984] Lustre: 29811:0:(genops.c:1818:obd_export_evict_by_uuid()) lustre-MDT0000: evicting fbdcce1d-2114-114c-1c77-51be5d58048e at adminstrative request [269317.806265] LustreError: 11-0: lustre-MDT0000-mdc-ffff8800ac76f800: operation ldlm_enqueue to node 0@lo failed: rc = -107 [269317.809371] LustreError: 29813:0:(file.c:4074:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5 [269319.367417] Lustre: DEBUG MARKER: == replay-single test 21: |X| open(O_CREAT), unlink touch new, replay, close (test mds_cleanup_orphans) ====================================================================================================== 23:11:04 (1515039064) [269319.575062] Turning device loop0 (0x700000) read-only [269319.600543] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [269319.615571] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [269320.012251] LustreError: 29273:0:(osd_internal.h:899:osd_fid2oi()) ASSERTION( osd-&gt;od_oi_table != NULL &amp;&amp; osd-&gt;od_oi_count &gt;= 1 ) failed: [0xa:0x2:0x0] [269320.013895] LustreError: 29273:0:(osd_internal.h:899:osd_fid2oi()) LBUG [269320.027745] Pid: 29273, comm: ll_mgs_0002 [269320.028226] Call Trace: [269320.029360] [&lt;ffffffffa026e7ce&gt;] libcfs_call_trace+0x4e/0x60 [libcfs] [269320.030022] [&lt;ffffffffa026e85c&gt;] lbug_with_loc+0x4c/0xb0 [libcfs] [269320.030589] [&lt;ffffffffa0bac790&gt;] __osd_oi_lookup+0x2e0/0x390 [osd_ldiskfs] [269320.031065] [&lt;ffffffffa0bae55a&gt;] osd_oi_lookup+0xca/0x190 [osd_ldiskfs] [269320.031590] [&lt;ffffffffa0baa422&gt;] osd_fid_lookup+0x4a2/0x1c40 [osd_ldiskfs] [269320.032062] [&lt;ffffffff810e3224&gt;] ? lockdep_init_map+0xc4/0x600 [269320.032692] [&lt;ffffffffa0babc21&gt;] osd_object_init+0x61/0x180 [osd_ldiskfs] [269320.033438] [&lt;ffffffffa03afcff&gt;] lu_object_alloc+0xdf/0x310 [obdclass] [269320.034136] [&lt;ffffffffa03b009c&gt;] lu_object_find_at+0x16c/0x290 [obdclass] [269320.034855] [&lt;ffffffffa03b1558&gt;] dt_locate_at+0x18/0xb0 [obdclass] [269320.035565] [&lt;ffffffffa0377520&gt;] llog_osd_open+0x4f0/0xf80 [obdclass] [269320.036257] [&lt;ffffffffa036414a&gt;] llog_open+0x13a/0x3b0 [obdclass] [269320.037013] [&lt;ffffffffa05f5a03&gt;] llog_origin_handle_read_header+0x1b3/0x630 [ptlrpc] [269320.038416] [&lt;ffffffffa063ba53&gt;] tgt_llog_read_header+0x33/0xe0 [ptlrpc] [269320.039153] [&lt;ffffffffa06451ab&gt;] tgt_request_handle+0x93b/0x13e0 [ptlrpc] [269320.039887] [&lt;ffffffffa05ea141&gt;] ptlrpc_server_handle_request+0x261/0xaf0 [ptlrpc] [269320.059312] [&lt;ffffffffa05edef8&gt;] ptlrpc_main+0xa58/0x1df0 [ptlrpc] [269320.059852] [&lt;ffffffff81706467&gt;] ? _raw_spin_unlock_irq+0x27/0x50 [269320.060367] [&lt;ffffffffa05ed4a0&gt;] ? ptlrpc_main+0x0/0x1df0 [ptlrpc] [269320.060842] [&lt;ffffffff810a2eba&gt;] kthread+0xea/0xf0 [269320.061311] [&lt;ffffffff810a2dd0&gt;] ? kthread+0x0/0xf0 [269320.062024] [&lt;ffffffff8170fb98&gt;] ret_from_fork+0x58/0x90 [269320.062780] [&lt;ffffffff810a2dd0&gt;] ? kthread+0x0/0xf0 [269320.063281] [269320.063683] Kernel panic - not syncing: LBUG [269320.064200] CPU: 0 PID: 29273 Comm: ll_mgs_0002 Tainted: P OE ------------ 3.10.0-debug #2 [269320.065091] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [269320.065559] ffffffffa028e212 00000000626be182 ffff8800a1bf78a0 ffffffff816fd3e4 [269320.066423] ffff8800a1bf7920 ffffffff816f8c34 ffffffff00000008 ffff8800a1bf7930 [269320.067317] ffff8800a1bf78d0 00000000626be182 00000000626be182 0000000000000001 [269320.068197] Call Trace: [269320.068621] [&lt;ffffffff816fd3e4&gt;] dump_stack+0x19/0x1b [269320.069060] [&lt;ffffffff816f8c34&gt;] panic+0xd8/0x1e7 [269320.069528] [&lt;ffffffffa026e874&gt;] lbug_with_loc+0x64/0xb0 [libcfs] [269320.070015] [&lt;ffffffffa0bac790&gt;] __osd_oi_lookup+0x2e0/0x390 [osd_ldiskfs] [269320.070547] [&lt;ffffffffa0bae55a&gt;] osd_oi_lookup+0xca/0x190 [osd_ldiskfs] [269320.071017] [&lt;ffffffffa0baa422&gt;] osd_fid_lookup+0x4a2/0x1c40 [osd_ldiskfs] [269320.071510] [&lt;ffffffff810e3224&gt;] ? lockdep_init_map+0xc4/0x600 [269320.071959] [&lt;ffffffffa0babc21&gt;] osd_object_init+0x61/0x180 [osd_ldiskfs] [269320.072477] [&lt;ffffffffa03afcff&gt;] lu_object_alloc+0xdf/0x310 [obdclass] [269320.072957] [&lt;ffffffffa03b009c&gt;] lu_object_find_at+0x16c/0x290 [obdclass] [269320.073602] [&lt;ffffffffa03b1558&gt;] dt_locate_at+0x18/0xb0 [obdclass] [269320.074419] [&lt;ffffffffa0377520&gt;] llog_osd_open+0x4f0/0xf80 [obdclass] [269320.075331] [&lt;ffffffffa036414a&gt;] llog_open+0x13a/0x3b0 [obdclass] [269320.076242] [&lt;ffffffffa05f5a03&gt;] llog_origin_handle_read_header+0x1b3/0x630 [ptlrpc] [269320.077666] [&lt;ffffffffa063ba53&gt;] tgt_llog_read_header+0x33/0xe0 [ptlrpc] [269320.078474] [&lt;ffffffffa06451ab&gt;] tgt_request_handle+0x93b/0x13e0 [ptlrpc] [269320.079298] [&lt;ffffffffa05ea141&gt;] ptlrpc_server_handle_request+0x261/0xaf0 [ptlrpc] [269320.080761] [&lt;ffffffffa05edef8&gt;] ptlrpc_main+0xa58/0x1df0 [ptlrpc] [269320.081560] [&lt;ffffffff81706467&gt;] ? _raw_spin_unlock_irq+0x27/0x50 [269320.082392] [&lt;ffffffffa05ed4a0&gt;] ? ptlrpc_register_service+0xeb0/0xeb0 [ptlrpc] [269320.083844] [&lt;ffffffff810a2eba&gt;] kthread+0xea/0xf0 [269320.084600] [&lt;ffffffff810a2dd0&gt;] ? kthread_create_on_node+0x140/0x140 [269320.085397] [&lt;ffffffff8170fb98&gt;] ret_from_fork+0x58/0x90 [269320.087581] [&lt;ffffffff810a2dd0&gt;] ? kthread_create_on_node+0x140/0x140 </pre> </div></div> <p>latest crash's crashdump is on onyx-68 in /exports/crashdumps/192.168.123.129-2018-01-03-23:11:10</p> <p>just hit this once more on current master-next</p> <p>just hit this again on master-next:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[151025.517057] Lustre: DEBUG MARKER: == sanity-pfl test complete, duration 805 sec ======================================================== 18:28:23 (1517700503) [151025.778703] Lustre: Failing over lustre-MDT0000 [151026.031080] LustreError: 13940:0:(osd_internal.h:911:osd_fid2oi()) ASSERTION( osd-&gt;od_oi_table != NULL &amp;&amp; osd-&gt;od_oi_count &gt;= 1 ) failed: [0xa:0x9:0x0] [151026.033598] LustreError: 13940:0:(osd_internal.h:911:osd_fid2oi()) LBUG [151026.034402] Pid: 13940, comm: ll_mgs_0000 [151026.035148] Call Trace: [151026.036641] [&lt;ffffffffa01d77ce&gt;] libcfs_call_trace+0x4e/0x60 [libcfs] [151026.037462] [&lt;ffffffffa01d785c&gt;] lbug_with_loc+0x4c/0xb0 [libcfs] [151026.038313] [&lt;ffffffffa0b52f80&gt;] __osd_oi_lookup+0x2e0/0x390 [osd_ldiskfs] [151026.041160] [&lt;ffffffffa0b54d5a&gt;] osd_oi_lookup+0xca/0x190 [osd_ldiskfs] [151026.041748] [&lt;ffffffffa0b50902&gt;] osd_fid_lookup+0x4a2/0x1c60 [osd_ldiskfs] [151026.042281] [&lt;ffffffff810e3224&gt;] ? lockdep_init_map+0xc4/0x600 [151026.042780] [&lt;ffffffffa0b52121&gt;] osd_object_init+0x61/0x110 [osd_ldiskfs] [151026.043366] [&lt;ffffffffa039999f&gt;] lu_object_alloc+0xdf/0x310 [obdclass] [151026.043895] [&lt;ffffffffa0399d3c&gt;] lu_object_find_at+0x16c/0x290 [obdclass] [151026.046491] [&lt;ffffffffa039b308&gt;] dt_locate_at+0x18/0xb0 [obdclass] [151026.047276] [&lt;ffffffffa0361800&gt;] llog_osd_open+0x4f0/0xf80 [obdclass] [151026.047924] [&lt;ffffffffa034e14a&gt;] llog_open+0x13a/0x3b0 [obdclass] [151026.048498] [&lt;ffffffffa061bad3&gt;] llog_origin_handle_read_header+0x1b3/0x630 [ptlrpc] [151026.049479] [&lt;ffffffffa0663263&gt;] tgt_llog_read_header+0x33/0xe0 [ptlrpc] [151026.050493] [&lt;ffffffffa066c9bb&gt;] tgt_request_handle+0x93b/0x13e0 [ptlrpc] [151026.051100] [&lt;ffffffffa0610211&gt;] ptlrpc_server_handle_request+0x261/0xaf0 [ptlrpc] [151026.052417] [&lt;ffffffffa0613fc8&gt;] ptlrpc_main+0xa58/0x1df0 [ptlrpc] [151026.053122] [&lt;ffffffff81706467&gt;] ? _raw_spin_unlock_irq+0x27/0x50 [151026.053837] [&lt;ffffffffa0613570&gt;] ? ptlrpc_main+0x0/0x1df0 [ptlrpc] [151026.054527] [&lt;ffffffff810a2eba&gt;] kthread+0xea/0xf0 [151026.055231] [&lt;ffffffff810a2dd0&gt;] ? kthread+0x0/0xf0 [151026.055902] [&lt;ffffffff8170fb98&gt;] ret_from_fork+0x58/0x90 [151026.056556] [&lt;ffffffff810a2dd0&gt;] ? kthread+0x0/0xf0 [151026.057205] [151026.057786] Kernel panic - not syncing: LBUG </pre> </div></div> <p>hit this few times locally:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> LustreError: 7249:0:(osd_internal.h:994:osd_fid2oi()) ASSERTION( osd-&gt;od_oi_table != NULL &amp;&amp; osd-&gt;od_oi_count &gt;= 1 ) lbug_with_loc+0x79/0x80 [libcfs] ? osd_oi_delete+0x32c/0x470 [osd_ldiskfs] ? osd_destroy+0x1ad/0x810 [osd_ldiskfs] ? osd_ref_del+0x1f8/0x700 [osd_ldiskfs] ? local_object_unlink+0x50d/0x1020 [obdclass] ? nodemap_cache_find_create+0x139/0x530 [ptlrpc] ? nodemap_save_config_cache+0x49/0x4d0 [ptlrpc] ? lu_object_put+0x230/0x370 [obdclass] ? nodemap_config_set_active_mgc+0xd6/0x1d0 [ptlrpc] ? mgc_process_log+0x27a8/0x27e0 [mgc] ? mgc_requeue_thread+0x278/0x800 [mgc] ? wake_up_q+0x60/0x60 ? kthread+0x100/0x140 ? mgc_process_config+0x13c0/0x13c0 [mgc] </pre> </div></div> <p>according to the dumped log it was a race between umount and mgc thread.</p> <p>this is a race: MDT umount vs mgs normal processing:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> ll_mgs_0002 D 0 10966 2 0x80000000 Call Trace: ? __schedule+0x2ad/0xb00 schedule+0x34/0x80 lbug_with_loc+0x79/0x80 [libcfs] ? __osd_oi_lookup.isra.0+0x32f/0x3d0 [osd_ldiskfs] ? osd_object_init+0x290/0x2110 [osd_ldiskfs] ? __raw_spin_lock_init+0x28/0x50 ? osd_object_alloc+0x117/0x3d0 [osd_ldiskfs] ? lu_object_start.isra.0+0x68/0x100 [obdclass] ? lu_object_find_at+0x317/0x8d0 [obdclass] ? dt_locate_at+0x13/0xa0 [obdclass] ? llog_osd_open+0x2b0/0xf90 [obdclass] ? __raw_spin_lock_init+0x28/0x50 ? llog_open+0x307/0x410 [obdclass] ? llog_origin_handle_read_header+0x178/0x500 [ptlrpc] ? tgt_llog_read_header+0x26/0xb0 [ptlrpc] ? tgt_request_handle+0x3fe/0x1770 [ptlrpc] umount I 0 11081 11080 0x00000000 Call Trace: ? __schedule+0x2ad/0xb00 schedule+0x34/0x80 ptlrpc_stop_all_threads+0x55d/0x590 [ptlrpc] ? wait_woken+0x90/0x90 ptlrpc_unregister_service+0xb6/0x1110 [ptlrpc] mgs_device_fini+0xa0/0x5c0 [mgs] class_cleanup+0x682/0xb50 [obdclass] class_process_config+0x153e/0x30f0 [obdclass] ? class_manual_cleanup+0xd1/0x670 [obdclass] ? class_manual_cleanup+0xd1/0x670 [obdclass] ? cache_alloc_debugcheck_after+0x138/0x150 ? __kmalloc+0x20c/0x2e0 class_manual_cleanup+0x197/0x670 [obdclass] server_put_super+0x1525/0x1d50 [obdclass] ? evict_inodes+0x138/0x180 generic_shutdown_super+0x5f/0xf0 </pre> </div></div> <p>Patch <a href="https://review.whamcloud.com/37615" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37615</a> should help.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LU-10395 osd: stop OI at device shutdown and not at obd_cleanup(). otherwise a race is possible: umount &lt;MDT&gt; stopping OI vs MGS accessing same OSD which results in the assertion: ASSERTION( osd-&gt;od_oi_table != NULL &amp;&amp; osd-&gt;od_oi_count &gt;= 1 ) Signed-off-by: Alex Zhuravlev &lt;bzzz@whamcloud.com&gt; Change-Id: I24fccea718f2e2663166cfb0ff26571039357535 </pre> </div></div> <p>I have got a local reproducer for this race. I've tried 37615 patch, and it helps.</p> <p>The original shutdown process frees a od_oi_table at the next trace</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 2833.653390] Call Trace: [ 2833.653395] [&lt;ffffffff816ae7c8&gt;] dump_stack+0x19/0x1b [ 2833.653398] [&lt;ffffffff8108ae58&gt;] __warn+0xd8/0x100 [ 2833.653401] [&lt;ffffffff8108af9d&gt;] warn_slowpath_null+0x1d/0x20 [ 2833.653409] [&lt;ffffffffc0e95109&gt;] osd_oi_fini+0x39/0x1f0 [osd_ldiskfs] [ 2833.653418] [&lt;ffffffffc0eb48ec&gt;] osd_scrub_cleanup+0x6c/0xb0 [osd_ldiskfs] [ 2833.653437] [&lt;ffffffffc0e79738&gt;] osd_shutdown+0x118/0x2e0 [osd_ldiskfs] [ 2833.653528] [&lt;ffffffffc0e92a37&gt;] osd_process_config+0x277/0x360 [osd_ldiskfs] [ 2833.653551] [&lt;ffffffffc10a17cd&gt;] lod_process_config+0x24d/0x15a0 [lod] [ 2833.653562] [&lt;ffffffffc04b5fd7&gt;] ? libcfs_debug_msg+0x57/0x80 [libcfs] [ 2833.653570] [&lt;ffffffffc0f500e6&gt;] mdd_process_config+0x146/0x5f0 [mdd] [ 2833.653587] [&lt;ffffffffc0fbd2e2&gt;] mdt_stack_fini+0x2c2/0xca0 [mdt] [ 2833.653596] [&lt;ffffffffc0fbe00b&gt;] mdt_device_fini+0x34b/0x930 [mdt] [ 2833.653623] [&lt;ffffffffc06519a8&gt;] class_cleanup+0x9a8/0xc40 [obdclass] [ 2833.653641] [&lt;ffffffffc06528cc&gt;] class_process_config+0x65c/0x2830 [obdclass] [ 2833.653650] [&lt;ffffffffc04b5fd7&gt;] ? libcfs_debug_msg+0x57/0x80 [libcfs] [ 2833.653667] [&lt;ffffffffc0654c66&gt;] class_manual_cleanup+0x1c6/0x710 [obdclass] [ 2833.653687] [&lt;ffffffffc0687441&gt;] server_put_super+0xa01/0x1120 [obdclass] [ 2833.653692] [&lt;ffffffff81223246&gt;] ? evict_inodes+0xd6/0x140 [ 2833.653695] [&lt;ffffffff81208085&gt;] generic_shutdown_super+0x75/0x100 [ 2833.653697] [&lt;ffffffff81208462&gt;] kill_anon_super+0x12/0x20 [ 2833.653714] [&lt;ffffffffc0657872&gt;] lustre_kill_super+0x32/0x50 [obdclass] [ 2833.653717] [&lt;ffffffff8120881e&gt;] deactivate_locked_super+0x4e/0x70 [ 2833.653719] [&lt;ffffffff81208fa6&gt;] deactivate_super+0x46/0x60 [ 2833.653722] [&lt;ffffffff8122655f&gt;] cleanup_mnt+0x3f/0x80 [ 2833.653725] [&lt;ffffffff812265f2&gt;] __cleanup_mnt+0x12/0x20 [ 2833.653727] [&lt;ffffffff810b087b&gt;] task_work_run+0xbb/0xe0 [ 2833.653730] [&lt;ffffffff8102ab52&gt;] do_notify_resume+0x92/0xb0 [ 2833.653734] [&lt;ffffffff816c1a5d&gt;] int_signal+0x12/0x17 [ 2833.653735] ---[ end trace bfc327a09eefce17 ]--- </pre> </div></div> <p>Also osd_shutdown frees quota related things, what if some quota requests are processed during shutdown? @Alex Zhuravlev, what do you think, is this case possible?</p> <p>OSD is able to work without quota, there is internal locking in the quota code to deal with possible race, see qsd_fini() and qsd_op_begin()</p> <p>Alexandr Boyko (c17825@cray.com) uploaded a new patch: <a href="https://review.whamcloud.com/37635" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37635</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10395" title="ASSERTION( osd-&gt;od_oi_table != NULL &amp;&amp; osd-&gt;od_oi_count &gt;= 1 ) failed" class="issue-link" data-issue-key="LU-10395"><del>LU-10395</del></a> tests: add test_280 sanity<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 997da4924fc37125bbf5fbae2c12c186bb05a284</p> <p>I've pushed the regression test for issue if it is OK I rebase it to the fix.</p> <p>thanks!</p> <p>Alex, I don't see how qsd_op_begin() pins qsd at memory during using. Only small checks about !=NULL and qsd_started, even qsd_stopping is missed. </p> <p>+1 on master sanity test_208:<br/> <a href="https://testing.whamcloud.com/test_sets/52f42644-0100-4d2d-bb11-794a4b7a1bf0" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/52f42644-0100-4d2d-bb11-794a4b7a1bf0</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 8058.688742] LustreError: 12938:0:(osd_internal.h:1010:osd_fid2oi()) ASSERTION( osd-&gt;od_oi_table != NULL &amp;&amp; osd-&gt;od_oi_count &gt;= 1 ) failed: [0xa:0x3:0x0] [ 8058.691039] LustreError: 12938:0:(osd_internal.h:1010:osd_fid2oi()) LBUG [ 8058.692144] Pid: 12938, comm: ll_mgs_0002 3.10.0-1062.9.1.el7_lustre.x86_64 #1 SMP Wed Feb 12 09:50:45 UTC 2020 [ 8058.693915] Call Trace: [ 8058.694388] [&lt;ffffffffc09b8f7c&gt;] libcfs_call_trace+0x8c/0xc0 [libcfs] [ 8058.695655] [&lt;ffffffffc09b902c&gt;] lbug_with_loc+0x4c/0xa0 [libcfs] [ 8058.696793] [&lt;ffffffffc11159d0&gt;] __osd_oi_lookup+0x310/0x3c0 [osd_ldiskfs] [ 8058.698145] [&lt;ffffffffc1117925&gt;] osd_oi_lookup+0x95/0x1e0 [osd_ldiskfs] [ 8058.699407] [&lt;ffffffffc1112ff5&gt;] osd_fid_lookup+0x455/0x1d60 [osd_ldiskfs] [ 8058.700594] [&lt;ffffffffc1114961&gt;] osd_object_init+0x61/0x110 [osd_ldiskfs] [ 8058.701927] [&lt;ffffffffc0bdbafb&gt;] lu_object_start.isra.31+0x8b/0x120 [obdclass] [ 8058.703596] [&lt;ffffffffc0bdfba2&gt;] lu_object_find_at+0x1b2/0x980 [obdclass] [ 8058.704808] [&lt;ffffffffc0be0fcd&gt;] dt_locate_at+0x1d/0xb0 [obdclass] [ 8058.705983] [&lt;ffffffffc0ba2c4e&gt;] llog_osd_open+0x50e/0xf30 [obdclass] [ 8058.707231] [&lt;ffffffffc0b8f08f&gt;] llog_open+0x25f/0x400 [obdclass] [ 8058.708380] [&lt;ffffffffc0edb5b6&gt;] llog_origin_handle_read_header+0x1b6/0x630 [ptlrpc] [ 8058.710027] [&lt;ffffffffc0f25ca3&gt;] tgt_llog_read_header+0x33/0xe0 [ptlrpc] [ 8058.711367] [&lt;ffffffffc0f2f68a&gt;] tgt_request_handle+0x95a/0x1610 [ptlrpc] [ 8058.712594] [&lt;ffffffffc0ed1066&gt;] ptlrpc_server_handle_request+0x256/0xb10 [ptlrpc] [ 8058.714032] [&lt;ffffffffc0ed5464&gt;] ptlrpc_main+0xbb4/0x1550 [ptlrpc] [ 8058.715207] [&lt;ffffffffa32c61f1&gt;] kthread+0xd1/0xe0 </pre> </div></div> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/37615/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37615/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10395" title="ASSERTION( osd-&gt;od_oi_table != NULL &amp;&amp; osd-&gt;od_oi_count &gt;= 1 ) failed" class="issue-link" data-issue-key="LU-10395"><del>LU-10395</del></a> osd: stop OI at device shutdown<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 2789978e1192dbf6d90399c96b5594e0dc049cd9</p> <p>Oleg Drokin (green@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/38153" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38153</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10395" title="ASSERTION( osd-&gt;od_oi_table != NULL &amp;&amp; osd-&gt;od_oi_count &gt;= 1 ) failed" class="issue-link" data-issue-key="LU-10395"><del>LU-10395</del></a> osd: stop OI at device shutdown<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: a7e345f49985b29d1d6f45a6065af56340102470</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/37635/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37635/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10395" title="ASSERTION( osd-&gt;od_oi_table != NULL &amp;&amp; osd-&gt;od_oi_count &gt;= 1 ) failed" class="issue-link" data-issue-key="LU-10395"><del>LU-10395</del></a> tests: add test_280 sanity<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: f4eeadee5ba5d4ab9d04918d8d81d18907daa831</p> <p>can we close the ticket now?</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/38153/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38153/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10395" title="ASSERTION( osd-&gt;od_oi_table != NULL &amp;&amp; osd-&gt;od_oi_count &gt;= 1 ) failed" class="issue-link" data-issue-key="LU-10395"><del>LU-10395</del></a> osd: stop OI at device shutdown<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: b27a323147d992b510fddcfbef8aaef508be7c87</p> Development Rank 1|hzzpjj: Rank (Obsolete) 9223372036854775807 Severity