Whamcloud Community JIRA

Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us

9.4.14

940014

05-12-2023

[LU-465] sanity.sh test_99e failed with timeout https://jira.whamcloud.com/browse/LU-465 Lustre <p>This issue was created by maloo for sarah <sarah@whamcloud.com></p> <p>This issue relates to the following test suite run: <a href="https://maloo.whamcloud.com/test_sets/6dd5828c-a049-11e0-aee5-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/6dd5828c-a049-11e0-aee5-52540025f9af</a>.</p>     * jenkins-ga832ab5-PRISTINE-2.6.18-238.12.1.el5 (i686)<br/>     * jenkins-ga832ab5-PRISTINE-2.6.18-238.12.1.el5_lustre (i686) LU-465

sanity.sh test_99e failed with timeout

Bug Minor Resolved Cannot Reproduce WC Triage Maloo Mon, 27 Jun 2011 02:20:08 +0000 Fri, 6 Sep 2019 22:44:08 +0000 Mon, 29 May 2017 02:22:16 +0000 0 2 <p>10:18:28:Lustre: DEBUG MARKER: == sanity test 99e: cvs update ======================================================================= 03:18:27 (1309083507)<br/> 10:18:29:Lustre: 8994:0:(sec.c:407:sptlrpc_req_get_ctx()) maximum lustre stack 3140<br/> 10:18:29: <span class="error">[<f916e52c>]</span> sptlrpc_req_get_ctx+0x28c/0x580 <span class="error">[ptlrpc]</span><br/> 10:18:29: <span class="error">[<f91040e2>]</span> __ptlrpc_request_bufs_pack+0x62/0x5d0 <span class="error">[ptlrpc]</span><br/> 10:18:29:low stack detected by irq handler<br/> 10:18:29: <span class="error">[<c04074c4>]</span> do_IRQ+0x87/0xc3<br/> 10:18:29: <span class="error">[<c040597a>]</span> common_interrupt+0x1a/0x20<br/> 10:18:29: <span class="error">[<c0425e94>]</span> vprintk+0x2b4/0x2e8<br/> 10:18:29: <span class="error">[<c0425ee0>]</span> printk+0x18/0x8e<br/> 10:18:29: <span class="error">[<c0440d63>]</span> __print_symbol+0x1a/0x23<br/> 10:18:29: <span class="error">[<c04258c3>]</span> release_console_sem+0x17e/0x1b8<br/> 10:18:29: <span class="error">[<f91040e2>]</span> __ptlrpc_request_bufs_pack+0x62/0x5d0 <span class="error">[ptlrpc]</span><br/> 10:18:29: <span class="error">[<f8d8e139>]</span> cfs_print_to_console+0xb9/0x100 <span class="error">[libcfs]</span><br/> 10:18:29: <span class="error">[<f8d9e4d2>]</span> libcfs_debug_vmsg2+0x562/0x9e0 <span class="error">[libcfs]</span><br/> 10:18:29: <span class="error">[<f91040e2>]</span> __ptlrpc_request_bufs_pack+0x62/0x5d0 <span class="error">[ptlrpc]</span><br/> 10:18:29: <span class="error">[<f91040e2>]</span> __ptlrpc_request_bufs_pack+0x62/0x5d0 <span class="error">[ptlrpc]</span><br/> 10:18:29: <span class="error">[<c0425ee0>]</span> printk+0x18/0x8e<br/> 10:18:29: <span class="error">[<f91040e2>]</span> __ptlrpc_request_bufs_pack+0x62/0x5d0 <span class="error">[ptlrpc]</span><br/> 10:18:29: <span class="error">[<f91040e2>]</span> __ptlrpc_request_bufs_pack+0x62/0x5d0 <span class="error">[ptlrpc]</span><br/> 10:18:29: <span class="error">[<f91040e2>]</span> __ptlrpc_request_bufs_pack+0x62/0x5d0 <span class="error">[ptlrpc]</span><br/> 10:18:30: <span class="error">[<f91040e2>]</span> __ptlrpc_request_bufs_pack+0x62/0x5d0 <span class="error">[ptlrpc]</span><br/> 10:18:30: <span class="error">[<c0405fe8>]</span> print_trace_address+0x1b/0x24<br/> 10:18:30: <span class="error">[<f91040e2>]</span> __ptlrpc_request_bufs_pack+0x62/0x5d0 <span class="error">[ptlrpc]</span><br/> 10:18:30: <span class="error">[<c040603f>]</span> dump_trace+0x4e/0x96<br/> 10:18:30: <span class="error">[<c0406097>]</span> show_trace_log_lvl+0x10/0x20<br/> 10:18:30: <span class="error">[<c040656f>]</span> show_trace+0xa/0xc<br/> 10:18:30: <span class="error">[<c040666c>]</span> dump_stack+0x15/0x17<br/> 10:18:30: <span class="error">[<f916e52c>]</span> sptlrpc_req_get_ctx+0x28c/0x580 <span class="error">[ptlrpc]</span><br/> 10:18:30: <span class="error">[<f91040e2>]</span> __ptlrpc_request_bufs_pack+0x62/0x5d0 <span class="error">[ptlrpc]</span><br/> 10:18:30: <span class="error">[<f91688c3>]</span> req_capsule_init+0x43/0xa0 <span class="error">[ptlrpc]</span><br/> 10:18:30: <span class="error">[<f90feedc>]</span> ptlrpc_prep_req_from_pool+0x1c/0xe0 <span class="error">[ptlrpc]</span><br/> 10:18:30: <span class="error">[<f9168831>]</span> req_capsule_set+0x21/0x70 <span class="error">[ptlrpc]</span><br/> 10:18:30: <span class="error">[<f9168e9b>]</span> req_capsule_set_size+0x5b/0x2a0 <span class="error">[ptlrpc]</span><br/> 10:18:30: <span class="error">[<f9103b1a>]</span> ptlrpc_request_alloc_internal+0x8a/0x510 <span class="error">[ptlrpc]</span><br/> 10:18:30: <span class="error">[<f9104c0a>]</span> ptlrpc_request_bufs_pack+0x4a/0x60 <span class="error">[ptlrpc]</span><br/> 10:18:30: <span class="error">[<f9104c37>]</span> ptlrpc_request_pack+0x17/0x20 <span class="error">[ptlrpc]</span><br/> 10:18:30: <span class="error">[<f94ced64>]</span> osc_brw_prep_request+0x274/0x1d90 <span class="error">[osc]</span><br/> 10:18:30: <span class="error">[<f9af9c0a>]</span> cl_lock_at_page+0x2ba/0x2d0 <span class="error">[obdclass]</span><br/> 10:18:30: <span class="error">[<f9b08add>]</span> cl_req_slice_add+0x5d/0x230 <span class="error">[obdclass]</span><br/> 10:18:30: <span class="error">[<f9505bc0>]</span> osc_req_attr_set+0x0/0x3d0 <span class="error">[osc]</span><br/> 10:18:31: <span class="error">[<f9505ca7>]</span> osc_req_attr_set+0xe7/0x3d0 <span class="error">[osc]</span><br/> 10:18:31: <span class="error">[<f934b315>]</span> ccc_req_attr_set+0x75/0x150 <span class="error">[lustre]</span><br/> 10:18:31: <span class="error">[<f9505bc0>]</span> osc_req_attr_set+0x0/0x3d0 <span class="error">[osc]</span><br/> 10:18:31: <span class="error">[<f9b07de2>]</span> cl_req_page_add+0x82/0x2d0 <span class="error">[obdclass]</span><br/> 10:18:31: <span class="error">[<f94da990>]</span> osc_send_oap_rpc+0x1480/0x2d10 <span class="error">[osc]</span><br/> 10:18:31: <span class="error">[<f92e1388>]</span> vvp_write_pending+0x58/0x230 <span class="error">[lustre]</span><br/> 10:18:31: <span class="error">[<f94dc3c0>]</span> osc_check_rpcs+0x1a0/0x690 <span class="error">[osc]</span><br/> 10:18:31: <span class="error">[<f9b01cc9>]</span> cl_page_list_move+0x69/0x240 <span class="error">[obdclass]</span><br/> 10:18:31: <span class="error">[<c04f1ff7>]</span> vsnprintf+0x2e7/0x4db<br/> 10:18:31: <span class="error">[<f94d8636>]</span> loi_list_maint+0x66/0xe0 <span class="error">[osc]</span><br/> 10:18:31: <span class="error">[<f950622c>]</span> osc_io_submit+0x29c/0x590 <span class="error">[osc]</span><br/> 10:18:31: <span class="error">[<f926ef02>]</span> lov_page_stripe+0x52/0x250 <span class="error">[lov]</span><br/> 10:18:31: <span class="error">[<f9505f90>]</span> osc_io_submit+0x0/0x590 <span class="error">[osc]</span><br/> 10:18:31: <span class="error">[<f9b015df>]</span> cl_io_submit_rw+0x8f/0x2b0 <span class="error">[obdclass]</span><br/> 10:18:31: <span class="error">[<f9272d69>]</span> lov_io_submit+0x4d9/0x1150 <span class="error">[lov]</span><br/> 10:18:31: <span class="error">[<f9af37d1>]</span> cl_page_get+0x51/0x220 <span class="error">[obdclass]</span><br/> 10:18:31: <span class="error">[<f935246c>]</span> vvp_page_unmap+0x4c/0xa0 <span class="error">[lustre]</span><br/> 10:18:31: <span class="error">[<f9352420>]</span> vvp_page_unmap+0x0/0xa0 <span class="error">[lustre]</span><br/> 10:18:31: <span class="error">[<f9aeee7a>]</span> cl_page_invoke+0x8a/0x270 <span class="error">[obdclass]</span><br/> 10:18:31: <span class="error">[<f9272890>]</span> lov_io_submit+0x0/0x1150 <span class="error">[lov]</span><br/> 10:18:31: <span class="error">[<f9b015df>]</span> cl_io_submit_rw+0x8f/0x2b0 <span class="error">[obdclass]</span><br/> 10:18:32: <span class="error">[<f9af2631>]</span> cl_page_gang_lookup+0x1e1/0x4c0 <span class="error">[obdclass]</span><br/> 10:18:32: <span class="error">[<f9b01a8b>]</span> cl_sync_io_init+0x4b/0x200 <span class="error">[obdclass]</span><br/> 10:18:32: <span class="error">[<f9aef06f>]</span> cl_page_unmap+0xf/0x20 <span class="error">[obdclass]</span><br/> 10:18:32: <span class="error">[<f9b05ad1>]</span> cl_io_submit_sync+0xb1/0x1b0 <span class="error">[obdclass]</span><br/> 10:18:32: <span class="error">[<f9afeee9>]</span> cl_lock_page_out+0x1e9/0x540 <span class="error">[obdclass]</span><br/> 10:18:32: <span class="error">[<f9501218>]</span> osc_lock_flush+0x38/0x70 <span class="error">[osc]</span><br/> 10:18:32: <span class="error">[<f9502c04>]</span> osc_lock_cancel+0x44/0x1e0 <span class="error">[osc]</span><br/> 10:18:32: <span class="error">[<f9af796d>]</span> cl_lock_trace0+0x12d/0x1a0 <span class="error">[obdclass]</span><br/> 10:18:32: <span class="error">[<f9af6dd8>]</span> cl_lock_cancel0+0x78/0x230 <span class="error">[obdclass]</span><br/> 10:18:32: <span class="error">[<f9af7a99>]</span> cl_lock_cancel+0xb9/0x230 <span class="error">[obdclass]</span><br/> 10:18:32: <span class="error">[<f9af7caf>]</span> cl_lock_mutex_tail+0x3f/0x50 <span class="error">[obdclass]</span><br/> 10:18:32: <span class="error">[<f95039dd>]</span> osc_ldlm_blocking_ast+0x24d/0x320 <span class="error">[osc]</span><br/> 10:18:32: <span class="error">[<f9503790>]</span> osc_ldlm_blocking_ast+0x0/0x320 <span class="error">[osc]</span><br/> 10:18:32: <span class="error">[<f90a647f>]</span> ldlm_cancel_callback+0x6f/0x1a0 <span class="error">[ptlrpc]</span><br/> 10:18:32: <span class="error">[<f8da299e>]</span> cfs_hash_bd_from_key+0x2e/0xa0 <span class="error">[libcfs]</span><br/> 10:18:32: <span class="error">[<f8da2756>]</span> cfs_hash_bd_lookup_intent+0x16/0xf0 <span class="error">[libcfs]</span><br/> 10:18:32: <span class="error">[<f90d1940>]</span> ldlm_cli_cancel_local+0xa0/0x690 <span class="error">[ptlrpc]</span><br/> 10:18:32: <span class="error">[<f90d184e>]</span> ldlm_prepare_lru_list+0x53e/0x590 <span class="error">[ptlrpc]</span><br/> 10:18:32: <span class="error">[<f90d3294>]</span> ldlm_cli_cancel_list_local+0xd4/0x310 <span class="error">[ptlrpc]</span><br/> 10:18:33: <span class="error">[<f90d6960>]</span> ldlm_cancel_lrur_policy+0x0/0x100 <span class="error">[ptlrpc]</span><br/> 10:18:33: <span class="error">[<f90d5263>]</span> ldlm_prep_elc_req+0x613/0x710 <span class="error">[ptlrpc]</span><br/> 10:18:33: <span class="error">[<f91688c3>]</span> req_capsule_init+0x43/0xa0 <span class="error">[ptlrpc]</span><br/> 10:18:33: <span class="error">[<f9168831>]</span> req_capsule_set+0x21/0x70 <span class="error">[ptlrpc]</span><br/> 10:18:33: <span class="error">[<f90d5389>]</span> ldlm_prep_enqueue_req+0x29/0x30 <span class="error">[ptlrpc]</span><br/> 10:18:33: <span class="error">[<f94d7e3b>]</span> osc_enqueue_base+0x21b/0x8f0 <span class="error">[osc]</span><br/> 10:18:33: <span class="error">[<c041ec40>]</span> __wake_up+0x2a/0x3d<br/> 10:18:33: <span class="error">[<f9504079>]</span> osc_lock_enqueue+0x259/0xdd0 <span class="error">[osc]</span><br/> 10:18:33: <span class="error">[<f95030b0>]</span> osc_lock_upcall+0x0/0x6e0 <span class="error">[osc]</span><br/> 10:18:33: <span class="error">[<f9af7f93>]</span> cl_lock_state_set+0x93/0x290 <span class="error">[obdclass]</span><br/> 10:18:33: <span class="error">[<f9503e20>]</span> osc_lock_enqueue+0x0/0xdd0 <span class="error">[osc]</span><br/> 10:18:33: <span class="error">[<f9afbdd4>]</span> cl_enqueue_try+0x124/0x5b0 <span class="error">[obdclass]</span><br/> 10:18:33: <span class="error">[<f9269cd4>]</span> lov_sublock_hold+0x24/0x320 <span class="error">[lov]</span><br/> 10:18:33: <span class="error">[<c041ec40>]</span> __wake_up+0x2a/0x3d<br/> 10:18:33: <span class="error">[<f926c630>]</span> lov_lock_enqueue+0x180/0xb60 <span class="error">[lov]</span><br/> 10:18:33: <span class="error">[<f926c4b0>]</span> lov_lock_enqueue+0x0/0xb60 <span class="error">[lov]</span><br/> 10:18:33: <span class="error">[<f9afbdd4>]</span> cl_enqueue_try+0x124/0x5b0 <span class="error">[obdclass]</span><br/> 10:18:33: <span class="error">[<f9af7601>]</span> cl_lock_user_add+0x51/0x200 <span class="error">[obdclass]</span><br/> 10:18:33: <span class="error">[<f9afd5d0>]</span> cl_lock_hold_mutex+0x650/0xaf0 <span class="error">[obdclass]</span><br/> 10:18:33: <span class="error">[<f9afccf6>]</span> cl_enqueue_locked+0x86/0x2c0 <span class="error">[obdclass]</span><br/> 10:18:34: <span class="error">[<f9aff81d>]</span> cl_lock_request+0x8d/0x2d0 <span class="error">[obdclass]</span><br/> 10:18:34: <span class="error">[<f9347b60>]</span> cl_glimpse_lock+0x170/0x570 <span class="error">[lustre]</span><br/> 10:18:34: <span class="error">[<f93481ea>]</span> cl_glimpse_size+0x28a/0x2a0 <span class="error">[lustre]</span><br/> 10:18:34: <span class="error">[<f92beecd>]</span> ll_inode_revalidate_it+0x7d/0x380 <span class="error">[lustre]</span><br/> 10:18:34: <span class="error">[<c048570c>]</span> __link_path_walk+0xd89/0xdab<br/> 10:18:34: <span class="error">[<f92bf205>]</span> ll_getattr_it+0x35/0x110 <span class="error">[lustre]</span><br/> 10:18:34: <span class="error">[<f92bf31f>]</span> ll_getattr+0x3f/0x60 <span class="error">[lustre]</span><br/> 10:18:34: <span class="error">[<f92bf2e0>]</span> ll_getattr+0x0/0x60 <span class="error">[lustre]</span><br/> 10:18:34: <span class="error">[<c047f797>]</span> vfs_getattr+0x40/0x9b<br/> 10:18:34: <span class="error">[<c047f819>]</span> vfs_lstat_fd+0x27/0x39<br/> 10:18:34: <span class="error">[<c0462a54>]</span> vma_prio_tree_insert+0x17/0x2a<br/> 10:18:34: <span class="error">[<c047f870>]</span> sys_lstat64+0xf/0x23<br/> 10:18:34: <span class="error">[<c044cb5b>]</span> audit_syscall_entry+0x18f/0x1b9<br/> 10:18:34: <span class="error">[<c0407f5f>]</span> do_syscall_trace+0xab/0xb1<br/> 10:18:34: <span class="error">[<c0404f4b>]</span> syscall_call+0x7/0xb<br/> 10:18:34: =======================</p> <p>To fix this issue, we should avoid issuing new IO by lock early cancellation. This can be done via the following solution:<br/> 1. implement a tagged cl_page radix tree;<br/> 2. call ->l_weigh_ast before a lock is gonna be early cancelled; and in ->l_weigh_ast, we should check if there are dirty pages under this lock. If this is the case, we should skip this lock(we may have a `washing' process to clean pages).</p> <p>Close old ticket.</p> <p>This was fixed by the patch:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LU-4300 ldlm: ELC picks locks in a safer policy Change the policy of ELC to pick locks that have no dirty pages, no page in writeback state, and no locked pages. Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com> Reviewed-on: http://review.whamcloud.com/9175 </pre> </div></div> Duplicate LU-4300 Development Rank 1|hzv4kf: Rank (Obsolete) 4273 Severity