https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-15127 Lustre <p>racer hits this deadlock few times a day:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> schedule,osc_extent_wait,osc_cache_wait_range,osc_cache_writeback_range,osc_io_fsync_start,cl_io_start,lov_io_call,cl_io_start,cl_io_loop,cl_sync_file_range,ll_delete_inode,evict,__dentry_kill,dentry_kill,dput,ll_dirty_page_discard_warn,vvp_page_completion_write,cl_page_completion,osc_ap_completion,osc_extent_finish,brw_interpret,ptlrpc_check_set,ptlrpcd PIDs(1): <span class="code-quote">"ptlrpcd_00_00"</span>:4889 schedule,osc_extent_wait,osc_cache_wait_range,osc_cache_writeback_range,osc_ldlm_blocking_ast,ldlm_cancel_callback,ldlm_cli_cancel_local,ldlm_cli_cancel,osc_ldlm_blocking_ast,ldlm_handle_bl_callback,ldlm_bl_thread_main PIDs(1): <span class="code-quote">"ldlm_bl_02"</span>:7759 schedule,ptlrpc_invalidate_import,ptlrpc_invalidate_import_thread PIDs(1): <span class="code-quote">"ll_imp_inval"</span>:293752 schedule,ptlrpc_invalidate_import,ptlrpc_set_import_active,osc_iocontrol,lov_iocontrol,ll_umount_begin,ksys_umount,__x64_sys_umount PIDs(1): <span class="code-quote">"umount"</span>:449648 </pre> </div></div> LU-15127

import invalidation vs writeback deadlock

Bug Minor Open Unresolved Patrick Farrell Alex Zhuravlev Tue, 19 Oct 2021 06:10:28 +0000 Mon, 14 Aug 2023 16:26:33 +0000 Upstream Lustre 2.16.0 0 6 <p>Ah, i think this is the classic dirty page discard warn issue… it takes a reference to print a debug message and can end up deadlocked because of it.</p> <p>I think I can figure out a fix (unless you’re already working on it).</p> <p>please, go ahead, I'm busy with another stuff.</p> <p>"Patrick Farrell &lt;pfarrell@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/45550" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/45550</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15127" title="import invalidation vs writeback deadlock" class="issue-link" data-issue-key="LU-15127">LU-15127</a> llite: Remove path from discard_warn<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: d04a1929b3adb776173b02e0f6b82d396046dd14</p> <p>the invalidation path still deadlock, but much less frequently. I guess this is a bit different issue?</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> schedule,ldlm_completion_ast,ldlm_lock_match_with_skip,osc_enqueue_base,osc_lock_enqueue,cl_lock_enqueue,lov_lock_enqueue,cl_lock_enqueue,cl_lock_request,cl_io_lock,cl_io_loop,cl_setattr_ost,ll_setattr_raw,do_truncate,path_openat,do_filp_open,do_sys_open PIDs(1): <span class="code-quote">"cp"</span>:367053 schedule,osc_object_invalidate,osc_ldlm_resource_invalidate,cfs_hash_for_each_relax,cfs_hash_for_each_nolock,osc_import_event,ptlrpc_invalidate_import,ptlrpc_invalidate_import_thread PIDs(1): <span class="code-quote">"ll_imp_inval"</span>:384198 schedule,osc_object_invalidate,osc_ldlm_resource_invalidate,cfs_hash_for_each_relax,cfs_hash_for_each_nolock,osc_import_event,ptlrpc_invalidate_import,ptlrpc_set_import_active,osc_iocontrol,lov_iocontrol,ll_umount_begin,ksys_umount,__x64_sys_umount PIDs(1): <span class="code-quote">"umount"</span>:384405 </pre> </div></div> <p>Hmm...  So osc_object_invalidate is probably waiting for nr_ios to be zero, and that's incremented in cl_io_iter_init (osc_io_iter_init) before the lock request is made.</p> <p>So I guess it's waiting for that competion ast, which is waiting for the lock to be granted or cancelled.</p> <p>So somehow that lock isn't getting granted or cancelled I guess?</p> <p>I'm not quite sure how pending lock requests are cancelled when an import is invalidated.</p> <p>Are you able to dump the LDLM namespaces for that hang?</p> <p>So looking at osc_import_event for INVALIDATE, we call:</p> <p>ldlm_namespace_cleanup<br/> osc_ldlm_resource_invalidate (which calls osc_object_invalidate)<br/> Then call<br/> ldlm_namespace_cleanup</p> <p>again</p> <p>ldlm_namespace_cleanup calls ldlm_resource_clean, then ldlm_resource_complain.</p> <p>ldlm_resource_complain shows that we can sometimes have locks left after ldlm_resource_clean</p> <p>So the thread doing 'cp' is trying to match an existing lock.</p> <p>So the lock survives the call to ldlm_resource_clean, then the osc oo_nr_ios is &gt; 0, so we cannot invalidate the OSC object, so we do not try to clean up the lock again.</p> <p>I don't know why a lock would survive ldlm_resource_clean, but that seems like the issue.</p> <p>Maybe we need to call ldlm_resource_clean from osc_object_invalidate if oo_nr_ios is &gt; 0 ?</p> <p>"Patrick Farrell &lt;pfarrell@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/45658" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/45658</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15127" title="import invalidation vs writeback deadlock" class="issue-link" data-issue-key="LU-15127">LU-15127</a> osc: Resource cleanup in osc invalidate<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 7bad59f258269dd358d066bd023defeaec955f6d</p> <p>There are a few fairly heroic guesses in that patch, but I think it's probably right...  Alex, if you can try it in your test rig...</p> Duplicate LU-12522 Related LU-15340 Development Rank 1|i027qf: Rank (Obsolete) 9223372036854775807 Severity