Whamcloud Community JIRA

Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us

9.4.14

940014

05-12-2023

[LU-14694] racer timeouts with LU-10948 https://jira.whamcloud.com/browse/LU-14694 Lustre <p>many racer runs timeout, bisected that to patch:<br/> <a href="https://review.whamcloud.com/32158/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32158/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10948" title="client cache open lock after N opens" class="issue-link" data-issue-key="LU-10948">LU-10948</a> llite: Introduce inode open heat counter<br/> Project: fs/lustre-release<br/> Branch: master<br/> Commit: 41d99c4902836b7265db946dfa49cf99381f0db4</p> LU-14694

racer timeouts with LU-10948

Bug Minor Resolved Fixed Alex Zhuravlev Alex Zhuravlev Thu, 20 May 2021 04:22:30 +0000 Tue, 10 Aug 2021 13:12:35 +0000 Tue, 10 Aug 2021 13:12:35 +0000 Upstream Lustre 2.15.0 0 6 <p>Alex, what kind of timeout? Stuck processes (with stack?), unable to unmount, oops, other? Single client mountpoint or multiple?</p> <p>Previously Oleg and I had discussed whether opencache locks needed to time out sooner. Could you try running with <tt>ldlm.namespaces.*.lru_max_age=30</tt> to see if that helps?</p> <p>Also, is this with opencache enabled, or just with the patch applied?</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> ... Resetting fail_loc on all nodes...done. PASS 1 (2720s) == racer test complete, duration 2768 sec ============================================================ 16:47:13 (1621529233) Stopping clients: tmp.xMEAXwSwZi /mnt/lustre (opts:-f) Stopping client tmp.xMEAXwSwZi /mnt/lustre opts:-f Stopping clients: tmp.xMEAXwSwZi /mnt/lustre2 (opts:-f) Stopping client tmp.xMEAXwSwZi /mnt/lustre2 opts:-f Stopping /mnt/lustre-mds1 (opts:-f) on tmp.xMEAXwSwZi modprobe: FATAL: Module dm-flakey not found. Stopping /mnt/lustre-mds2 (opts:-f) on tmp.xMEAXwSwZi </pre> </div></div> <p>then unique traces:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> schedule,do_select,trace_hardirqs_off_thunk PIDs(1): <span class="code-quote">"sshd"</span>:2287 schedule,do_select,_copy_to_user PIDs(1): <span class="code-quote">"sshd"</span>:2587 schedule,top_trans_stop,lod_trans_stop,mdd_trans_stop,mdt_mfd_close,mdt_obd_disconnect,class_fail_export,ping_evictor_main PIDs(1): <span class="code-quote">"ll_evictor"</span>:7754 schedule,ptlrpc_set_wait,ptlrpc_queue_wait,osp_send_update_req,osp_send_update_thread PIDs(1): <span class="code-quote">"osp_up0-1"</span>:7989 schedule,top_trans_stop,lod_trans_stop,mdd_trans_stop,mdt_mfd_close,mdt_obd_disconnect,class_disconnect_export_list,class_disconnect_exports,class_cleanup,class_process_config,class_manual_cleanup,server_put_super,generic_shutdown_super,kill_anon_super,deactivate_locked_super,cleanup_mnt,task_work_run,exit_to_usermode_loop PIDs(1): <span class="code-quote">"umount"</span>:535030 </pre> </div></div> <p>basically it's MDSCOUNT=2 REFORMAT=yes bash racer.sh</p> <blockquote><p>Also, is this with opencache enabled, or just with the patch applied?</p></blockquote> <p>this is the latest clean master</p> <p>sanity/29 fails as well:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> == sanity test 29: IT_GETATTR regression ============================================================ 11:52:13 (1621597933) Lustre: DEBUG MARKER: == sanity test 29: IT_GETATTR regression ============================================================ 11:52:13 (1621597933) striped dir -i1 -c2 -H all_char /mnt/lustre/d29 first d29 Lustre: DEBUG MARKER: first d29 total 0 -rw-r--r-- 1 root root 0 May 21 11:52 foo second d29 Lustre: DEBUG MARKER: second d29 total 0 -rw-r--r-- 1 root root 0 May 21 11:52 foo done Lustre: DEBUG MARKER: done sanity test_29: @@@@@@ FAIL: CURRENT: 10 > 9 Lustre: DEBUG MARKER: sanity test_29: @@@@@@ FAIL: CURRENT: 10 > 9 Trace dump: = ./../tests/test-framework.sh:6280:error() = sanity.sh:3398:test_29() = ./../tests/test-framework.sh:6583:run_one() = ./../tests/test-framework.sh:6630:run_one_logged() = ./../tests/test-framework.sh:6457:run_test() = sanity.sh:3410:main() Dumping lctl log to /tmp/ltest-logs/sanity.test_29.*.1621597935.log Dumping logs only on local client. Resetting fail_loc on all nodes...done. FAIL 29 (2s) </pre> </div></div> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> --- a/lustre/tests/racer.sh +++ b/lustre/tests/racer.sh @@ -180,6 +180,7 @@ test_1() { lss_cleanup fi + cancel_lru_locks mdc <span class="code-keyword">return</span> $rrc } run_test 1 <span class="code-quote">"racer on clients: ${CLIENTS:-$(hostname)} DURATION=$DURATION"</span> </pre> </div></div> <p>this helps.. it looks like openhandle locks are lost at umount, causing MDT to close them at export cleanup.</p> <p>basically the problem is that:<br/> 1) openlock is taken on directories<br/> 2) the client doesn't cancel locks on umount:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-comment">/* obd_force == local only */</span> ldlm_cli_cancel_unused(obd->obd_namespace, NULL, obd->obd_force ? LCF_LOCAL : 0, NULL); </pre> </div></div> <p>3) thus MDT has to clear those "lost" locks, so close the directories<br/> 4) a close may result in directory removal<br/> 5) directory can be striped, thus needs all involved MDTs to be healty<br/> 6) MDTs are stopped one by one</p> <p>locally I "solved" the problem disabling opencache for directories:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> --- a/lustre/mdt/mdt_open.c +++ b/lustre/mdt/mdt_open.c @@ -820,6 +820,9 @@ <span class="code-keyword">static</span> <span class="code-object">int</span> mdt_object_open_lock(struct mdt_thread_info *info, <span class="code-keyword">if</span> (req_is_replay(mdt_info_req(info))) RETURN(0); + <span class="code-keyword">if</span> (S_ISDIR(lu_object_attr(&obj->mot_obj))) + open_flags &= ~MDS_OPEN_LOCK; + <span class="code-keyword">if</span> (S_ISREG(lu_object_attr(&obj->mot_obj))) { </pre> </div></div> <p>but it looks like we've got few real problems to solve here. the most serious one, IMO, is to handle such a close on MDT in a better manner, so that umount doesn't get stuck indefinitely.</p> <p>probably a bit better option is to re-use existing infrastructure to skip orphan removal:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> --- a/lustre/mdt/mdt_handler.c +++ b/lustre/mdt/mdt_handler.c @@ -6438,7 +6438,8 @@ <span class="code-keyword">static</span> <span class="code-object">int</span> mdt_export_cleanup(struct obd_export *exp) rc = mdt_ctxt_add_dirty_flag(&env, info, mfd); <span class="code-comment">/* Don't unlink orphan on failover umount, LU-184 */</span> - <span class="code-keyword">if</span> (exp->exp_flags & OBD_OPT_FAILOVER) { + <span class="code-keyword">if</span> (exp->exp_flags & OBD_OPT_FAILOVER || + exp->exp_obd->obd_stopping) { ma->ma_valid = MA_FLAGS; ma->ma_attr_flags |= MDS_KEEP_ORPHAN; } </pre> </div></div> <p>I think we need something like above on its own as umount with just evicted client is a legal case.</p> <p>Alex Zhuravlev (bzzz@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/43783" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/43783</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14694" title="racer timeouts with LU-10948" class="issue-link" data-issue-key="LU-14694"><del>LU-14694</del></a> mdt: do not remove orphans at umount<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 9f9e98a5e94e021c68882db5ab3d03484048b810</p> <p>"Oleg Drokin <green@whamcloud.com>" merged in patch <a href="https://review.whamcloud.com/43783/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/43783/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14694" title="racer timeouts with LU-10948" class="issue-link" data-issue-key="LU-14694"><del>LU-14694</del></a> mdt: do not remove orphans at umount<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 0b94a058fe6732907d7aa3f5de0cbd7888d28ce1</p> <p>Landed for 2.15</p> Related LU-10948 Development Rank 1|i01v1j: Rank (Obsolete) 9223372036854775807 Severity