https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-8927 Lustre <p>Ran jobs which created remote directories (not striped) and then ran mdtest within them, several MDS nodes are using &gt;80% of their cpu time for osp-syn-* processes.</p> <p>There are 36 osp-syn-* processes.</p> <p>The processes are spending almost all their time contending for osq_lock. According to perf, the offending stack is:</p> <p>osq_lock<br/> __mutex_lock_slowpath<br/> mutex_lock<br/> spa_config_enter<br/> bp_get_dsize<br/> dmu_tx_hold_free<br/> osd_declare_object_destroy<br/> llog_osd_declare_destroy<br/> llog_declare_destroy<br/> llog_cancel_rec<br/> llog_cat_cancel_records<br/> osp_sync_process_committed<br/> osp_sync_process_queues<br/> llog_process_thread<br/> llog_process_or_fork<br/> llog_cat_process_cb<br/> llog_process_thread<br/> llog_process_or_fork<br/> llog_cat_process_or_fork<br/> llog_cat_process<br/> osp_sync_thread<br/> kthread<br/> ret_from_fork<br/> osp-syn-X-Y</p> lustre-2.8.0_5.chaos-2.ch6.x86_64<br/> zfs-0.7.0-0.6llnl.ch6.x86_64<br/> DNE with 16 MDTs LU-8927

osp-syn processes contending for osq_lock drives system cpu usage > 80%

Bug Major Resolved Won't Fix Alex Zhuravlev Olaf Faaland llnl zfs Thu, 8 Dec 2016 23:21:54 +0000 Mon, 18 Sep 2017 21:29:26 +0000 Tue, 5 Sep 2017 16:58:09 +0000 0 6 <p>Our stack is available to Intel engineers via repository named "lustre-release-fe-llnl" hosted on your gerritt server.</p> <p>I see that llog_cancel_rec() contains the following:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> rc = llog_declare_write_rec(env, loghandle, &amp;llh-&gt;llh_hdr, index, if (rc &lt; 0) GOTO(out_trans, rc); if ((llh-&gt;llh_flags &amp; LLOG_F_ZAP_WHEN_EMPTY)) rc = llog_declare_destroy(env, loghandle, th); th-&gt;th_wait_submit = 1; rc = dt_trans_start_local(env, dt, th); </pre> </div></div> <p>So it seems to declare that it will destroy the llog object every time it cancels a record, as if every record is the last one. Why is that? Shouldn't it also depend on how many active records the llog contains? </p> <p>when we declare llog cancelation we don't known whether it will be last one or not, otherwise we'd have to lock llog since declaration upto transaction stop killing concurrency. the newer versions of Lustre will fix this problem in osd-zfs module.</p> <p>Alex</p> <p>Could you please elaborate about the work underway in this area?</p> <p>Thanks</p> <p>Peter</p> <p>Yes, please elaborate. I know there are many ways to work on this and it would be great to know the nature and scope of the fix you have in mind.</p> <p>I looked again and the MDTs are still working to clear llog records from jobs run about 45 hours ago (contended the entire time). I don't think we can go into production without a fix for this.</p> <p>in very few words - I've been working to make declarations with ZFS cheap. right now those are quite expensive because DMU API works with dnode numbers, so every time it needs to translate dnode number into dnode structure using the global hash table. few patches have been landed already onto master branch and released as a part of 2.9 (e.g. <a href="https://jira.whamcloud.com/browse/LU-7898" title="remove unnecessary declarations from osd-zfs" class="issue-link" data-issue-key="LU-7898"><del>LU-7898</del></a> osd: remove unnecessary declarations). yet more improvements are expected with landing of the following patches:</p> <p><a href="https://jira.whamcloud.com/browse/LU-8882" title="osd-zfs to use bynode methods" class="issue-link" data-issue-key="LU-8882"><del>LU-8882</del></a> osd: use bydnode methods to access ZAP<br/> <a href="https://jira.whamcloud.com/browse/LU-8928" title="osd-zfs should use dnode_t instead of dbuf" class="issue-link" data-issue-key="LU-8928"><del>LU-8928</del></a> osd: convert osd-zfs to reference dnode, not db<br/> <a href="https://jira.whamcloud.com/browse/LU-8873" title="use sa_handle_get_from_db()" class="issue-link" data-issue-key="LU-8873"><del>LU-8873</del></a> osd: use sa_handle_get_from_db()<br/> <a href="https://jira.whamcloud.com/browse/LU-2435" title="inode accounting in osd-zfs is racy" class="issue-link" data-issue-key="LU-2435"><del>LU-2435</del></a> osd-zfs: use zfs native dnode accounting<br/> and <a href="https://github.com/zfsonlinux/zfs/pull/5464" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/zfsonlinux/zfs/pull/5464</a></p> <p>this way the declarations should become mostly lockless and much cheaper.</p> <p>OK, thanks.</p> <p>In the above list of tickets, you include <a href="https://jira.whamcloud.com/browse/LU-8893" title="SSK - sanity test_126 cannot touch, Permission denied" class="issue-link" data-issue-key="LU-8893">LU-8893</a>. Did you mean <a href="https://jira.whamcloud.com/browse/LU-8873" title="use sa_handle_get_from_db()" class="issue-link" data-issue-key="LU-8873"><del>LU-8873</del></a>?</p> <p>I looked briefly at the <a href="https://jira.whamcloud.com/browse/LU-7898" title="remove unnecessary declarations from osd-zfs" class="issue-link" data-issue-key="LU-7898"><del>LU-7898</del></a> patch to remove unnecessary declarations. I'll see if I can apply and test it.</p> <p>you're right, I mean <a href="https://jira.whamcloud.com/browse/LU-8873" title="use sa_handle_get_from_db()" class="issue-link" data-issue-key="LU-8873"><del>LU-8873</del></a>, basically yet another point to save on dnode#-&gt;dnode_t lookup.</p> <p>Alex,</p> <p>I applied <a href="https://jira.whamcloud.com/browse/LU-7898" title="remove unnecessary declarations from osd-zfs" class="issue-link" data-issue-key="LU-7898"><del>LU-7898</del></a> on top of our 2.8.0+patch stack and see the same symptoms. The patch didn't appear to me to change any of the functions in the contending stacks, so not surprising.</p> <p>The full set of patches above would be too much for a stable branch, I would think. So I've rewritten llog_cancel_rec() to destroy the llog in a second transaction, if it's necessary. Maybe this is a poor approach; feedback or an alternative would be welcome. In any case I've pushed it to gerrit and will do local testing after it passes maloo.</p> <p><a href="https://review.whamcloud.com/#/c/24687/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/24687/</a></p> <p>I would like a level set on this ticket. All of the planned work to improve metadata performance for ZFS has now landed to master (and b2_10). Are there any specific tasks identified and remaining beyond that?</p> <p>This lock contention has not resulted in problems in production, and there is so much related change in 2.10 and master that it's quite possible the problem does not occur there. Closing the ticket.</p> Related LU-2435 LU-8873 LU-8882 LU-8928 Development Rank 1|hzyxzr: Rank (Obsolete) 9223372036854775807 Severity