https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-12651 Lustre <p>We discovered that on our systems with lustre mounted, the kworker is using signifficant amount of CPU. <br/> perf top shows on an idle system: </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> 39.44% [kernel] [k] osc_should_shrink_grant 12.14% [kernel] [k] osc_grant_work_handler 2.81% [kernel] [k] process_one_work 2.64% [kernel] [k] __queue_work 2.56% [kernel] [k] read_tsc </pre> </div></div> <p>We currently have grant_shrink=0 on this system. </p> <p>Looks like doing just du -hs /fs makes the problem go away for some time. <br/> Also unmounting the filesystem makes the problem go away. <br/> This is Centos 7.6 system with Lustre 2.12.0</p> LU-12651

High kworker CPU usage (osc_grant_work_handler) on IDLE connections

Bug Major Resolved Fixed Alexander Zarochentsev Jacek Tomaka LTS12 Fri, 9 Aug 2019 09:05:34 +0000 Tue, 25 Feb 2020 12:58:34 +0000 Fri, 14 Feb 2020 13:41:52 +0000 Lustre 2.12.2 Lustre 2.14.0 Lustre 2.12.5 0 8 <p>Most likely regression from <a href="https://jira.whamcloud.com/browse/LU-8708" title="Grant shrinking disabled all the time" class="issue-link" data-issue-key="LU-8708"><del>LU-8708</del></a></p> <p>Any news on this ticket?</p> <p>Alexander Zarochentsev (c17826@cray.com) uploaded a new patch: <a href="https://review.whamcloud.com/37429" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37429</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12651" title="High kworker CPU usage (osc_grant_work_handler) on IDLE connections" class="issue-link" data-issue-key="LU-12651"><del>LU-12651</del></a> osc: always call update_next_shrink<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 2986155c51914c5a63f6c351908c9a49dbe5042f</p> <p>Jasek Tomaka, <br/> can you try <a href="https://review.whamcloud.com/37429" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37429</a> ?</p> <p>my experiments with 2.12-based lustre and grant_shrink=0:</p> <p>w/o the fix, kworker starts to eat 100% CPU after 20 min from Lustre mount time (default grant shrinking interval)</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>top - 00:03:08 up 2 days, 11:32, 3 users, load average: 2.95, 2.47, 2.22 Tasks: 258 total, 3 running, 255 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.0 us, 25.0 sy, 0.0 ni, 75.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem : 2914024 total, 1138684 free, 544988 used, 1230352 buff/cache KiB Swap: 2113532 total, 2113532 free, 0 used. 2190536 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 21631 root 20 0 0 0 0 R 100.0 0.0 3:03.08 kworker/3:2 1 root 20 0 191032 3912 2584 S 0.0 0.1 0:06.70 systemd 2 root 20 0 0 0 0 S 0.0 0.0 0:00.06 kthreadd 3 root 20 0 0 0 0 S 0.0 0.0 0:01.06 ksoftirqd/0 5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H </pre> </div></div> <p>with the fix,  22 min after start, system is idle:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>top - 00:32:05 up 2 days, 12:01, 3 users, load average: 2.00, 2.01, 2.06 Tasks: 261 total, 2 running, 259 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.1 us, 0.1 sy, 0.0 ni, 99.8 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem : 2914024 total, 1133004 free, 549940 used, 1231080 buff/cache KiB Swap: 2113532 total, 2113532 free, 0 used. 2185136 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 367 root 20 0 162180 2456 1584 R 0.3 0.1 0:00.03 top 1 root 20 0 191032 3912 2584 S 0.0 0.1 0:06.85 systemd 2 root 20 0 0 0 0 S 0.0 0.0 0:00.07 kthreadd 3 root 20 0 0 0 0 S 0.0 0.0 0:01.10 ksoftirqd/0 5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H 7 root rt 0 0 0 0 S 0.0 0.0 0:00.61 migration/0 8 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh </pre> </div></div> <p>Hi Alexander, <br/> Thanks for looking into it. Would you be so kind to provide patch for 2.12.3 as well?<br/> Regards.<br/> Jacek Tomaka</p> <p>Jacek,<br/> &gt;Thanks for looking into it. Would you be so kind to provide patch for 2.12.3 as well?<br/> The same patch applies to b2_12.</p> <p>Hi Alexander, <br/> Our initial testing on a machine with patched client(2.12.3 + <a href="https://jira.whamcloud.com/browse/LU-12759" title="parameter grant_shrink gets reset to 1 after client reconnects" class="issue-link" data-issue-key="LU-12759"><del>LU-12759</del></a> + this patch) shows that the kworker does not go crazy anymore. <br/> Great job! Thanks!<br/> Will let you know if we run into any issues with this patch. <br/> Jacek Tomaka</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/37429/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37429/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12651" title="High kworker CPU usage (osc_grant_work_handler) on IDLE connections" class="issue-link" data-issue-key="LU-12651"><del>LU-12651</del></a> osc: always call update_next_shrink<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 117f587bc3e60f4dd1c939f8488e43cb752c12ca</p> <p>Landed for 2.14</p> <p>Minh Diep (mdiep@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/37572" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37572</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12651" title="High kworker CPU usage (osc_grant_work_handler) on IDLE connections" class="issue-link" data-issue-key="LU-12651"><del>LU-12651</del></a> osc: always call update_next_shrink<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: 70d299f149e1cb5f396576baf452a5eba911a30a</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/37572/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37572/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12651" title="High kworker CPU usage (osc_grant_work_handler) on IDLE connections" class="issue-link" data-issue-key="LU-12651"><del>LU-12651</del></a> osc: always call update_next_shrink<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: 10a799263964422df575038d3dfb507a09bfa221</p> Duplicate Related LU-8708 Development Rank 1|i00l0n: Rank (Obsolete) 9223372036854775807 Severity