https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-14125 Lustre <p>Something is causing cur_grant_bytes for some OSCs to go below 1MB. Which OSCs and which nodes are apparently random. The OSTs themselves have many TB of free space. Sequential writes (e.g. dd if=/dev/urandom of=file_on_ost_3 bs=1M count=40 where the file has just one stripe) produces osc_enter_cache() debug entries reporting it is falling back to sync I/O. We also see osc_update_grant() report it got 0 extra grant.</p> <p>We have not been able to identify a workload or other trigger that pushes cur_grant_bytes low in the first place. We also have not been able to find a workaround that results in the OST returning extra grant.</p> <p>We set grant_shrink=0 on all clients (using set_param -P on the mgs) and then stopped and started all the OSTs on the file system. This did not change the symptoms in any obvious way.</p> <p>Client snippet with debug="+cache":</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000008:00000020:55.0:1604686702.850333:0:15766:0:(osc_cache.c:1613:osc_enter_cache()) lsrza-OST0003-osc-ffff8b2f37fc1000: grant { dirty: 0/512000 dirty_pages: 0/16450184 dropped: 0 avail: 997461, dirty_grant: 0, reserved: 0, flight: 0 } lru {in list: 9984, left: 256, waiters: 0 }need:1703936 00000008:00000020:55.0:1604686702.850335:0:15766:0:(osc_cache.c:1543:osc_enter_cache_try()) lsrza-OST0003-osc-ffff8b2f37fc1000: grant { dirty: 0/512000 dirty_pages: 0/16450184 dropped: 0 avail: 997461, dirty_grant: 0, reserved: 0, flight: 0 } lru {in list: 9984, left: 256, waiters: 0 }need:1703936 00000008:00000020:55.0:1604686702.850337:0:15766:0:(osc_cache.c:1658:osc_enter_cache()) lsrza-OST0003-osc-ffff8b2f37fc1000: grant { dirty: 0/512000 dirty_pages: 0/16450184 dropped: 0 avail: 997461, dirty_grant: 0, reserved: 0, flight: 0 } lru {in list: 9984, left: 256, waiters: 0 }no grant space, fall back to sync i/o 00000008:00400020:55.0:1604686702.850352:0:15766:0:(osc_io.c:127:osc_io_submit()) 256 1 00000008:00000020:55.0:1604686702.850385:0:15766:0:(osc_cache.c:1743:osc_update_pending()) obj ffff8b2c6bc58640 ready 0|-|- wr 256|+|- rd 0|- update pending cmd 2 delta 256. 00000008:00000020:55.0:1604686702.850387:0:15766:0:(osc_cache.c:2297:osc_io_unplug0()) Queue writeback work for client ffff8b1efb0d25e0. 00000008:00000020:19.0:1604686702.850400:0:20698:0:(osc_request.c:3171:brw_queue_work()) Run writeback work for client obd ffff8b1efb0d25e0. 00000008:00000020:19.0:1604686702.850402:0:20698:0:(osc_cache.c:2222:osc_check_rpcs()) obj ffff8b2c6bc58640 ready 0|-|- wr 256|+|- rd 0|- 0 in flight 00000008:00000020:19.0:1604686702.850404:0:20698:0:(osc_cache.c:1697:osc_makes_rpc()) high prio request forcing RPC 00000008:00000020:19.0:1604686702.850405:0:20698:0:(osc_cache.c:1888:try_to_add_extent_for_io()) extent ffff8b2c6b9c7c30@{[9984 -&gt; 10239/10239], [1|0|+|lockdone|wShu|ffff8b2c6bc58640], [0|256|+|-| (null)|256| (null)]} trying to add this extent 00000008:00000020:19.0:1604686702.850408:0:20698:0:(osc_cache.c:1743:osc_update_pending()) obj ffff8b2c6bc58640 ready 0|-|- wr 0|-|- rd 0|- update pending cmd 2 delta -256. 00000008:00000020:19.0:1604686702.850441:0:20698:0:(osc_request.c:705:osc_announce_cached()) dirty: 0 undirty: 1879048191 dropped 0 grant: 997461 00000008:00000020:19.0:1604686702.850443:0:20698:0:(osc_request.c:714:osc_update_next_shrink()) next time 6200398 to shrink grant 00000008:00000020:60.0:1604686703.244890:0:20699:0:(osc_request.c:727:osc_update_grant()) got 0 extra grant </pre> </div></div> <p>Server snippet:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000020:00000020:1.0:1604686702.851666:0:15401:0:(tgt_grant.c:413:tgt_grant_statfs()) lsrza-OST0003: cli 726797b8-322a-1989-0cb5-3645daf9a6ce/ffff8fe6bb09d800 free: 263316194721792 avail: 263316186333184 00000020:00000020:1.0:1604686702.851668:0:15401:0:(tgt_grant.c:477:tgt_grant_space_left()) lsrza-OST0003: cli 726797b8-322a-1989-0cb5-3645daf9a6ce/ffff8fe6bb09d800 avail 263316186333184 left 262158115930112 unstable 3407872 tot_grant 1158069646026 pending 3407872 00000020:00000020:1.0:1604686702.851670:0:15401:0:(tgt_grant.c:519:tgt_grant_incoming()) lsrza-OST0003: cli 726797b8-322a-1989-0cb5-3645daf9a6ce/ffff8fe6bb09d800 reports grant 997461 dropped 0, local 1882456063 00000020:00000020:1.0:1604686702.851672:0:15401:0:(tgt_grant.c:848:tgt_grant_check()) lsrza-OST0003: cli 726797b8-322a-1989-0cb5-3645daf9a6ce/ffff8fe6bb09d800 granted: 0 ungranted: 1703936 grant: 1882456063 dirty: 0 </pre> </div></div> <p>Each pool contains only one OST, and the storage is used for nothing else. All the OSTs have about the same amount of free space. This is the pool containing OST0003.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@brass8:toss-4917-grant]# zpool list NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT brass8 580T 243T 337T - 25% 41% 1.00x ONLINE - </pre> </div></div> zfs-0.7 <br/> lustre-2.12.5_5.llnl-1.ch6.x86_64 LU-14125

client starved for grant but OST has plenty of free space

Bug Major Reopened Unresolved Mikhail Pershin Olaf Faaland llnl topllnl Fri, 6 Nov 2020 18:44:42 +0000 Fri, 29 Jul 2022 23:11:06 +0000 Lustre 2.14.0 0 21 <p>We can get a debug patch on either the server or the client within about 1 week (it would need to be pushed to gerrit and pass tests first). For instance, there are no CDEBUGs in osc_should_shrink_grant().</p> <p>We haven't identified a workaround other than stopping and starting servers, which is obviously not good enough. Workaround suggestions would be very useful, as this is significantly impacting our users.</p> <p>Mike</p> <p>Could you please advise</p> <p>Thanks</p> <p>Peter</p> <p>Thanks Peter</p> <p>For my tracking, my local ticket is TOSS-4917</p> <p>There was a grant-related overflow fixed in patch <a href="https://review.whamcloud.com/39380" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39380</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-13763" title="ptlrpc_invalidate_import()) lsrza-OST0000_UUID: Unregistering RPCs found (0). Network is sluggish? Waiting them to error out." class="issue-link" data-issue-key="LU-13763"><del>LU-13763</del></a> osc: don't allow negative grants</tt>" that may be causing issues here. If the client thinks it has little or no grant, but the server thinks it has lots of grant, the client will keep asking, but the server will not give it more grant until it uses its current amount.</p> <p>The patch was landed to b2_12 but not until after 2.12.5 was released. </p> <p>Mike, for debugging purposes, it would be useful to add per-export grant parameters on the server (<tt>obdfilter.&#42;.exports.&#42;.grant&#42;</tt>" to match the clients). That would allow comparing the values on the client and server and see what kind of difference there is. </p> <p>Thanks, Andreas.</p> <blockquote><p>Mike, for debugging purposes, it would be useful to add per-export grant parameters on the server (obdfilter.<b>.exports.</b>.grant*" to match the clients). That would allow comparing the values on the client and server and see what kind of difference there is.</p></blockquote> <p>Like this?<br/> <a href="https://review.whamcloud.com/#/c/39324/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/39324/</a></p> <blockquote><p>There was a grant-related overflow fixed in patch <a href="https://review.whamcloud.com/39380" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39380</a> "<a href="https://jira.whamcloud.com/browse/LU-13763" title="ptlrpc_invalidate_import()) lsrza-OST0000_UUID: Unregistering RPCs found (0). Network is sluggish? Waiting them to error out." class="issue-link" data-issue-key="LU-13763"><del>LU-13763</del></a> osc: don't allow negative grants" that may be causing issues here. If the client thinks it has little or no grant, but the server thinks it has lots of grant, the client will keep asking, but the server will not give it more grant until it uses its current amount.</p></blockquote> <p>Ah. Next week, the systems are scheduled to get an update with that patch.</p> <p>Specifically, this:<br/> <a href="https://github.com/LLNL/lustre/compare/2.12.5_5.llnl...2.12.5_10.llnl" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/LLNL/lustre/compare/2.12.5_5.llnl...2.12.5_10.llnl</a></p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/40563" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40563</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14125" title="client starved for grant but OST has plenty of free space" class="issue-link" data-issue-key="LU-14125">LU-14125</a> obdclass: add grant fields to export procfile<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: efeaa656a5533fb6cff20c06447c7bf686db2bce</p> <p>ORNL is also running into this problem and we have the grant overflow fix patch.</p> <blockquote><p>ORNL is also running into this problem and we have the grant overflow fix patch.</p></blockquote> <p>Thanks James, that's good to know.</p> <p>I'd forgotten, the lustre version running on these two machines already has the per-export grant parameters patch.</p> <p>From the server:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@brass8:192.168.128.133@o2ib32]# cat export 726797b8-322a-1989-0cb5-3645daf9a6ce: name: lsrza-OST0003 client: 192.168.128.133@o2ib32 connect_flags: [ write_grant, server_lock, version, request_portal, truncate_lock, max_byte_per_rpc, early_lock_cancel, adaptive_timeouts, lru_resize, alt_checksum_algorithm, fid_is_enabled, version_recovery, grant_shrink, full20, layout_lock, 64bithash, object_max_bytes, jobstats, einprogress, grant_param, lvb_type, short_io, lfsck, bulk_mbits, second_flags, lockaheadv2 ] connect_data: flags: 0xa0425af2e3440478 instance: 42 target_version: 2.12.5.0 initial_grant: 2097152 max_brw_size: 1048576 grant_block_size: 1048576 grant_inode_size: 4096 grant_max_extent_size: 1073741824 grant_extent_tax: 655360 cksum_types: 0xf7 max_object_bytes: 9223372036854775807 export_flags: [ ] grant: granted: 1882456063 dirty: 0 pending: 0 </pre> </div></div> <p>From the client:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@rztopaz133:toss-4917-grant]# lctl list_nids 192.168.128.133@o2ib32 [root@rztopaz133:toss-4917-grant]# lctl get_param osc.lsrza-OST0003*.cur_grant_bytes osc.lsrza-OST0003-osc-ffff8b2f37fc1000.cur_grant_bytes=997461 </pre> </div></div> <p>So <tt>1882456063 = 0x7033ffff</tt>, which means that it is very close to overflowing, and likely had overflowed at some point in the past?</p> <p>Would it be possible to check all of the exports on that OSS (and/or other OSSes) and see if the high <tt>granted:</tt> line on the OSS, and a low <tt>cur_grant_bytes</tt> on the client correlates to those clients that are having slow writes? They should in theory be identical between client and OSS when the client does not have any dirty cached pages. It may not be 100% correlation, because some clients might not have actually overflowed yet. </p> <p>Hi Andreas, the high <b>granted:</b> line on the OST does not correlate well with low <b>cur_grant_bytes</b> on the clients.</p> <p>One cluster, rztopaz, has 6 clients with <b>cur_grant_bytes</b> &lt; 1M for OST0003. The OST reports that 238 exports to nodes in that cluster have <b>granted:</b> value of 1882456063.</p> <p>It's interesting there are so many nodes with granted = 1882456063. <br/> I spot-checked 5 clients with that value for granted, and cur_grant_bytes on those 5 clients are all over the place.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>/proc/fs/lustre/obdfilter/lsrza-OST0003/exports/192.168.128.102@o2ib32/export: granted: 1882456063 /proc/fs/lustre/obdfilter/lsrza-OST0003/exports/192.168.128.104@o2ib32/export: granted: 1882456063 /proc/fs/lustre/obdfilter/lsrza-OST0003/exports/192.168.128.108@o2ib32/export: granted: 1882456063 /proc/fs/lustre/obdfilter/lsrza-OST0003/exports/192.168.128.109@o2ib32/export: granted: 1882456063 /proc/fs/lustre/obdfilter/lsrza-OST0003/exports/192.168.128.110@o2ib32/export: granted: 1882456063 </pre> </div></div> <p>vs</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>e102: osc.lsrza-OST0003-osc-ffff9bbe688de000.cur_grant_bytes=82790394 e104: osc.lsrza-OST0003-osc-ffff9b69c9dea000.cur_grant_bytes=1412694016 e108: osc.lsrza-OST0003-osc-ffffa1006893e000.cur_grant_bytes=271487350 e109: osc.lsrza-OST0003-osc-ffff8ac65aa26800.cur_grant_bytes=1072398336 e110: osc.lsrza-OST0003-osc-ffff8f737194f800.cur_grant_bytes=1875640319 </pre> </div></div> <p>Olaf, what values do you have on the client(s) for <tt>osc.&#42;.max_pages_per_rpc</tt>, <tt>osc.&#42;.max_rpcs_in_flight</tt>, and <tt>osc.&#42;.max_dirty_mb</tt>? I'm trying to figure out if there is some kind of overflow in the calculation of the max grant. There are several patches in this area that could be related:</p> <ul class="alternate" type="square"> <li><a href="https://review.whamcloud.com/32288" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32288</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-10990" title="Get rid of per-osc max_dirty_mb setting" class="issue-link" data-issue-key="LU-10990"><del>LU-10990</del></a> osc: increase default max_dirty_mb to 2G</tt>" (new in 2.12.0)</li> <li><a href="https://review.whamcloud.com/39380" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39380</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-13763" title="ptlrpc_invalidate_import()) lsrza-OST0000_UUID: Unregistering RPCs found (0). Network is sluggish? Waiting them to error out." class="issue-link" data-issue-key="LU-13763"><del>LU-13763</del></a> osc: don't allow negative grants</tt>" (new in 2.12.6, but I think it is in use at LLNL)</li> <li><a href="https://review.whamcloud.com/35896" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35896</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-12687" title="Fast ENOSPC on direct I/O" class="issue-link" data-issue-key="LU-12687"><del>LU-12687</del></a> osc: consume grants for direct I/O</tt>" (new in 2.12.6, but <em>may</em> in use at LLNL?)</li> </ul> <p>but this is just speculation so far.</p> <p>The other question is whether there are any error messages in the client logs like "<tt>dirty NNNN &gt; dirty_max MMMM</tt>", or related to grant?</p> <p>Some things to try if this is relatively easily reproduced on a client after a fresh mount:</p> <ul class="alternate" type="square"> <li>run "<tt>lctl set_param osc.&#42;.max_dirty_mb=1024</tt>" to limit the amount of dirty data per OSC to see if this prevents the overflow. This should still be below <tt>(max_rpcs_in_flight &#42; max_pages_per_rpc)</tt>, typically <tt>(8 * 16MB = 128MB)</tt> so would not limit performance.</li> </ul> <p>Andreas, we have (I just show OST0003 but the same for all OSTs and all clients)</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>osc.lsrza-OST0003-osc-ffff8b2f37fc1000.max_pages_per_rpc=256 osc.lsrza-OST0003-osc-ffff8b2f37fc1000.max_rpcs_in_flight=8 osc.lsrza-OST0003-osc-ffff8b2f37fc1000.max_dirty_mb=2000 </pre> </div></div> <p>No console log warnings or errors on the clients at all, except for reconnections when we bounced OSTs or MDTs.</p> <p>The current patch stack does not include the negative grants / direct IO patches. It is:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>* 27e1e03 (tag: 2.12.5_5.llnl) log lfs setstripe paths to syslog * f7a3e6e (tag: 2.12.5_4.llnl) LU-13599 mdt: fix mti_big_lmm buffer usage * 8179deb LU-13599 mdt: fix logic of skipping local locks in reply_state * 52a48df (tag: 2.12.5_3.llnl) LU-13709 utils: 'lfs mkdir -i -1' doesn't work * 81479ad LU-13657 kernel: kernel update RHEL8.2 [4.18.0-193.6.3.el8_2] * 2744b2f LU-13421 kernel: kernel update RHEL8.1 [4.18.0-147.8.1.el8_1] * 0aaf383 LU-10395 osd: stop OI at device shutdown * fb8ae25 LU-13766 obdclass: add grant fields to export procfile * e6ee866 (tag: 2.12.5_2.chaos) LU-13667 ptlrpc: fix endless loop issue * 049ed85 LU-11623 llite: hash just created files if lock allows * e1e865f (tag: 2.12.5_1.llnl, tag: 2.12.5_1.chaos) Don't install lustre init script on systemd systems * 6eebb51 LLNL build customizations * 407ac6c TOSS-4431 build: build ldiskfs only for x86_64 * 78d712a (tag: v2_12_5, tag: 2.12.5) New release 2.12.5 * </pre> </div></div> <p>We have several clusters where we see this occur in fairly short order (4-8 hours) after a reboot. But so far we do not have a procedure for reproducing the issue.</p> <p>I'm dropping max_dirty_mb to 1024 on Pascal; however, the CZ is relatively clean now after restarting all the OSTs last week (and remounting Lustre on a handful of uncooperative compute nodes), so I don't think we'd be hitting the issue anytime soon anyway.</p> <p>I will also set it on rztopaz which hasn't had the cleanup.</p> <p>Olaf Faaland-LLNL (faaland1@llnl.gov) uploaded a new patch: <a href="https://review.whamcloud.com/40615" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40615</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14125" title="client starved for grant but OST has plenty of free space" class="issue-link" data-issue-key="LU-14125">LU-14125</a> osc: prevent overflow of o_dropped<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: 968f580554f5211ea78bb825d8666cf70a17c288</p> <p>As another reference point, we seem to be hitting this issue at NOAA as well, with 2.12.3_ddn44 on the clients and lustre-2.12.3_ddn31 on the servers.  Unmounting and remounting a client seems to improve performance, at least for a little while.</p> <p>A colleague found a way to force reconnection between client and server when working a different issue, and I tried it in this case and was able to get cur_grant_bytes to renegotiate without the heavy-handed unmount.  Is this a dangerous thing to do as a workaround?</p> <p><tt><span class="error">&#91;root@Jet:k11 ~&#93;</span># lctl set_param osc.*.grant_shrink=0</tt></p> <p><tt><span class="error">&#91;root@Jet:k11 ~&#93;</span># lctl get_param osc.*.cur_grant_bytes | sort -nk2 -t = | head -n 5</tt><br/> <tt>osc.lfs1-OST0002-osc-ffff8e7aed73c800.cur_grant_bytes=66638</tt><br/> <tt>osc.lfs1-OST0018-osc-ffff8e7aed73c800.cur_grant_bytes=651802</tt><br/> <tt>osc.lfs1-OST0029-osc-ffff8e7aed73c800.cur_grant_bytes=3013787</tt><br/> <tt>osc.lfs1-OST001a-osc-ffff8e7aed73c800.cur_grant_bytes=3146809</tt><br/> <tt>osc.lfs1-OST003b-osc-ffff8e7aed73c800.cur_grant_bytes=3363014</tt></p> <p><tt><span class="error">&#91;root@Jet:k11 ~&#93;</span># lctl device_list | egrep "lfs1-OST00(02|18)"</tt><br/> <tt>  4 UP osc lfs1-OST0018-osc-ffff8e7aed73c800 d39fcae6-1604-7b71-2bba-0ddeb32aa971 4</tt><br/> <tt> 25 UP osc lfs1-OST0002-osc-ffff8e7aed73c800 d39fcae6-1604-7b71-2bba-0ddeb32aa971 4</tt></p> <p><tt><span class="error">&#91;root@Jet:k11 ~&#93;</span># lctl --device 4 deactivate; sleep 1; lctl --device 4 activate</tt><br/> <tt><span class="error">&#91;root@Jet:k11 ~&#93;</span># lctl --device 25 deactivate; sleep 1; lctl --device 25 activate</tt></p> <p><tt><span class="error">&#91;root@Jet:k11 ~&#93;</span># lctl get_param osc.*.cur_grant_bytes | egrep "lfs1-OST00(02|18)"</tt><br/> <tt>osc.lfs1-OST0002-osc-ffff8e7aed73c800.cur_grant_bytes=541450240</tt><br/> <tt>osc.lfs1-OST0018-osc-ffff8e7aed73c800.cur_grant_bytes=542035968</tt></p> <p>Nathan,<br/> You can also likely use "<tt>--device Xxxx recover</tt>" to do the same thing in one step.</p> <p>However, it should be noted that disconnecting and reconnecting the device like this could cause in-flight RPCs to that OST to be aborted, so this should only be done on quiescent clients after "<tt>lctl set_param ldlm.namespaces.&#42;.lru_size=clear</tt>" to flush all dirty data from the client.</p> <p>So it is <em>mostly</em> ok, but not safe to do randomly during the day. </p> <p>Hiya,</p> <p>using the small dd test from our <a href="https://jira.whamcloud.com/browse/LU-14124" title="super slow i/o on client maybe related to low grant" class="issue-link" data-issue-key="LU-14124"><del>LU-14124</del></a> I've been tracking how the issue appears on our compute nodes over time, and setting slow nodes to drain. it's interesting that after they are drained of jobs, perhaps 1/2 or 2/3 of our compute nodes recover and no longer seem to have any grant or i/o issues. but some nodes still have 1 or more slow OSTs even when there are no user processes left on them.</p> <p>before the slow nodes drained of jobs, I used lsof and lfs getstripe to find all files using the slow OSTs from those nodes. in all of these cases there were exe's ('txt' in lsof) or .so's ('mem') using the slow OSTs. sometimes there was a regular file as well, but in most cases the slow OST was only referenced by one exe (eg. 32 references to a 32-way OpenMP exe) or a few .so's (with eg. 22 refs each).</p> <p>(( a caveat that this could be a skewed observation for many reasons - unknown node history prior to going slow / 2 user's jobs are on a disproportionate number of these nodes / there can be a lot of .so's open compared to regular files / ... but it seemed like a pattern. ))</p> <p>so could this issue be something to do with mmap rather than regular i/o?<br/> on the nodes that recovered it looked like these 'txt' and 'mem' files were holding the grant low until the processes ended. no idea why it was some nodes and not others that recovered... perhaps there is more than one thing going on...</p> <p>probably way off-track, but I'm reminded of <a href="https://jira.whamcloud.com/browse/LU-13588" title="sigbus sent to mmap writer that is a long way below quota" class="issue-link" data-issue-key="LU-13588"><del>LU-13588</del></a> which judging from Oleg's comment was something to do with grant and mmap, and perhaps not entirely to do with quota.</p> <p>also FYI max_dirty_mb=1000 doesn't seem to work. we set that on 2 login nodes after a reboot, and one was broken again within a day or so.</p> <p>cheers,<br/> robin</p> <p>I was looking at where the <tt>cl_lost_grant</tt> might be coming from. According to a comment in the code:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> /** * Free grant after IO is finished or canceled. * * @lost_grant is used to remember how many grants we have allocated but not * used, we should <span class="code-keyword">return</span> these grants to OST. There're two cases where grants * can be lost: * 1. truncate; * 2. blocksize at OST is less than PAGE_SIZE and a partial page was * written. In <span class="code-keyword">this</span> <span class="code-keyword">case</span> OST may use less chunks to serve <span class="code-keyword">this</span> partial * write. OSTs don't actually know the page size on the client side. so * clients have to calculate lost grant by the blocksize on the OST. * See filter_grant_check() <span class="code-keyword">for</span> details. */ <span class="code-keyword">static</span> void osc_free_grant(struct client_obd *cli, unsigned <span class="code-object">int</span> nr_pages, unsigned <span class="code-object">int</span> lost_grant, unsigned <span class="code-object">int</span> dirty_grant) </pre> </div></div> <p>so it is possible that the <tt>cl_lost_grant</tt> overflow problem from Olaf's patch <a href="https://review.whamcloud.com/40615" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40615</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-14125" title="client starved for grant but OST has plenty of free space" class="issue-link" data-issue-key="LU-14125">LU-14125</a> osc: prevent overflow of o_dropped</tt>" happens more frequently for ZFS OSTs. I can't imagine that this would be the case for ldiskfs OSTs.</p> <p>There definitely still seems to be some kind of grant accounting error. Even on my idle single-client test system, the client "<tt>osc.&#42;.cur_grant_bytes</tt>" (about 8MB) does not match what the OST "<tt>obdfilter.&#42;.tot_granted</tt>" thinks it has granted (about 256KB).</p> <p>Olaf Faaland-LLNL (faaland1@llnl.gov) uploaded a new patch: <a href="https://review.whamcloud.com/40659" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40659</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14125" title="client starved for grant but OST has plenty of free space" class="issue-link" data-issue-key="LU-14125">LU-14125</a> osc: prevent overflow of o_dropped<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: d80c9e075c0422994865435b35929f9af30476a7</p> <p>Our Lustre 2.12 machines all got updates over the last week, to a new Lustre version that has the two grant patches. Everything started out without the symptom because of the client + server reboots. Still waiting for the issue to reappear.</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/40563/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40563/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14125" title="client starved for grant but OST has plenty of free space" class="issue-link" data-issue-key="LU-14125">LU-14125</a> obdclass: add grant fields to export procfile<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 53ee416097a9a77ca0ee352714af02e77489e3f8</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/40659/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40659/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14125" title="client starved for grant but OST has plenty of free space" class="issue-link" data-issue-key="LU-14125">LU-14125</a> osc: prevent overflow of o_dropped<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 82e9a11056a55289c880786da71d8b1125f357b2</p> <p>The patches have landed for 2.14. Let's reopen if the problem still reappears with them in place.</p> <p>Olaf, and news on the client grant front?</p> <p>Are the patches to prevent overflow of o_dropped and add debugging information available for 2.12.x?  We have a maintenance window coming up, during which we are planning to upgrade clients to 2.12.5+ anyway.</p> <p>There is patch <a href="https://review.whamcloud.com/40615" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40615</a></p> <p>Andreas,<br/> No news. We haven't seen the issue now in about two weeks, but we believe that may be due to the system updates which resulted in everything getting a clean boot.</p> <p>Since my last update in November our production machines for the most part have been running Lustre 2.12.5_10.llnl, which has these grant-related patches landed post 2.12.5:</p> <ul> <li>663e688 <a href="https://jira.whamcloud.com/browse/LU-12687" title="Fast ENOSPC on direct I/O" class="issue-link" data-issue-key="LU-12687"><del>LU-12687</del></a> osc: consume grants for direct I/O</li> <li>ef65452 <a href="https://jira.whamcloud.com/browse/LU-13763" title="ptlrpc_invalidate_import()) lsrza-OST0000_UUID: Unregistering RPCs found (0). Network is sluggish? Waiting them to error out." class="issue-link" data-issue-key="LU-13763"><del>LU-13763</del></a> osc: don't allow negative grants</li> <li>fb8ae25 <a href="https://jira.whamcloud.com/browse/LU-13766" title="tgt_grant_check() lsrza-OST000a: cli dfdf1aff-07d9-53b3-5632-c18a78027eb2 claims 1703936 GRANT, real grant 0" class="issue-link" data-issue-key="LU-13766"><del>LU-13766</del></a> obdclass: add grant fields to export procfile</li> </ul> <p>With that patch set, disabling grant_shrink and then umounting and remounting the OSTs not only gets cur_grant_bytes and tot_granted back in sync but also prevents grant starvation.</p> <p>Our next update coming up in a few weeks will be running 2.12.6_3.llnl which also has:</p> <ul> <li>8d78d2e <a href="https://jira.whamcloud.com/browse/LU-14125" title="client starved for grant but OST has plenty of free space" class="issue-link" data-issue-key="LU-14125">LU-14125</a> osc: prevent overflow of o_dropped</li> </ul> <p>After that update we will re-enable grant_shrink and see if the problem recurs.</p> <p>&gt; fb8ae25 <a href="https://jira.whamcloud.com/browse/LU-13766" title="tgt_grant_check() lsrza-OST000a: cli dfdf1aff-07d9-53b3-5632-c18a78027eb2 claims 1703936 GRANT, real grant 0" class="issue-link" data-issue-key="LU-13766"><del>LU-13766</del></a> obdclass: add grant fields to export procfile<br/> <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=ofaaland" class="user-hover" rel="ofaaland">ofaaland</a>, can you say more about that one? I don't see it posted to gerrit yet.</p> <p>Hi Cory, </p> <p><a href="https://review.whamcloud.com/#/c/39324/5" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/39324/5</a></p> <p>that patch has been on our 2.12 stack at LLNL for several weeks and hasn't caused us any trouble. I'm not sure why it hasn't landed to b2_12. I added gerrit gatekeeper as a reviewer and posted a query to that effect.</p> <p>The landings to b2_12 have been paused while we focus efforts on getting 2.14.0 out the door. There are a number of patches queued up for 2.12.7 once Oleg has cycles to run them through his test rig.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=ofaaland" class="user-hover" rel="ofaaland">ofaaland</a>, ah, thanks for pointing it out. That patch is from <b>this</b> ticket and not directly from <a href="https://jira.whamcloud.com/browse/LU-13766" title="tgt_grant_check() lsrza-OST000a: cli dfdf1aff-07d9-53b3-5632-c18a78027eb2 claims 1703936 GRANT, real grant 0" class="issue-link" data-issue-key="LU-13766"><del>LU-13766</del></a>. I just didn't see a patch in gerrit for <a href="https://jira.whamcloud.com/browse/LU-13766" title="tgt_grant_check() lsrza-OST000a: cli dfdf1aff-07d9-53b3-5632-c18a78027eb2 claims 1703936 GRANT, real grant 0" class="issue-link" data-issue-key="LU-13766"><del>LU-13766</del></a> (<a href="https://review.whamcloud.com/#/q/LU-13766" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/q/LU-13766</a>) and thought I was missing something. Thanks for letting us know!</p> <p>My update from Jan 5 said:</p> <blockquote><p>Our next update coming up in a few weeks will be running 2.12.6_3.llnl which also has:</p> <ul> <li>* 8d78d2e <a href="https://jira.whamcloud.com/browse/LU-14125" title="client starved for grant but OST has plenty of free space" class="issue-link" data-issue-key="LU-14125">LU-14125</a> osc: prevent overflow of o_dropped</li> </ul> <p>After that update we will re-enable grant_shrink and see if the problem recurs.</p></blockquote> <p>Our updates are delayed so it will be another week or two before we can re-enable grant_shrink and then another couple of weeks before we see if the symptoms reappear.</p> <p>I'm reopening this, because I don't think it is clear the problem has been fixed. I believe it was just closed because the two <a href="https://jira.whamcloud.com/browse/LU-14125" title="client starved for grant but OST has plenty of free space" class="issue-link" data-issue-key="LU-14125">LU-14125</a> patches were landed for 2.14.0.</p> <p>Another site reported that just the <tt>o_dropped</tt> does not seem to have (fully?) resolved the problem, since they are still seeing clients with low grant. I'm still trying to find out whether the site is running with <tt>grant_shrink=1</tt> or not.</p> <p>One theory that I had was that something with the grant shrink is causing incremental loss of grant because it is using non-PAGE_SIZE grant amounts? I notice that the clients and servers have values that are not even PAGE_SIZE multiples, so maybe there is some kind of rounding problems between the client and server?</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/39324/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39324/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14125" title="client starved for grant but OST has plenty of free space" class="issue-link" data-issue-key="LU-14125">LU-14125</a> obdclass: add grant fields to export procfile<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: 7a354e82d99d57103ed52cb7872cd64090b43383</p> <p>FWIW, we have a couple of reports. One customer running 2.12.6 saw trouble with performance. One set of nodes was set to grant_shrink=0, which mitigated the problem. Another customer running ~2.12.4 saw their grant problems disappear with this patch and <a href="https://jira.whamcloud.com/browse/LU-11409" title="only first 100 OSC can shrink grants" class="issue-link" data-issue-key="LU-11409"><del>LU-11409</del></a>. Related to <a href="https://jira.whamcloud.com/browse/LU-11409" title="only first 100 OSC can shrink grants" class="issue-link" data-issue-key="LU-11409"><del>LU-11409</del></a>: <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=vsaveliev" class="user-hover" rel="vsaveliev">vsaveliev</a> has spotted that tgt_grant_sanity_check() is a no-op and the grant check gets turned off with more than 100 exports.</p> <p>Cory, thanks for the update. You may be conflating two issues here. <a href="https://jira.whamcloud.com/browse/LU-11409" title="only first 100 OSC can shrink grants" class="issue-link" data-issue-key="LU-11409"><del>LU-11409</del></a> is applicable to clients connecting to more than 100 OSTs. It would be useful to know from the reporters on this ticket if that applies to the systems where this problem is being seen.</p> <p>Separately, <tt>tgt_grant_sanity_check()</tt> is a server-side verification of the grants given to the clients vs. the total granted counters for the target at disconnect time, which is disabled on systems with more than 100 connected clients because it adds significant overhead at that point (O(n^2) with the number of connected clients). However, that check is not being triggered in this case (AFAIK) because it isn't a problem with the per-export vs. global counters on the OST, but a disconnect between what the client is counting and what the server is counting. </p> <blockquote><p>Cory, thanks for the update. You may be conflating two issues here. <a href="https://jira.whamcloud.com/browse/LU-11409" title="only first 100 OSC can shrink grants" class="issue-link" data-issue-key="LU-11409"><del>LU-11409</del></a> is applicable to clients connecting to more than 100 OSTs. It would be useful to know from the reporters on this ticket if that applies to the systems where this problem is being seen.</p></blockquote> <p>At LLNL the clients where we're seeing this problem do connect to more than 100 OSTs (across 3 file systems).</p> <p>Olaf, the backported patch <a href="https://review.whamcloud.com/40564" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40564</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-11409" title="only first 100 OSC can shrink grants" class="issue-link" data-issue-key="LU-11409"><del>LU-11409</del></a> osc: grant shrink shouldn't account skipped OSC</tt>" just landed to b2_12 but is not in any tag yet. This is a very simple client-only patch, so could be added to your clients relatively easily to see if it solves the problem. </p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/40615/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40615/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14125" title="client starved for grant but OST has plenty of free space" class="issue-link" data-issue-key="LU-14125">LU-14125</a> osc: prevent overflow of o_dropped<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: 1da8349175a136df0aadb28ae0e0f64ac0385961</p> <p>There is a new patch <a href="https://review.whamcloud.com/42129" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/42129</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-14543" title="tgt_grant_discard(): avoid tgd-&gt;tgd_tot_granted overflowing" class="issue-link" data-issue-key="LU-14543"><del>LU-14543</del></a> target: prevent overflowing of tgd-&gt;tgd_tot_granted</tt>" that may be of interest here. I'm not 100% sure it is related, since it involves an <b>underflow</b> of <tt>tot_grant</tt> and/or <tt>tot_dirty</tt> AFAICS, so if that happened it would likely affect all clients, but worthwhile to mention it here.</p> <p>We've seen the issue at LLNL again:</p> <ul> <li>with 40615 "prevent overflow of o_dropped"</li> <li>without 40564 "grant shrink shouldn't account skipped OSC"</li> <li>without 42129 "prevent overflowing of tgd-&gt;tgd_tot_granted"</li> <li>with grant_shrink enabled</li> </ul> <p>The same clients mounted another file system which had grant_shrink <em>disabled</em> and those OSCs did <em>not</em> encounter the issue.</p> <p>Our clients will get 40564 "grant shrink shouldn't account skipped OSC" in the next few weeks, but it typically takes weeks for the issue to become easily detectable.</p> <p> </p> Duplicate LU-14124 Related LU-11409 LU-13766 LU-14901 LU-9704 LU-14543 Development Rank 1|i01emn: Rank (Obsolete) 9223372036854775807 Severity