https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11939 Lustre <p>We just hit the following LBUG with Lustre 2.12 on an OSS (Fir). All clients are running Lustre 2.12 also.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[1708550.581820] LustreError: 123124:0:(tgt_grant.c:1079:tgt_grant_discard()) ASSERTION( tgd-&gt;tgd_tot_granted &gt;= ted-&gt;ted_grant ) failed: fir-OST001b: tot_granted 50041695803 cli d5e4b60f-fe33-b991-7d48-5b8db7e07ab0/ffff926b10975c00 ted_grant -49152 [1708550.603611] LustreError: 123124:0:(tgt_grant.c:1079:tgt_grant_discard()) LBUG [1708550.610923] Pid: 123124, comm: ll_ost00_019 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [1708550.621180] Call Trace: [1708550.623814] [&lt;ffffffffc0aa37cc&gt;] libcfs_call_trace+0x8c/0xc0 [libcfs] [1708550.630548] [&lt;ffffffffc0aa387c&gt;] lbug_with_loc+0x4c/0xa0 [libcfs] [1708550.636935] [&lt;ffffffffc0f220bc&gt;] tgt_grant_discard+0x1dc/0x1e0 [ptlrpc] [1708550.643892] [&lt;ffffffffc14c81d4&gt;] ofd_obd_disconnect+0x74/0x220 [ofd] [1708550.650541] [&lt;ffffffffc0e60157&gt;] target_handle_disconnect+0xd7/0x450 [ptlrpc] [1708550.658005] [&lt;ffffffffc0efeb77&gt;] tgt_disconnect+0x37/0x140 [ptlrpc] [1708550.664609] [&lt;ffffffffc0f0635a&gt;] tgt_request_handle+0xaea/0x1580 [ptlrpc] [1708550.671734] [&lt;ffffffffc0eaa92b&gt;] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [1708550.679628] [&lt;ffffffffc0eae25c&gt;] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [1708550.686136] [&lt;ffffffff8dcc1c31&gt;] kthread+0xd1/0xe0 [1708550.691224] [&lt;ffffffff8e374c24&gt;] ret_from_fork_nospec_begin+0xe/0x21 [1708550.697873] [&lt;ffffffffffffffff&gt;] 0xffffffffffffffff [1708550.703065] Kernel panic - not syncing: LBUG [1708550.707509] CPU: 20 PID: 123124 Comm: ll_ost00_019 Kdump: loaded Tainted: G OE ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [1708550.720273] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [1708550.728015] Call Trace: [1708550.730645] [&lt;ffffffff8e361e41&gt;] dump_stack+0x19/0x1b [1708550.735962] [&lt;ffffffff8e35b550&gt;] panic+0xe8/0x21f [1708550.740937] [&lt;ffffffffc0aa38cb&gt;] lbug_with_loc+0x9b/0xa0 [libcfs] [1708550.747346] [&lt;ffffffffc0f220bc&gt;] tgt_grant_discard+0x1dc/0x1e0 [ptlrpc] [1708550.754230] [&lt;ffffffffc14c81d4&gt;] ofd_obd_disconnect+0x74/0x220 [ofd] [1708550.760880] [&lt;ffffffffc0e9ed81&gt;] ? lustre_pack_reply+0x11/0x20 [ptlrpc] [1708550.767783] [&lt;ffffffffc0ec3933&gt;] ? req_capsule_server_pack+0x43/0xf0 [ptlrpc] [1708550.775207] [&lt;ffffffffc0e60157&gt;] target_handle_disconnect+0xd7/0x450 [ptlrpc] [1708550.782634] [&lt;ffffffffc0efeb77&gt;] tgt_disconnect+0x37/0x140 [ptlrpc] [1708550.789194] [&lt;ffffffffc0f0635a&gt;] tgt_request_handle+0xaea/0x1580 [ptlrpc] [1708550.796272] [&lt;ffffffffc0edfa51&gt;] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [1708550.804022] [&lt;ffffffffc0aa3bde&gt;] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [1708550.811281] [&lt;ffffffffc0eaa92b&gt;] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [1708550.819142] [&lt;ffffffffc0ea77b5&gt;] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc] [1708550.826110] [&lt;ffffffff8dcd67c2&gt;] ? default_wake_function+0x12/0x20 [1708550.832548] [&lt;ffffffff8dccba9b&gt;] ? __wake_up_common+0x5b/0x90 [1708550.838589] [&lt;ffffffffc0eae25c&gt;] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [1708550.845068] [&lt;ffffffffc0ead760&gt;] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc] [1708550.852636] [&lt;ffffffff8dcc1c31&gt;] kthread+0xd1/0xe0 [1708550.857688] [&lt;ffffffff8dcc1b60&gt;] ? insert_kthread_work+0x40/0x40 [1708550.863956] [&lt;ffffffff8e374c24&gt;] ret_from_fork_nospec_begin+0xe/0x21 [1708550.870567] [&lt;ffffffff8dcc1b60&gt;] ? insert_kthread_work+0x40/0x40 </pre> </div></div> CentOS 7.6, 3.10.0-957.1.3.el7_lustre.x86_64 LU-11939

ASSERTION( tgd->tgd_tot_granted >= ted->ted_grant ) on OSS

Bug Critical Resolved Fixed Mikhail Pershin Stephane Thiell Wed, 6 Feb 2019 21:25:33 +0000 Mon, 22 Nov 2021 15:52:40 +0000 Thu, 18 Nov 2021 00:11:57 +0000 Lustre 2.12.0 Lustre 2.14.0 Lustre 2.12.8 0 7 <p>Same crash happened on OST unmount while trying to fix another issue (@tcp announcing clients, <a href="https://jira.whamcloud.com/browse/LU-11888" title="Unreachable client NID confusing Lustre 2.12" class="issue-link" data-issue-key="LU-11888">LU-11888</a>):</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[1739660.322150] Lustre: Failing over fir-OST000b [1739660.326649] Lustre: Skipped 5 previous similar messages [1739660.337361] LustreError: 24209:0:(tgt_grant.c:1079:tgt_grant_discard()) ASSERTION( tgd-&gt;tgd_tot_granted &gt;= ted-&gt;ted_grant ) failed: fir-OST0005: tot_granted 2114072895 cli e43da944-d239-923f-8f68-10646264727b/ffff90a28977ac00 ted_grant -12582912 [1739660.359265] LustreError: 24209:0:(tgt_grant.c:1079:tgt_grant_discard()) LBUG [1739660.366492] Pid: 24209, comm: umount 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [1739660.376143] Call Trace: [1739660.378780] [&lt;ffffffffc0afd7cc&gt;] libcfs_call_trace+0x8c/0xc0 [libcfs] [1739660.385519] [&lt;ffffffffc0afd87c&gt;] lbug_with_loc+0x4c/0xa0 [libcfs] [1739660.391906] [&lt;ffffffffc13fb0bc&gt;] tgt_grant_discard+0x1dc/0x1e0 [ptlrpc] [1739660.398876] [&lt;ffffffffc12631d4&gt;] ofd_obd_disconnect+0x74/0x220 [ofd] [1739660.405522] [&lt;ffffffffc0d2e7d6&gt;] class_disconnect_export_list+0x1c6/0x680 [obdclass] [1739660.413582] [&lt;ffffffffc0d2eda5&gt;] class_disconnect_exports+0x115/0x310 [obdclass] [1739660.421279] [&lt;ffffffffc0d493e7&gt;] class_cleanup+0x297/0xbd0 [obdclass] [1739660.428031] [&lt;ffffffffc0d4a9ac&gt;] class_process_config+0x65c/0x2830 [obdclass] [1739660.435465] [&lt;ffffffffc0d4cd46&gt;] class_manual_cleanup+0x1c6/0x710 [obdclass] [1739660.442816] [&lt;ffffffffc0d7dc2e&gt;] server_put_super+0x8de/0xcd0 [obdclass] [1739660.449827] [&lt;ffffffff84243dbd&gt;] generic_shutdown_super+0x6d/0x100 [1739660.456303] [&lt;ffffffff842441b2&gt;] kill_anon_super+0x12/0x20 [1739660.462077] [&lt;ffffffffc0d4f8b2&gt;] lustre_kill_super+0x32/0x50 [obdclass] [1739660.468989] [&lt;ffffffff8424456e&gt;] deactivate_locked_super+0x4e/0x70 [1739660.475456] [&lt;ffffffff84244cf6&gt;] deactivate_super+0x46/0x60 [1739660.481314] [&lt;ffffffff8426326f&gt;] cleanup_mnt+0x3f/0x80 [1739660.486739] [&lt;ffffffff84263302&gt;] __cleanup_mnt+0x12/0x20 [1739660.492338] [&lt;ffffffff840be79b&gt;] task_work_run+0xbb/0xe0 [1739660.497948] [&lt;ffffffff8402bc65&gt;] do_notify_resume+0xa5/0xc0 [1739660.503813] [&lt;ffffffff84775124&gt;] int_signal+0x12/0x17 [1739660.509154] [&lt;ffffffffffffffff&gt;] 0xffffffffffffffff [1739660.514347] Kernel panic - not syncing: LBUG [1739660.518793] CPU: 0 PID: 24209 Comm: umount Kdump: loaded Tainted: G OE ------------ 3.10.0-957.1.3.el7_lustre.x86_64 #1 [1739660.530863] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.6.7 10/29/2018 [1739660.538601] Call Trace: [1739660.541232] [&lt;ffffffff84761e41&gt;] dump_stack+0x19/0x1b [1739660.546543] [&lt;ffffffff8475b550&gt;] panic+0xe8/0x21f [1739660.551512] [&lt;ffffffffc0afd8cb&gt;] lbug_with_loc+0x9b/0xa0 [libcfs] [1739660.557903] [&lt;ffffffffc13fb0bc&gt;] tgt_grant_discard+0x1dc/0x1e0 [ptlrpc] [1739660.564783] [&lt;ffffffffc12631d4&gt;] ofd_obd_disconnect+0x74/0x220 [ofd] [1739660.571411] [&lt;ffffffffc0c89f02&gt;] ? libcfs_nid2str_r+0xe2/0x130 [lnet] [1739660.578124] [&lt;ffffffffc0d2e7d6&gt;] class_disconnect_export_list+0x1c6/0x680 [obdclass] [1739660.586143] [&lt;ffffffffc0d2eda5&gt;] class_disconnect_exports+0x115/0x310 [obdclass] [1739660.593814] [&lt;ffffffffc0d493e7&gt;] class_cleanup+0x297/0xbd0 [obdclass] [1739660.600517] [&lt;ffffffffc0b03f07&gt;] ? libcfs_debug_msg+0x57/0x80 [libcfs] [1739660.607324] [&lt;ffffffffc0d30836&gt;] ? class_name2dev_nolock+0x46/0xb0 [obdclass] [1739660.614735] [&lt;ffffffffc0d4a9ac&gt;] class_process_config+0x65c/0x2830 [obdclass] [1739660.622131] [&lt;ffffffffc0b03f07&gt;] ? libcfs_debug_msg+0x57/0x80 [libcfs] [1739660.628939] [&lt;ffffffffc0d4cd46&gt;] class_manual_cleanup+0x1c6/0x710 [obdclass] [1739660.636265] [&lt;ffffffffc0d7dc2e&gt;] server_put_super+0x8de/0xcd0 [obdclass] [1739660.643223] [&lt;ffffffff84243dbd&gt;] generic_shutdown_super+0x6d/0x100 [1739660.649661] [&lt;ffffffff842441b2&gt;] kill_anon_super+0x12/0x20 [1739660.655427] [&lt;ffffffffc0d4f8b2&gt;] lustre_kill_super+0x32/0x50 [obdclass] [1739660.662296] [&lt;ffffffff8424456e&gt;] deactivate_locked_super+0x4e/0x70 [1739660.668736] [&lt;ffffffff84244cf6&gt;] deactivate_super+0x46/0x60 [1739660.674571] [&lt;ffffffff8426326f&gt;] cleanup_mnt+0x3f/0x80 [1739660.679968] [&lt;ffffffff84263302&gt;] __cleanup_mnt+0x12/0x20 [1739660.685541] [&lt;ffffffff840be79b&gt;] task_work_run+0xbb/0xe0 [1739660.691115] [&lt;ffffffff8402bc65&gt;] do_notify_resume+0xa5/0xc0 [1739660.696947] [&lt;ffffffff84775124&gt;] int_signal+0x12/0x17 </pre> </div></div> <p>vmcore is available if needed</p> <p>Patrick is investigating</p> <p>VMcore would be welcome, first off, dmesg in both cases would be great to have.</p> <p>Few questions.</p> <ol> <li>OST backing file system - ZFS or ldiskfs?</li> <li>Any evictions?  (Hoping to see that in dmesg)</li> </ol> <p>Hi Patrick,</p> <p>Thanks for investigating and congrats on your new position!! <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>1. ldiskfs<br/> 2. many evictions due to clients configured with tcp0 nids even though the servers are IB only, see <a href="https://jira.whamcloud.com/browse/LU-11888" title="Unreachable client NID confusing Lustre 2.12" class="issue-link" data-issue-key="LU-11888">LU-11888</a>/<a href="https://jira.whamcloud.com/browse/LU-11937" title="lnet.service randomly load tcp NIDs" class="issue-link" data-issue-key="LU-11937"><del>LU-11937</del></a></p> <p>Attached the dmesg to the ticket:</p> <ul> <li>first crash is vmcore-dmesg_fir-io3-s2_2019-02-06_12_50_35.txt</li> <li>second crash while unmounting is vmcore-dmesg_fir-io1-s2_2019-02-06_21_29_56.txt</li> </ul> <p>I also just uploaded the two vmcores to the ftp:</p> <ul> <li>vmcore_fir-io3-s2_2019-02-06_12_50_35.gz</li> <li>vmcore_fir-io1-s2_2019-02-06_21_29_56.gz<br/> I believe the debuginfo rpms are already available in the ftp:</li> <li>kernel-debuginfo-3.10.0-957.1.3.el7_lustre.x86_64.rpm</li> <li>kernel-debuginfo-common-x86_64-3.10.0-957.1.3.el7_lustre.x86_64.rpm<br/> and I also uploaded our lustre debuginfo rpm:</li> <li>lustre-debuginfo-2.12.0-1.el7.x86_64.rpm</li> </ul> <p>No more crash so far since we restarted all servers and fixed our clients announcing *@tcp NIDs.</p> <p>Thanks!<br/> Stephane</p> <p>Thanks, Stephane!</p> <p>So as you've probably guessed, there's some sort of bug related to grant handling on evictions.  If you're not having evictions, you shouldn't see this.  So you're probably good to go from a "site reliability" perspective.</p> <p>I'm going to look in to the grant behavior at eviction, and also see about not asserting here.  Clean up and print an error instead.</p> <p>Patrick Farrell (pfarrell@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/34215" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34215</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11939" title="ASSERTION( tgd-&gt;tgd_tot_granted &gt;= ted-&gt;ted_grant ) on OSS" class="issue-link" data-issue-key="LU-11939"><del>LU-11939</del></a> tgt: Do not assert during grant cleanup<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: f44d793b8a4786042a4cf38cf967ce9686da5b81</p> <p>Stephane,</p> <p>Looking a little in to the underlying bug...</p> <p>What are your max_pages_per_rpc, max_dirty_mb, and max_rpcs_in_flight settings on the client?</p> <p>Thanks Patrick!</p> <p>On our clients, we have:</p> <ul class="alternate" type="square"> <li>max_rpcs_in_flight=8 (default), only the data transfer nodes and robinhood server have max_rpcs_in_flight=32</li> <li>max_dirty_mb=2000 (default), only the data transfer nodes and robinhood server have max_dirty_mb=128</li> </ul> <p>As for max_pages_per_rpc, it should be set to 4096 and brw_size=16, but I noticed that it doesn't seem to be the case on all clients:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@sh-101-20 ~]# cd /proc/fs/lustre/osc; for o in fir-*; do echo -n "$o:"; cat $o/max_pages_per_rpc; done fir-OST0000-osc-ffff9d0cad3de000:4096 fir-OST0001-osc-ffff9d0cad3de000:4096 fir-OST0002-osc-ffff9d0cad3de000:1024 fir-OST0003-osc-ffff9d0cad3de000:1024 fir-OST0004-osc-ffff9d0cad3de000:1024 fir-OST0005-osc-ffff9d0cad3de000:1024 fir-OST0006-osc-ffff9d0cad3de000:1024 fir-OST0007-osc-ffff9d0cad3de000:4096 fir-OST0008-osc-ffff9d0cad3de000:1024 fir-OST0009-osc-ffff9d0cad3de000:1024 fir-OST000a-osc-ffff9d0cad3de000:1024 fir-OST000b-osc-ffff9d0cad3de000:1024 fir-OST000c-osc-ffff9d0cad3de000:1024 fir-OST000d-osc-ffff9d0cad3de000:1024 fir-OST000e-osc-ffff9d0cad3de000:1024 fir-OST000f-osc-ffff9d0cad3de000:1024 fir-OST0010-osc-ffff9d0cad3de000:4096 fir-OST0011-osc-ffff9d0cad3de000:4096 fir-OST0012-osc-ffff9d0cad3de000:4096 fir-OST0013-osc-ffff9d0cad3de000:1024 fir-OST0014-osc-ffff9d0cad3de000:1024 fir-OST0015-osc-ffff9d0cad3de000:4096 fir-OST0016-osc-ffff9d0cad3de000:1024 fir-OST0017-osc-ffff9d0cad3de000:1024 fir-OST0018-osc-ffff9d0cad3de000:4096 fir-OST0019-osc-ffff9d0cad3de000:4096 fir-OST001a-osc-ffff9d0cad3de000:1024 fir-OST001b-osc-ffff9d0cad3de000:1024 fir-OST001c-osc-ffff9d0cad3de000:4096 fir-OST001d-osc-ffff9d0cad3de000:4096 fir-OST001e-osc-ffff9d0cad3de000:1024 fir-OST001f-osc-ffff9d0cad3de000:1024 fir-OST0020-osc-ffff9d0cad3de000:4096 fir-OST0021-osc-ffff9d0cad3de000:4096 fir-OST0022-osc-ffff9d0cad3de000:1024 fir-OST0023-osc-ffff9d0cad3de000:1024 fir-OST0024-osc-ffff9d0cad3de000:1024 fir-OST0025-osc-ffff9d0cad3de000:1024 fir-OST0026-osc-ffff9d0cad3de000:1024 fir-OST0027-osc-ffff9d0cad3de000:1024 fir-OST0028-osc-ffff9d0cad3de000:1024 fir-OST0029-osc-ffff9d0cad3de000:1024 fir-OST002a-osc-ffff9d0cad3de000:4096 fir-OST002b-osc-ffff9d0cad3de000:4096 fir-OST002c-osc-ffff9d0cad3de000:4096 fir-OST002d-osc-ffff9d0cad3de000:1024 fir-OST002e-osc-ffff9d0cad3de000:1024 fir-OST002f-osc-ffff9d0cad3de000:1024 </pre> </div></div> <p>We used this on the MGS:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lctl set_param -P fir-OST*.osc.max_pages_per_rpc=4096 </pre> </div></div> <p>So this isn't good. I just re-applied this command on the MGS:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[138882.544463] Lustre: Modifying parameter osc.fir-OST*.osc.max_pages_per_rpc in log params </pre> </div></div> <p>and a newly mounted client is now set up at 4096. Do you think that could have caused this issue?</p> <p>As for brw_size:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># clush -w @oss -b 'cat /proc/fs/lustre/obdfilter/*/brw_size' --------------- fir-io[1-4]-s[1-2] (8) --------------- 16 16 16 16 16 16 </pre> </div></div> <p>Pages per RPC isn't a big deal (so, no, probably not), but max_dirty_mb may be.</p> <p>Hmm, 2000 is <b>not</b> the default (the default is, I think, max_rpcs_in_flight * RPC size...  It's certainly much smaller than this value.).  So that's getting set somewhere.</p> <p>It's also potentially too high.  max_dirty_mb is used in calculating grant requests, and there are some grant overflow bugs that occur when it's set that high.  (Particularly with 16 MiB RPCs.)  All the ones I know of are fixed in 2.12, but...</p> <p> </p> <p>I strongly suspect you may have hit an overflow, leading to the grant inconsistency, leading to this crash. </p> <p>The grant value reported for the export for this client is negative - ted_grant -49152 in one case, in the other ted_grant -12582912.  These small-ish negative values strongly suggest overflow.  The server side value being compared against (tot_granted) is unsigned, and comparison with this negative value is why the "total grant &gt;= grant for this export" assert we hit failed.  (The fact that your max_dirty_mb is at 2 GiB just makes the overflow explanation more likely.)</p> <p> </p> <p>Your max_dirty_mb is above what should help performance, so tuning it down is a good idea.</p> <p>The rule I use for max_dirty_mb is 2 * mb_per_rpc * rpcs_in_flight - the idea being that you can accumulate some dirty data so you're always ready to make an RPC when one completes, but you don't have tons of dirty data sitting around if it isn't getting processed fast enough.  (There's some docs floating around that say 4*, but with RPC sizes and counts increasing, that tends to be too much data.  2* should be plenty for good performance.)</p> <p>So for you, that's 2*16*8 = 256, or in the case of your RBH nodes, that's 2*16*32=1024.</p> <p>So I'd suggest turning down your max_dirty_mb to no more than 1 GiB.</p> <p>Wow, thanks much for the detailed explanation. This is SUPER helpful. But... I don't think we have explicitly changed the value of <tt>max_dirty_mb</tt>. So I've been trying to track down why it is so high for ALL of our Lustre filesystems mounted on Sherlock (regal, oak and fir). If I understand correctly, <tt>/sys/fs/lustre/max_dirty_mb</tt> is used by the udev script provided by the lustre-client RPM right? and then the values probably max out at 2000?<br/>  </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@sh-ln06 ~]# cat /sys/fs/lustre/version 2.12.0 [root@sh-ln06 ~]# cat /sys/fs/lustre/max_dirty_mb 32107 [root@sh-ln06 ~]# ls -l /sys/fs/lustre/max_dirty_mb -rw-r--r-- 1 root root 4096 Feb  8 16:54 /sys/fs/lustre/max_dirty_mb [root@sh-ln06 ~]# cat /etc/udev/rules.d/99-lustre.rules KERNEL=="obd", MODE="0666" # set sysfs values on client SUBSYSTEM=="lustre", ACTION=="change", ENV{PARAM}=="?*", RUN+="/usr/sbin/lctl set_param '$env{PARAM}=$env{SETTING}'" [root@sh-ln06 ~]# rpm -q --info lustre-client Name : lustre-client Version : 2.12.0 Release : 1.el7 Architecture: x86_64 Install Date: Tue 05 Feb 2019 05:17:58 PM PST Group : System Environment/Kernel Size : 2007381 License : GPL Signature : (none) Source RPM : lustre-client-2.12.0-1.el7.src.rpm Build Date : Fri 21 Dec 2018 01:53:18 PM PST Build Host : trevis-307-el7-x8664-3.trevis.whamcloud.com Relocations : (not relocatable) URL : https://wiki.whamcloud.com/ Summary : Lustre File System Description : Userspace tools and files for the Lustre file system. [root@sh-ln06 ~]# lctl get_param osc.*.max_dirty_mb osc.fir-OST0000-osc-ffff9bad01395000.max_dirty_mb=2000 osc.fir-OST0001-osc-ffff9bad01395000.max_dirty_mb=2000 ... osc.fir-OST002e-osc-ffff9bad01395000.max_dirty_mb=2000 osc.fir-OST002f-osc-ffff9bad01395000.max_dirty_mb=2000 osc.oak-OST0000-osc-ffff9baceaa3d800.max_dirty_mb=2000 osc.oak-OST0001-osc-ffff9baceaa3d800.max_dirty_mb=2000 ... osc.oak-OST0071-osc-ffff9baceaa3d800.max_dirty_mb=2000 osc.regal-OST0000-osc-ffff9bace6e28800.max_dirty_mb=2000 ... osc.regal-OST006a-osc-ffff9bace6e28800.max_dirty_mb=2000 osc.regal-OST006b-osc-ffff9bace6e28800.max_dirty_mb=2000 </pre> </div></div> <p> </p> <p>Hmm, I'm not familiar with the script, so I don't really know.  I don't think so, though...?</p> <p>It's possible you're hitting:<br/> <a href="https://jira.whamcloud.com/browse/LU-11919" class="external-link" rel="nofollow">https://jira.whamcloud.com/browse/LU-11919</a></p> <p>Which is basically "cl_max_dirty_mb is supposed to start at zero, but instead starts with whatever was in memory".  Then, whatever was in memory is processed like it was a setting from userspace.  So if it's not zero (the most likely case, especially at startup), it's reasonably likely (though not guaranteed - it's more complicated than just "existing value in memory is &gt; 2000 means 2000") to get set to the max.</p> <p>Anyway, you can override that with a set_param -P.</p> <p>OK, we never noticed that before (with 2.10 clients). Thanks for your help! I used set_param -P on the MGS of Fir to set max_dirty_mb to 256 and it did work.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lctl set_param -P osc.*.max_dirty_mb=256 </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@sh-ln06 ~]# lctl get_param osc.*.max_dirty_mb osc.fir-OST0000-osc-ffff9bad01395000.max_dirty_mb=256 osc.fir-OST0001-osc-ffff9bad01395000.max_dirty_mb=256 ... osc.fir-OST002e-osc-ffff9bad01395000.max_dirty_mb=256 osc.fir-OST002f-osc-ffff9bad01395000.max_dirty_mb=256 osc.oak-OST0000-osc-ffff9baceaa3d800.max_dirty_mb=256 osc.oak-OST0001-osc-ffff9baceaa3d800.max_dirty_mb=256 ... osc.oak-OST0070-osc-ffff9baceaa3d800.max_dirty_mb=256 osc.oak-OST0071-osc-ffff9baceaa3d800.max_dirty_mb=256 osc.regal-OST0000-osc-ffff9bace6e28800.max_dirty_mb=256 osc.regal-OST0001-osc-ffff9bace6e28800.max_dirty_mb=256 osc.regal-OST0002-osc-ffff9bace6e28800.max_dirty_mb=256 ... osc.regal-OST006b-osc-ffff9bace6e28800.max_dirty_mb=256 </pre> </div></div> <p>So that should be much better. I'll report any new event regarding this issue, but so far so good. Thanks again.</p> <p>So ok to close this one as a duplicate of <a href="https://jira.whamcloud.com/browse/LU-11919" title="cl_max_dirty_pages not zeroed on startup" class="issue-link" data-issue-key="LU-11919"><del>LU-11919</del></a>?</p> <p>Nah, we've still got a patch to track under this</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=tappro" class="user-hover" rel="tappro">tappro</a>:</p> <p>Mike,</p> <p>Didn't you fix this grant bug in another LU?  I can't find it right now...</p> <p>Patrick, do you mean patch from <a href="https://jira.whamcloud.com/browse/LU-12120" title="LustreError: 15069:0:(tgt_grant.c:561:tgt_grant_incoming()) LBUG " class="issue-link" data-issue-key="LU-12120"><del>LU-12120</del></a>?</p> <p>Yes, that looks like the right one.  Do you agree that should take care of this issue as well?</p> <p>Mike confirms that this is a duplicate of <a href="https://jira.whamcloud.com/browse/LU-12120" title="LustreError: 15069:0:(tgt_grant.c:561:tgt_grant_incoming()) LBUG " class="issue-link" data-issue-key="LU-12120"><del>LU-12120</del></a></p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/34215/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34215/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11939" title="ASSERTION( tgd-&gt;tgd_tot_granted &gt;= ted-&gt;ted_grant ) on OSS" class="issue-link" data-issue-key="LU-11939"><del>LU-11939</del></a> tgt: Do not assert during grant cleanup<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: af2d3ac30eafead6b47c5db20d76433c091d89de</p> <p>"Mike Pershin &lt;mpershin@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/45489" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/45489</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11939" title="ASSERTION( tgd-&gt;tgd_tot_granted &gt;= ted-&gt;ted_grant ) on OSS" class="issue-link" data-issue-key="LU-11939"><del>LU-11939</del></a> tgt: Do not assert during grant cleanup<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: 81f17cf04fc7d4d4bd7ab87cfe572b7f59cf81f3</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/45489/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/45489/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11939" title="ASSERTION( tgd-&gt;tgd_tot_granted &gt;= ted-&gt;ted_grant ) on OSS" class="issue-link" data-issue-key="LU-11939"><del>LU-11939</del></a> tgt: Do not assert during grant cleanup<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: 372c77f0a11573e9f8818751c24735e151aafc74</p> Duplicate LU-12120 Related Development Rank 1|i00b5r: Rank (Obsolete) 9223372036854775807 Severity