https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2499 Lustre <p>We are seeing the following error.</p> <p>Dec 13 08:35:39 nbp2-oss1 kernel: LustreError: 0:0:(ldlm_lockd.c:358:waiting_locks_callback()) ### lock callback timer expired after 351s: evicting client at 10.151.34.219@o2ib ns: filter-nbp2-OST0018_UUID lock: ffff8804c55d8480/0x1ca7e7e6c780ff4d lrc: 3/0,0 mode: PW/PW res: 182889173/0 rrc: 5 type: EXT <span class="error">&#91;0-&gt;18446744073709551615&#93;</span> (req 0-&gt;18446744073709551615) flags: 0x20 remote: 0xd281632991b12020 expref: 6 pid: 19246 timeout 7391670727</p> <p>With the client evicted we get dirty_page_discards like this.</p> <p>Dec 13 08:35:40 r305i3n1 kernel: <span class="error">&#91;1164772.491928&#93;</span> Lustre: 7178:0:(llite_lib.c:2283:ll_dirty_page_discard_warn()) nbp2: dirty page discard: 10.151.26.5@o2ib:/nbp2/fid: <span class="error">&#91;0x5677ca33040:0x2d5:0x0&#93;</span>//mlellis/RunStilt/runs/20120523-Cherskii-d01-WRF-TEST-20121213.15.46.32.UTC/run_d01/Exe/Copy8/cdump may get corrupted (rc -4)</p> <p>We have seen this happen at the beginning of a job. Now we are runing lflush before the start of every job. Could lflush cause this?</p> <p>We stilling trying to to reproduce it and gather additional logs. </p> LU-2499

Help debug waiting_locks_callback causing client eviction

Bug Critical Resolved Cannot Reproduce Zhenyu Xu Mahmoud Hanafi ptr Fri, 14 Dec 2012 18:53:04 +0000 Tue, 29 Oct 2013 18:40:39 +0000 Tue, 29 Oct 2013 18:40:39 +0000 Lustre 2.1.3 0 5 <p>Bobijam</p> <p>lflush is a tool-produced by LLNL. You may find some information on it by Googling. Could you please see what conditions would trigger this error and possible reasons? </p> <p>Thanks</p> <p>Peter</p> <p>Chris</p> <p>I think that you were involved in the creation of lflush. Is LLNL still using this tool on your 2.1.x production system? Have you ever seen any errors of this nature as a result if so?</p> <p>Peter</p> <p>See the scripts directory of this project:</p> <p> <a href="https://github.com/chaos/lustre-tools-llnl" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/chaos/lustre-tools-llnl</a></p> <p>It is fairly simple. These days it could be done even shorter if we just used an "lctl set_param".</p> <p>We do still use it in the slurm epilog script at the end of every job. We're not seeing that problem. At least not specifically associated with lflush, to the best of my knowledge.</p> <p>But "lock callback timer expired" is a very, very common error that we have seen, for many different reasons. Many nodes dropping their locks at the same time could certainly provide the load that uncovers a bug, network problem, or something else. Full logs will be needed to figure out what happened in this case.</p> <p>Do you have a detailed log around the time when this issue happens?</p> <p>this can be closed</p> <p>Thanks Mahmoud</p> Development Rank 1|hzvdvz: Rank (Obsolete) 5857 Severity