https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7425 Lustre <p>Error occurred during soak testing of build '20151112' (see: <a href="https://wiki.hpdd.intel.com/display/Releases/Soak+Testing+on+Lola#SoakTestingonLola-20151112" class="external-link" target="_blank" rel="nofollow noopener">https://wiki.hpdd.intel.com/display/Releases/Soak+Testing+on+Lola#SoakTestingonLola-20151112</a>). DNE is enabled. OST had been formatted using <em>zfs</em>, MDTs using <em>ldiskfs</em> as backend.</p> <p>Approximately 20% of the jobs executed during soak testing fail with the following typical error shown below. The error selected for the eviction (8 jobs affected) reads as:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lola-2.log:Nov 13 01:18:11 lola-2 kernel: LustreError: 0:0:(ldlm_lockd.c:342:waiting_locks_callback()) ### lock callback timer expired after 100s: evicting client at 192.168.1.133@o2ib100 ns: filter-soaked-OST0000_UUID lock: ffff8801cb9ded00/0x4823861a3a714ead lrc: 3/0,0 mode: PR/PR res: [0x2d2261b:0x0:0x0].0x0 rrc: 5 type: EXT [105534627840-&gt;18446744073709551615] (req 105534627840-&gt;105556344831) flags: 0x60000000030020 nid: 192.168.1.133@o2ib100 remote: 0x1b74614c7d18de8f expref: 20 pid: 9256 timeout: 4311080944 lvb_type: 0 </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lola-31.log:Nov 13 01:18:12 lola-31 kernel: LustreError: 11-0: soaked-OST0000-osc-ffff8808657d2800: op eration ost_read to node 192.168.1.102@o2ib10 failed: rc = -107 lola-31.log:Nov 13 01:18:12 lola-31 kernel: Lustre: soaked-OST0000-osc-ffff8808657d2800: Connection to soaked-OST0000 (at 192.168.1.102@o2ib10) was lost; in progress operations using this service will wai t for recovery to complete lola-31.log:Nov 13 01:18:12 lola-31 kernel: Lustre: Skipped 1 previous similar message lola-31.log:Nov 13 01:18:12 lola-31 kernel: LustreError: 167-0: soaked-OST0000-osc-ffff8808657d2800: This client was evicted by soaked-OST0000; in progress operations using this service will fail. lola-31.log:Nov 13 01:18:22 lola-31 kernel: Lustre: soaked-OST0000-osc-ffff8808657d2800: Connection restored to 192.168.1.102@o2ib10 (at 192.168.1.102@o2ib10) lola-33.log:Nov 13 01:18:12 lola-33 kernel: LustreError: 11-0: soaked-OST0000-osc-ffff88033a596800: operation ldlm_enqueue to node 192.168.1.102@o2ib10 failed: rc = -107 lola-33.log:Nov 13 01:18:12 lola-33 kernel: Lustre: soaked-OST0000-osc-ffff88033a596800: Connection to soaked-OST0000 (at 192.168.1.102@o2ib10) was lost; in progress operations using this service will wait for recovery to complete lola-33.log:Nov 13 01:18:12 lola-33 kernel: Lustre: Skipped 1 previous similar message lola-33.log:Nov 13 01:18:12 lola-33 kernel: LustreError: 167-0: soaked-OST0000-osc-ffff88033a596800: This client was evicted by soaked-OST0000; in progress operations using this service will fail. lola-33.log:Nov 13 01:18:12 lola-33 kernel: Lustre: 32171:0:(llite_lib.c:2628:ll_dirty_page_discard_warn()) soaked: dirty page discard: 192.168.1.108@o2ib10:192.168.1.109@o2ib10:/soaked/fid: [0x24006e1d4:0xd58d:0x0]/ may get corrupted (rc -108) lola-33.log:Nov 13 01:19:01 lola-33 kernel: Lustre: soaked-OST0000-osc-ffff88033a596800: Connection restored to 192.168.1.102@o2ib10 (at 192.168.1.102@o2ib10) </pre> </div></div> <p>No other messages, kernel debug logs were written on other nodes. No recovery went on, nor any error was injected during the lifetime of the jobs.</p> <p>The syslog, console and kernel debug logs of OSS node (lola-2), soak clients involved and slurm job log files had been attached to the ticket.</p> lola<br/> build: 2.7.62-40-gebda41d, ebda41d8de7956f19fd27f86208c668e43c6957c + patches LU-7425

client evicted with high rate

Bug Critical Closed Cannot Reproduce Bruno Faccini Frank Heckes soak Fri, 13 Nov 2015 13:04:19 +0000 Mon, 8 Feb 2016 13:13:42 +0000 Mon, 8 Feb 2016 13:13:42 +0000 Lustre 2.9.0 0 4 <p>I double check the IB fabric and found no indication for errors in general and especially not for the nodes in question.</p> <p>slurm job log files containing the time stamp of the event as most important information.</p> <p>Bruno</p> <p>Could you please look into this one?</p> <p>Thanks</p> <p>Peter</p> <p>I have spent some time to parse the provided logs/infos, but unfortunately the Lustre debug logs of OSS/Clients do not contain something related and interesting at the time of the problem, no log record/line with eviction msg nor referencing the concerned remote/local lock.</p> <p>Frank, since you indicate this is a frequent issue on soaktest system, I wonder if you could re-run the same workload with maximum debug mask enabled on Clients/Servers ?</p> <p>Yes, this would be possilbe, but have to be synchronized with Di which currently needs a debug mask which is less verbose.<br/> I'll update the tickets as soon as I found an agreement with Di.</p> <p>Hello Franck,<br/> Is this still occurring during soak testing??<br/> If not, we may want to close/resolve this ticket as CanNotReproduce, what do you think??</p> <p>Hi Bruno, I agree, we don't observed the scenario again. Let's close it with the state you suggested.</p> Development Rank 1|hzxszz: Rank (Obsolete) 9223372036854775807 Severity