https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1809 Lustre <p>NOAA hit a problem that looks a lot like <a href="https://jira.whamcloud.com/browse/LU-441" title="ll_fill_super()) Unable to process log: -108" class="issue-link" data-issue-key="LU-441"><del>LU-441</del></a>. The clients were unable to mount the filesystem for a while after rebooting. </p> <p>Here's the client's syslog:<br/> Aug 30 15:00:09 s1 kernel: Lustre:<br/> 2524:0:(client.c:1487:ptlrpc_expire_one_request()) @@@ Request<br/> x1410486709891636 sent from MGC10.179.16.120@o2ib to NID<br/> 10.179.16.121@o2ib 0s ago has failed due to network error (35s prior to<br/> deadline).<br/> Aug 30 15:00:09 s1 kernel: req@ffff8805fc06e400 x1410486709891636/t0<br/> o250-&gt;MGS@MGC10.179.16.120@o2ib_1:26/25 lens 368/584 e 0 to 1 dl<br/> 1346338844 ref 1 fl Rpc:N/0/0 rc 0/0<br/> Aug 30 15:00:09 s1 kernel: LustreError:<br/> 112398:0:(client.c:858:ptlrpc_import_delay_req()) @@@ IMP_INVALID<br/> req@ffff88041c05e800 x1410486709891637/t0<br/> o501-&gt;MGS@MGC10.179.16.120@o2ib_1:26/25 lens 264/432 e 0 to 1 dl 0 ref 1<br/> fl Rpc:/0/0 rc 0/0<br/> Aug 30 15:00:09 s1 kernel: LustreError: 15c-8: MGC10.179.16.120@o2ib:<br/> The configuration from log 'lfs2-client' failed (-108). This may be the<br/> result of communication errors between this node and the MGS, a bad<br/> configuration, or other errors. See the syslog for more information.<br/> Aug 30 15:00:09 s1 kernel: LustreError:<br/> 112398:0:(llite_lib.c:1095:ll_fill_super()) Unable to process log: -108<br/> Aug 30 15:00:09 s1 kernel: Lustre: client lfs2-client(ffff88041c2ea400)<br/> umount complete<br/> Aug 30 15:00:09 s1 kernel: LustreError:<br/> 112398:0:(obd_mount.c:2065:lustre_fill_super()) Unable to mount (-108)</p> <p>MDS logs to come. </p> LU-1809

Clients unable to mount (-108)

Bug Minor Resolved Duplicate Keith Mannthey Kit Westneat Fri, 31 Aug 2012 09:33:08 +0000 Fri, 19 Oct 2012 08:50:47 +0000 Fri, 19 Oct 2012 08:50:47 +0000 Lustre 1.8.8 0 7 <p>Likely this is a dup of <a href="https://jira.whamcloud.com/browse/LU-630" title="mount failure after MGS connection lost and file system is unmounted" class="issue-link" data-issue-key="LU-630"><del>LU-630</del></a>. The message below said an outgoing message sent 0 second ago failed with error:<br/> 2524:0:(client.c:1487:ptlrpc_expire_one_request()) @@@ Request x1410486709891636 sent from MGC10.179.16.120@o2ib to NID 10.179.16.121@o2ib 0s ago has failed due to network error (35s prior to deadline).</p> <p>This looked like a local error, i.e. the message did not go out on wire. Please:<br/> 1. On all clients and servers: options ko2iblnd peer_timeout=0<br/> 2. On some clients where mount failed: echo +neterror &gt; /proc/sys/lnet/printk<br/> This must be done after each client reboot.</p> <p>If this is a dup of <a href="https://jira.whamcloud.com/browse/LU-630" title="mount failure after MGS connection lost and file system is unmounted" class="issue-link" data-issue-key="LU-630"><del>LU-630</del></a>, then step 1 should fix it. If it still persists, step 2 would allow more debug data to go to syslog; /proc/sys/lnet/peers would also provide useful data in this case.</p> <p>It looks like the patch is fairly simple, can we get it landed on b1_8?</p> <p>In the meantime I will communicate the workaround to the customer. I think it is pretty rare though. </p> <p>Thanks,<br/> Kit</p> <p>Hi Isaac,</p> <p>What are the implications of peer_timeout=0? That is to say, what exactly does it do? </p> <p>Also, does it have to be on all the servers and clients? or can it be just the servers or just the clients?</p> <p>Thanks,<br/> Kit</p> <p>peer_timeout=0 disables a feature that should only be turned on for routers - it was a bug to be able to enable it anywhere but the routers. In other words, peer_timeout=0 fixes it without any code changes.</p> <p>The feature does not work on clients and servers and will cause messages to be dropped, so "peer_timeout=0" must be set on all clients and servers.</p> <p>Is it ok to do "peer_timeout=0" on the clients before the servers? Or does it need to be set at the same time everywhere?</p> <p>There's no requirement on order. You can do it in any order that's most convenient.</p> <p>could we get this landed to b1_8? It appears to have fixed the issue.</p> <p>I likely missed some notifications when JIRA was upgraded a while back. I agree that it's a simple patch that fixes a class of problems hard to diagnose when they manifest themselves at upper layers. I'd defer to Peter whether to land it to b1_8.</p> <p>Thanks Isaac. Keith can you please backport the patch from master to b1_8?</p> <p>Quite likely the patch would apply to b1_8 without any changes, just ignore white space changes with patch --ignore-whitespace. </p> <p>I was able to cherry-pick the patch from <a href="https://jira.whamcloud.com/browse/LU-630" title="mount failure after MGS connection lost and file system is unmounted" class="issue-link" data-issue-key="LU-630"><del>LU-630</del></a> into b1_8. </p> <p><a href="http://review.whamcloud.com/4287" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4287</a> is the b1_8 patch. </p> <p>duplicate of <a href="https://jira.whamcloud.com/browse/LU-630" title="mount failure after MGS connection lost and file system is unmounted" class="issue-link" data-issue-key="LU-630"><del>LU-630</del></a></p> Development Rank 1|hzvgmf: Rank (Obsolete) 6342 Severity