https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-990 Lustre <p>The customer saw the following error messages on MDS during 128 clients were creating the many small files. (mostly 1 milion files), then they also saw Input/output error to the some files, when they opened these files.</p> <p>(lov_request.c:690:lov_update_create_set()) error creating fid xxx rc=-107</p> <p>As far as we see the log files on OSSs, at the same time, there are many slow IO messages at the same time.</p> <p>-107 = -ENOTCONN, is this happened connection loss between server and client? (MDS to OSS as well?)</p> <p>Please advise.</p> lustre-1.8.3 LU-990

(lov_request.c:690:lov_update_create_set()) error creating fid xxx rc=-107

Bug Minor Resolved Cannot Reproduce Zhenyu Xu Shuichi Ihara Fri, 13 Jan 2012 09:37:05 +0000 Thu, 6 Feb 2014 19:06:06 +0000 Thu, 6 Feb 2014 19:06:06 +0000 0 3 <p>Bobijam</p> <p>Could you please look into this one?</p> <p>Thanks</p> <p>Peter</p> <p>I saw several different kind of error messages in the logs:</p> <p>in OS1, there are many network error messages.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Dec 21 00:45:02 os1 kernel: LustreError: 11686:0:(o2iblnd_cb.c:1232:kiblnd_connect_peer()) Can't resolve addr for 172.20.1.23@o2ib: -101 Dec 21 00:45:02 os1 kernel: LustreError: 11683:0:(o2iblnd_cb.c:1232:kiblnd_connect_peer()) Can't resolve addr for 172.20.1.28@o2ib: -101 Dec 21 00:46:44 os1 kernel: LustreError: 11639:0:(o2iblnd_cb.c:1232:kiblnd_connect_peer()) Can't resolve addr for 172.20.1.26@o2ib: -101 Dec 21 00:46:45 os1 kernel: LustreError: 11608:0:(o2iblnd_cb.c:1232:kiblnd_connect_peer()) Can't resolve addr for 172.20.1.25@o2ib: -101 Dec 21 00:48:52 os1 kernel: LustreError: 11876:0:(o2iblnd_cb.c:1232:kiblnd_connect_peer()) Can't resolve addr for 172.20.1.22@o2ib: -101 Dec 21 00:48:56 os1 kernel: LustreError: 11855:0:(o2iblnd_cb.c:1232:kiblnd_connect_peer()) Can't resolve addr for 172.20.1.24@o2ib: -101 Dec 21 02:44:43 os1 kernel: LustreError: 11909:0:(o2iblnd_cb.c:1232:kiblnd_connect_peer()) Can't resolve addr for 172.20.1.1@o2ib: -101 ... </pre> </div></div> <p>#define ENETUNREACH 101 /* Network is unreachable */</p> <p>on OS2</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Dec 22 00:54:22 os2 kernel: LustreError: 11568:0:(lib-move.c:2441:LNetPut()) Error sending PUT to 12345-172.20.1.61@o2ib: -113 Dec 22 00:54:22 os2 kernel: LustreError: 8091:0:(lib-move.c:2441:LNetPut()) Error sending PUT to 12345-172.20.1.59@o2ib: -113 Dec 22 16:19:55 os2 kernel: LustreError: 11667:0:(lib-move.c:2441:LNetPut()) Error sending PUT to 12345-172.20.1.56@o2ib: -113 ... </pre> </div></div> <p>#define EHOSTUNREACH 113 /* No route to host */</p> <p>Are there network problems in the site?</p> <p>thanks. we asked about the network issue, but they said there are no error messages on the infiniband side.<br/> But let me ask again. they did run an large job which uses a lot of CPU resources. client might be delay response or took a long time for anything.<br/> even there were network issues, we can't see evicted messages. So, the client were not killed completely. </p> <p>any advise to prevent the this issue when if they run the large job again?</p> <p>w/o debug log I don't know what exact problem there is while I do see many heavy IO load warnning messages in the syslogs.</p> <p>Please checkout <a href="https://jira.whamcloud.com/browse/LU-952" title="Hung thread with HIGH OSS load" class="issue-link" data-issue-key="LU-952"><del>LU-952</del></a>, there are discussion and a patch for high load issue. Disabling read only cache and write through cache will help this issue, and if not, please collect lustre debug log and upload.</p> <p>We haven't seen same issue again very much. Please close this issue.</p> <p>ok - thanks Ihara!</p> Development Rank 1|hzvja7: Rank (Obsolete) 6838 Severity