https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3447 Lustre <p>During an IOR-like benchmark doing directIO from multiple clients (16, 64) clients get disconnected and evicted. The MPI process dies in misery and some of it's processes aren't even killable.</p> <p>We've seen that there was a similar bug a while ago that was marked as solved, it was occuring on lnet routers (<a href="https://bugzilla.lustre.org/show_bug.cgi?id=13607" class="external-link" target="_blank" rel="nofollow noopener">https://bugzilla.lustre.org/show_bug.cgi?id=13607</a>). This one is on clients.</p> <p>What can lead to the "RDMA too fragmented" issue? Any hint or suggestion? Client log messages are in the attached file.</p> <p>Regards,<br/> Erich</p> Lustre servers running 2.1.5, Lustre clients with 1.8.9. LU-3447

Client RDMA too fragmented: 128/255 src 128/256 dst frags

Bug Major Resolved Fixed Bruno Faccini Erich Focht client Mon, 10 Jun 2013 17:08:43 +0000 Sat, 15 Mar 2014 01:21:36 +0000 Sat, 15 Mar 2014 01:21:36 +0000 Lustre 2.1.5 0 8 <p>Increasing the MTT size on the client nodes seems to solve the problem. For instructions: <a href="http://community.mellanox.com/docs/DOC-1120" class="external-link" target="_blank" rel="nofollow noopener">http://community.mellanox.com/docs/DOC-1120</a><br/> We've set log_num_mtt to 24.</p> <p>Having a more meaningful error message would be nice.</p> <p>This bug can be closed.</p> <p>Hello Eric,<br/> Thank's for the hint that solved the issue on your side.<br/> But to be complete on this it would be nice to give a try to the "map_on_demand" dynamic feature (o2iblnd proc/module parameter, but this has to be set on all nodes) that may also be a way to fix such problem.</p> <p>Bruno</p> <p>Can you please advise?</p> <p>Thanks</p> <p>Peter</p> <p>Hi Bruno,</p> <p>is that option available on 1.8.9 as well as on 2.X? Thanks for pointing me to it!</p> <p>It is difficult to do that in the customer's environment if we need to set this on both clients and servers, he has 3-4 Lustre filesystems (not all from us), a mix of versions, and 3.5k clients. But I'll try to find an opportunity to do it and discuss with the customer.</p> <p>Best regards,<br/> Erich</p> <p>Hello Eric,<br/> Working more on this very un-frequent problem, it seems highly possible that it is caused by upper-layer/application doing big and un-aligned I/Os. Since you indicated that your customer got it when running some MPI application doing Direct-IOs, can you also check on his side about the fact that these I/Os could be unaligned (page boundaries) and about their size ??</p> <p>Hello Eric,<br/> Any news on your side ??</p> <p>Hi Bruno,</p> <p>unfortunately we cannot use the module option there. It is a huge enironment with several Lustre setups and the customer is not willing to switch that option over everywhere. Which we'd need to do (as far as I understand) on clients as well as on servers. So we can't switch the clients selectively over. But we will test it as soon as we can on another (upcoming) installation.</p> <p>Regards,<br/> Erich</p> <p>Erich,<br/> Do you want us to keep this ticket open?<br/> Maybe you have had a chance to test the issue on a later installation?<br/> Thanks,<br/> ~ jfc.</p> <p>Customer was able to resolve problem. No more required here.</p> Development Rank 1|hzvsyf: Rank (Obsolete) 8618 Severity