https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1277 Lustre <p>Hi, we have an customer who have existing lustre filesystem (usage is about 60-70%). When they mount the lustre on the clients and start the IO the client, the initial IO performance is always slow.<br/> When it writes the files and remove them... repeating many times, the system is going to be normal and IO performance is back. <br/> This is odd behavior, but it always happens after the lustre starts.</p> LU-1277

Initial IO is slow after mount on the client

Bug Minor Closed Fixed Johann Lombardi Shuichi Ihara Mon, 2 Apr 2012 06:23:37 +0000 Mon, 20 Aug 2012 09:51:13 +0000 Mon, 20 Aug 2012 09:51:09 +0000 Lustre 1.8.7 0 5 <p>attached is syslog (we ran sysraq+t) on one of OSSs, we saw these call-trace on all of OSS, when this problem happens.</p> <p>Hi Ihara,</p> <p>Once performance are back to "normal" from a given client, if you then mount a new client, are performance bad again on this new client until you repeat the test several times?<br/> I wonder if this bug is instead related to OST restart, like the problem we have with bitmap loading (as mentioned in <a href="https://jira.whamcloud.com/browse/LU-15" title="strange slow IO messages and bad performance " class="issue-link" data-issue-key="LU-15"><del>LU-15</del></a>).</p> <p>If this is really related to the lustre client itself, then it might be due to grant. It will take several bulk writes for the client to own 32MB of grant space. To isolate the problem, could you please:</p> <ul> <li>collect rpc_stats (on client) &amp; brw_stats (on OSTs) for a good and bad run?</li> <li>tell us if this problem is only observed with writes or also with reads?</li> </ul> <p>Johann,</p> <p>we just tried to mount on a new client after back to the normal performance, but the performance was no problem on that client.</p> <p>btw, we saw similar behavior that was reported on <a href="https://jira.whamcloud.com/browse/LU-15" title="strange slow IO messages and bad performance " class="issue-link" data-issue-key="LU-15"><del>LU-15</del></a> - when it writes the files after all OSTs are mounted, there are many small read IOs although there are noshing read from user space.<br/> This filesystem was formatted on 1.8.4, but already upgraded 1.8.7 few month ago.</p> <p>If the problem only happens after OST restart, then i'm afraid that the only "solution" would be to write some tool/patch to pre-load the bitmaps in memory.</p> <p>Now, if the problem can also happen while the OST wasn't restarted recently, it could mean that some bitmaps got evicted from memory and it might make sense to try to disable the OSS read cache feature (there are actually 2 parameters to disable) if not done already.</p> <p>So far, the problem only happens after mount the OSTs..<br/> I thought <a href="https://jira.whamcloud.com/browse/LU-15" title="strange slow IO messages and bad performance " class="issue-link" data-issue-key="LU-15"><del>LU-15</del></a> was fixed in 1.8.6, but we still have a potential to get this problem even on 1.8.7?</p> <p>I think this is a longstanding issue related to bitmap loading and calculating of extent tables on ext4 mount.<br/> I think people used debugfs right after start to preload the bitmaps and it helped.</p> <p>Hi Ihara, are you ok to close this bug?</p> <p>yes, please. we will try dumpe2fs before start OSTs to load bitmap and extent table calculation. Thanks for advice!</p> Development Rank 1|hzvh4n: Rank (Obsolete) 6424 Severity