https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-277 Lustre <p>This issue was created by maloo for sarah &lt;sarah@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://maloo.whamcloud.com/test_sets/7a33c9b8-71c7-11e0-80b5-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/7a33c9b8-71c7-11e0-80b5-52540025f9af</a>.</p> <p>This one looks like lu-184 which has been fixed for a while. Actually I found the similar issue on replay-single and commented on <a href="https://jira.whamcloud.com/browse/LU-184" title="Test failure on test suite insanity, subtest test_0" class="issue-link" data-issue-key="LU-184"><del>LU-184</del></a> to make sure if it was the same problem. Here is the comment's link<br/> <a href="http://jira.whamcloud.com/browse/LU-184?focusedCommentId=12227&amp;page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12227" class="external-link" rel="nofollow">http://jira.whamcloud.com/browse/LU-184?focusedCommentId=12227&amp;page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12227</a> </p> LU-277

Test failure on test suite replay-single

Bug Minor Resolved Cannot Reproduce Niu Yawei Maloo Wed, 4 May 2011 14:30:08 +0000 Mon, 13 Jun 2011 14:21:57 +0000 Mon, 13 Jun 2011 14:21:57 +0000 0 1 <p>Niu</p> <p>As you worked previously on LU184 could you please comment?</p> <p>Thanks</p> <p>Peter</p> <p>The failure is caused by the open replay from client-6-ib and client-21-ib. Actually, these two clients should not be involved in this test (our intention is to test client-23-ib), however, there were lots of open replay reqeusts kept on the other two clients, these open replays participated in the recovery, and result in test failure at the end.</p> <p>I guess these open replays come from previous tests, maybe runracer. (runracer is before replay_single? Sarah, please correct me if I'm wrong) Will look into the runracer test to see if there is anything wrong in the script.</p> <p>How can the open replay fail? One possible reason occur to me: we often set the MDS as read only in replay_single tests, so the open_create will not commit into disk sometimes, and in the test_20b, client-23-ib was evicted by MDS, so some open_create replay from this client will be lost, and the open replays to the same file from other two clients will fail for EONENT. </p> <p>As shown on the maloo system, runracer is just ran before this replay-single test: This replay-single test started on 2011-04-28 11:25:09 UTC, and a runracer started on 2011-04-28 11:21:52 UTC and last 197 seconds.</p> <p>So I highly suspect that there is something wrong in the runracer script, which caused some racer test threads was not termninated properly. (as I mentioned in previous comment, since replay-single test often run replay barrier, these unexpected racer test thread could result in recovery failure)</p> <p>The following line of runracer confused me:<br/> running=$(do_nodes $clients "ps uax | grep $RDIR " | egrep -v "(acceptance|grep|pdsh|bash)" || true)<br/> I don't see why 'acceptance' should be matched, it might be the culprit.</p> <p>Hi, Sarah<br/> Since I don't have any reserved nodes, could you help me to do following test?</p> <ul class="alternate" type="square"> <li>Run runracer with auster. (on two or three clients, client-23-ib, client-21-ib and client-6-ib for instance)</li> <li>While test is running, get the output of "do_nodes $clients "ps uax | grep $RDIR "</li> <li>While test is running, get the output of "do_nodes $clients "ps uax | grep $RDIR " | egrep -v "(acceptance|grep|pdsh|bash)"<br/> I want to see if the criteria of finding racer test threads is still valid for auster. Thanks.</li> </ul> <p>Well, I finally get 3 nodes on Toro today, and after several runs of runracer, I found that pdsh often return errors like:</p> <p>pdsh@client-16-ib: client-17: read: protocol failure: Connection reset by peer<br/> or<br/> pdsh@client-16-ib: client-17: rcmd: xpoll (setting up stderr): Interrupted system call</p> <p>so when such error happens, the script will incorrectly think there isn't racer threads running by following check:<br/> running=$(do_nodes $clients "ps uax | grep $RDIR " | egrep -v "(acceptance|grep|pdsh|bash)" || true)</p> <p>Will try to come up with a patch to deal with the pdsh errors in the script.</p> <p>I tried to reproduce this bug with running "runracer + replay-single 20" over three clients many times, but it never hit it, so I can't make sure if the open-replays come from runracer.</p> <p>Hi Sarah, how often did you encounter this bug? If there isn't any reproducer, and it's only be seen very few times, I suggest we leave hold the investigating on it until it becomes a real issue. Thanks.</p> <p>Reopen if reoccurs</p> Development Rank 1|hzw17j: Rank (Obsolete) 10277 Severity