Whamcloud Community JIRA

Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us

9.4.14

940014

05-12-2023

[LU-6570] sanityn test_40a: create is blocked https://jira.whamcloud.com/browse/LU-6570 Lustre <p>This issue was created by maloo for Jinshan Xiong <jinshan.xiong@intel.com></p> <p>This issue relates to the following test suite run: <a href="https://testing.hpdd.intel.com/test_sets/0859146e-f35b-11e4-9186-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/0859146e-f35b-11e4-9186-5254006e85c2</a>.</p> <p>The sub-test test_40a failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>create is blocked </pre> </div></div> <p>Please provide additional information about the failure here.</p> <p>Info required for matching: sanityn 40a</p> LU-6570

sanityn test_40a: create is blocked

Bug Blocker Resolved Fixed Di Wang Maloo dne2 Wed, 6 May 2015 06:09:29 +0000 Fri, 9 Sep 2016 20:13:30 +0000 Thu, 9 Jul 2015 21:53:17 +0000 Lustre 2.8.0 Lustre 2.5.5 Lustre 2.8.0 0 9 <p>It looks like this test may be racy under load:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># check that pid exists hence second operation wasn't blocked by first one # if it is so then there is no conflict, return 0 # else second operation is conflicting with first one, return 1 check_pdo_conflict() { local pid=$1 local conflict=0 sleep 1 # to ensure OP1 is finished on client if OP2 is blocked by OP1 if [[ `ps --pid $pid | wc -l` == 1 ]]; then conflict=1 echo "Conflict" else echo "No conflict" fi return $conflict } </pre> </div></div> <p>If a process is around for 1s then it would cause a failure? In any case, this has only ever happened once, so it isn't clear if it was just a very small race condition or something else. I looked through the other sanityn tests for the patch <a href="http://review.whamcloud.com/14100" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/14100</a> where this was hit, but no other cases of this failure.</p> <p>On master branch, this is a regression failure introduced by patch <a href="http://review.whamcloud.com/14495" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/14495</a>.</p> <p>No, there are no protocol changes for the patch. The failures on 2.5.5 is because of mistakes they made when porting the patch <a href="http://review.whamcloud.com/#/c/14495/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/14495/</a> from master to b2_5. I am not sure about the failure on master, and it is not even about DNE because all of operations in sanityn 40a are being done under non-striped directory, not sure why it is related with 14495.</p> <p>Just check the debug log, the failure seems caused by the test-script itself.</p> <p>On the client1, it mkdir $DIR1/$tfile</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>0000080:00200000:1.0:1432173251.411997:0:5365:0:(namei.c:1153:ll_mkdir()) VFS Op:name=f40a.sanityn, dir=[0x200000007:0x1:0x0](ffff88006b8c0a98)</pre> </div></div> <p>on the server side, because of OBD_FAIL_MDS_PDO_LOCK, it will be scheduled timeout for 15 seconds. Note: during timeout, it actually holds the update lock of DIR1 (root)</p> <p>Then on client2, which will do touch touch $DIR2/$tfile-2, first it will try to revalidate DIR2 <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/check.png" height="16" width="16" align="absmiddle" alt="" border="0"/>, usually client only requires LOOKUP lock for this revalidation.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000002:00010000:0.0:1432173252.415097:0:5367:0:(mdc_locks.c:1148:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 00 </pre> </div></div> <p>But it seems on the MDT side, UPDATE lock is added back because of</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> /* If the file has not been changed for some time, we * return not only a LOOKUP lock, but also an UPDATE * lock and this might save us RPC on later STAT. For * directories, it also let negative dentry cache start * working for this dir. */ if (ma->ma_valid & MA_INODE && ma->ma_attr.la_valid & LA_CTIME && info->mti_mdt->mdt_namespace->ns_ctime_age_limit + ma->ma_attr.la_ctime < cfs_time_current_sec()) child_bits |= MDS_INODELOCK_UPDATE; </pre> </div></div> <p>But unfortunately, this is FID revalidate(no name in the request) so no PDO lock. then it will be blocked by the lock above (holded by client1 request handler). </p> <p>Hmm, I guess we can just add "touch $DIR2" before client1 "mkdir $IR1/$tfile", then this ctime_age_limit will not be triggered, i.e. UPDATE lock will not be required during revalidate /, the problem should be fixed.</p> <p>wangdi (di.wang@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/15366" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/15366</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6570" title="sanityn test_40a: create is blocked" class="issue-link" data-issue-key="LU-6570"><del>LU-6570</del></a> tests: fix 40a in sanityn.sh<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 459bb83d92ddc50aa5caa71b629ba29e53266120</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/15366/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/15366/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6570" title="sanityn test_40a: create is blocked" class="issue-link" data-issue-key="LU-6570"><del>LU-6570</del></a> tests: fix 40a in sanityn.sh<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 2083f8a34b556c9e64e09a4529df7d9bdbdd7532</p> <p>Landed for 2.8</p> Duplicate LU-6672 Related Development Rank 1|hzxclb: Rank (Obsolete) 9223372036854775807 Severity