https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3 Lustre <p>As originally reported on Lustre Bugzilla:<br/> <a href="https://bugzilla.lustre.org/show_bug.cgi?id=15962" class="external-link" target="_blank" rel="nofollow noopener">https://bugzilla.lustre.org/show_bug.cgi?id=15962</a></p> <p>This issue will cause LBUG on client-side as following:</p> <p>=====================<br/> LustreError: 10521:0:(namei.c:826:ll_create_node()) ASSERTION(list_empty(&amp;inode-&gt;i_dentry)) failed^M<br/> LustreError: 10521:0:(namei.c:826:ll_create_node()) LBUG^M<br/> Pid: 10521, comm: dir_create.sh^M<br/> ^M<br/> Call Trace:^M<br/> <span class="error">&#91;&lt;00000000f8e00640&gt;&#93;</span> libcfs_debug_dumpstack+0x50/0x70 <span class="error">&#91;libcfs&#93;</span>^M<br/> <span class="error">&#91;&lt;00000000f8e00d9d&gt;&#93;</span> lbug_with_loc+0x6d/0xd0 <span class="error">&#91;libcfs&#93;</span>^M<br/> <span class="error">&#91;&lt;00000000f8e0c1f6&gt;&#93;</span> libcfs_assertion_failed+0x66/0x70 <span class="error">&#91;libcfs&#93;</span>^M<br/> <span class="error">&#91;&lt;00000000f927c762&gt;&#93;</span> ll_create_nd+0xb62/0xbd0 <span class="error">&#91;lustre&#93;</span>^M<br/> <span class="error">&#91;&lt;00000000c0482b33&gt;&#93;</span> vfs_create+0xc8/0x12f^M<br/> <span class="error">&#91;&lt;00000000c04854e4&gt;&#93;</span> open_namei+0x16a/0x5f9^M<br/> <span class="error">&#91;&lt;00000000c0474b02&gt;&#93;</span> do_filp_open+0x1c/0x31^M<br/> <span class="error">&#91;&lt;00000000c0474b55&gt;&#93;</span> do_sys_open+0x3e/0xae^M<br/> <span class="error">&#91;&lt;00000000c0474bf2&gt;&#93;</span> sys_open+0x16/0x18^M<br/> <span class="error">&#91;&lt;00000000c0404f17&gt;&#93;</span> syscall_call+0x7/0xb^M<br/> =====================</p> <p>It has been reported many times on lustre-1.8 in racer test. Iit can be reproduced locally with testing racer repeatedly.</p> LU-3

Missing concurrent access control between statahead and VFS create/unlink operation caused dcache confused

Bug Major Closed Fixed nasf nasf Thu, 16 Sep 2010 00:56:24 +0000 Sat, 8 Oct 2011 04:12:34 +0000 Sat, 8 Oct 2011 04:12:34 +0000 Lustre 2.1.0 Lustre 1.8.6 0 4 <p>This is a mixed patch containing:</p> <p>1) introduce rw_semaphore "lli_sa_rwsem" for the synchronization control between statahead and VFS create operation.<br/> 2) introduce spin_lock "lli_sa_lock" (special) to replace "lli_lock" (shared) for statahead related critical protection.<br/> 3) fix statahead related LUBG reported in lustre-discuss mail list recently: "(statahead.c:289:ll_sai_entry_fini()) LBUG".<br/> 4) drop unnecessary debug messages.<br/> 5) code cleanup.</p> <p>This patch will make current statahead on lustre-1.8 more smoothly.</p> <p>The patch has been attached to Lustre bugzilla bug 15962, to be inspected by Oracle's engineers.</p> <p>Any progress on this? We have seen this happen at several sites and in fact have statahead disabled by default in our builds.</p> <p>The patch attached on bug 15962 is workable, but some inspector does not like it because it is hack, not only for the patch, but also for the statahead implementation. So it maybe not landed to the main branch. Currently, we are thinking about big change the statahead implementation to avoid those hack.</p> <p>So if you want, I think you can apply such patch as temporary solution.</p> <p>new patch is in inspection:<br/> <a href="http://review.whamcloud.com/#change,2" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,2</a></p> <p>got this issue when running runracer on three clients:<br/> build: lustre-master/rhel5-x86_64/#16<br/> -------------------------------------------------------</p> <p>Apr 6 19:33:18 client-21 kernel: Lustre: DEBUG MARKER: == runracer test 1: racee<br/> r on clients: client-5-ib,client-21-ib,client-22-ib DURATION=120 ============= 11<br/> 9:30:15 (1302143415)<br/> Apr 6 19:33:18 client-21 xinetd<span class="error">&#91;3131&#93;</span>: EXIT: shell status=0 pid=11966 duration==<br/> 0(sec)<br/> Apr 6 19:33:18 client-21 xinetd<span class="error">&#91;3131&#93;</span>: START: shell pid=11980 from=192.168.4.5<br/> Apr 6 19:33:18 client-21 rshd<span class="error">&#91;11981&#93;</span>: root@192.168.4.5 as root: cmd='(PATH=$PATT<br/> H:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib644<br/> /lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "DURATION=120 /usr/lib64/lustree<br/> /tests/racer/racer.sh /mnt/lustre/racer ");echo XXRETCODE:$?'<br/> Apr 6 19:33:34 client-21 kernel: LustreError: 11-0: an error occurred while comm<br/> municating with 192.168.4.128@o2ib. The mds_getattr operation failed with -2<br/> Apr 6 19:33:34 client-21 kernel: LustreError: Skipped 5 previous similar messagg<br/> es<br/> Apr 6 19:33:53 client-21 kernel: LustreError: 21677:0:(file.c:2151:ll_inode_revv<br/> alidate_fini()) failure -2 inode 144115205339619325<br/> Apr 6 19:34:08 client-21 kernel: LustreError: 1339:0:(file.c:2151:ll_inode_revaa<br/> lidate_fini()) failure -116 inode 144115205322901335<br/> Apr 6 19:34:19 client-21 kernel: LustreError: 10966:0:(file.c:2151:ll_inode_revv<br/> alidate_fini()) failure -2 inode 144115205339624220<br/> Apr 6 19:35:07 client-21 kernel: LustreError: 24950:0:(namei.c:743:ll_create_noo<br/> de()) ASSERTION(list_empty(&amp;inode-&gt;i_dentry)) failed<br/> Apr 6 19:35:07 client-21 kernel: LustreError: 24950:0:(namei.c:743:ll_create_noo<br/> de()) LBUG<br/> Apr 6 19:35:07 client-21 kernel: Pid: 24950, comm: file_concat.sh<br/> Apr 6 19:35:07 client-21 kernel:<br/> Apr 6 19:35:07 client-21 kernel: Call Trace:<br/> Apr 6 19:35:07 client-21 kernel: <span class="error">&#91;&lt;ffffffff886625f1&gt;&#93;</span> libcfs_debug_dumpstack+00<br/> x51/0x60 <span class="error">&#91;libcfs&#93;</span><br/> Apr 6 19:35:07 client-21 kernel: <span class="error">&#91;&lt;ffffffff88662b2a&gt;&#93;</span> lbug_with_loc+0x7a/0xd0 <br/> <span class="error">&#91;libcfs&#93;</span><br/> Apr 6 19:35:07 client-21 kernel: <span class="error">&#91;&lt;ffffffff8866d960&gt;&#93;</span> cfs_tracefile_init+0x0/00<br/> x10a <span class="error">&#91;libcfs&#93;</span><br/> Apr 6 19:35:07 client-21 kernel: <span class="error">&#91;&lt;ffffffff88aa00dc&gt;&#93;</span> ll_create_nd+0x35c/0x9d00<br/> <span class="error">&#91;lustre&#93;</span><br/> Apr 6 19:35:07 client-21 kernel: <span class="error">&#91;&lt;ffffffff80022924&gt;&#93;</span> d_alloc+0x174/0x1a9<br/> Apr 6 19:35:07 client-21 kernel: <span class="error">&#91;&lt;ffffffff8003a579&gt;&#93;</span> vfs_create+0xe6/0x158<br/> Apr 6 19:35:07 client-21 kernel: <span class="error">&#91;&lt;ffffffff8001b0d9&gt;&#93;</span> open_namei+0x19d/0x6d5<br/> Apr 6 19:35:07 client-21 kernel: <span class="error">&#91;&lt;ffffffff80027533&gt;&#93;</span> do_filp_open+0x1c/0x38<br/> Apr 6 19:35:07 client-21 kernel: <span class="error">&#91;&lt;ffffffff80019e5d&gt;&#93;</span> do_sys_open+0x44/0xbe<br/> Apr 6 19:35:07 client-21 kernel: <span class="error">&#91;&lt;ffffffff8005d116&gt;&#93;</span> system_call+0x7e/0x83<br/> Apr 6 19:35:07 client-21 kernel:</p> <p>For 2.6.38 or newer kernel, RCU is used to make a significant part of the entire path walk. This is known as "rcu-walk" path walking. But even if that, the parent's "i_muext" is necessary for the caller of "do_lookup()" to prevent concurrent lookup/create operations with the same under the same parent.</p> <p>======<br/> mutex_lock(&amp;dir-&gt;i_mutex);<br/> /*</p> <ul> <li>First re-do the cached lookup just in case it was created</li> <li>while we waited for the directory semaphore, or the first</li> <li>lookup failed due to an unrelated rename.<br/> *</li> <li>This could use version numbering or similar to avoid unnecessary</li> <li>cache lookups, but then we'd have to do the first lookup in the</li> <li>non-racy way. However in the common case here, everything should</li> <li>be hot in cache, so would it be a big win?<br/> */<br/> dentry = d_lookup(parent, name);<br/> if (likely(!dentry)) { dentry = d_alloc_and_lookup(parent, name, nd); mutex_unlock(&amp;dir-&gt;i_mutex); if (IS_ERR(dentry)) goto fail; goto done; } <p> /*</p></li> <li>Uhhuh! Nasty case: the cache was re-populated while</li> <li>we waited on the semaphore. Need to revalidate.<br/> */<br/> mutex_unlock(&amp;dir-&gt;i_mutex);<br/> ======</li> </ul> <p>So we still have to adjust statahead to make it controllable under concurrent access mode with other VFS create/unlink operations.</p> <p>The patch for removing statahead dcache hack:</p> <p><a href="http://review.whamcloud.com/#change,1208" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,1208</a></p> <p>The race conditions caused by statahead will be fixed by the patch(es) for ORNL-7</p> Bugzilla ID 15962.0 Development Epic statahead Rank 1|hzw1e7: Rank (Obsolete) 10307 Severity