https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3228 Lustre <p>== sanity test 183: No crash or request leak in case of strange dispositions ========================= 20:23:20 (1366860200)<br/> CMD: client-24vm3 /usr/sbin/lctl set_param fail_loc=0x148<br/> fail_loc=0x148<br/> ls: cannot open directory /mnt/lustre/d0.sanity/d183: No such file or directory<br/> cat: /mnt/lustre/d0.sanity/d183/f.sanity.183: No such file or directory<br/> CMD: client-24vm3 /usr/sbin/lctl set_param fail_loc=0<br/> fail_loc=0<br/> touch: cannot touch ‘/mnt/lustre/d0.sanity/d183/f.sanity.183’: No such file or directory<br/> rm: cannot remove ‘/mnt/lustre/d0.sanity/d183’: Directory not empty<br/> sanity test_183: @@@@@@ FAIL: test_183 failed with 1 <br/> Trace dump:<br/> = /usr/lib64/lustre/tests/test-framework.sh:4024:error_noexit()<br/> = /usr/lib64/lustre/tests/test-framework.sh:4047:error()<br/> = /usr/lib64/lustre/tests/test-framework.sh:4301:run_one()<br/> = /usr/lib64/lustre/tests/test-framework.sh:4334:run_one_logged()<br/> = /usr/lib64/lustre/tests/test-framework.sh:4189:run_test()<br/> = /usr/lib64/lustre/tests/sanity.sh:9626:main()<br/> Dumping lctl log to /logdir/test_logs/2013-04-24/lustre-reviews-el6-x86_64-vs-lustre-reviews-fc18-x86_64-<del>review</del>-2_4_1_<em>15074</em>_-70194495514420-185046/sanity.test_183.*.1366860201.log<br/> CMD: client-24vm1,client-24vm2.lab.whamcloud.com,client-24vm3,client-24vm4 /usr/sbin/lctl dk &gt; /logdir/test_logs/2013-04-24/lustre-reviews-el6-x86_64-vs-lustre-reviews-fc18-x86_64-<del>review</del>-2_4_1_<em>15074</em>_-70194495514420-185046/sanity.test_183.debug_log.\$(hostname -s).1366860201.log;<br/> dmesg &gt; /logdir/test_logs/2013-04-24/lustre-reviews-el6-x86_64-vs-lustre-reviews-fc18-x86_64-<del>review</del>-2_4_1_<em>15074</em>_-70194495514420-185046/sanity.test_183.dmesg.\$(hostname -s).1366860201.log</p> <p><a href="https://maloo.whamcloud.com/test_sets/83b66856-ad8e-11e2-bbea-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/83b66856-ad8e-11e2-bbea-52540035b04c</a></p> LU-3228

fc18: sanity test_183: @@@@@@ FAIL: test_183 failed with 1

Bug Blocker Resolved Fixed Jian Yu Minh Diep yuc2 Thu, 25 Apr 2013 16:28:02 +0000 Tue, 1 Jul 2014 01:05:56 +0000 Sat, 25 Jan 2014 05:29:07 +0000 Lustre 2.4.0 Lustre 2.4.1 Lustre 2.4.0 0 5 <p>Yu, Jian</p> <p>Could you please look into this one?</p> <p>Thanks</p> <p>Peter</p> <p>Lustre Branch: master (tag 2.4.50)<br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-master/1502/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-master/1502/</a></p> <p>The sanity test 183 was added for <a href="https://jira.whamcloud.com/browse/LU-2275" title="Open request leak" class="issue-link" data-issue-key="LU-2275"><del>LU-2275</del></a> in <a href="http://review.whamcloud.com/4458" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4458</a>.</p> <p>It passed on RHEL6.4 client:<br/> <a href="https://maloo.whamcloud.com/test_sets/456be8de-c2db-11e2-b2eb-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/456be8de-c2db-11e2-b2eb-52540035b04c</a></p> <p>but failed on FC18 client:<br/> <a href="https://maloo.whamcloud.com/test_sets/a6a8e2d4-c2d4-11e2-b2eb-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/a6a8e2d4-c2d4-11e2-b2eb-52540035b04c</a></p> <p>After fail_loc was set to 0 on MDS, the touch operation failed on client node:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>fail_loc=0 touch: cannot touch '/mnt/lustre/d0.sanity/d183/f.sanity.183': No such file or directory </pre> </div></div> <p>Debug log on client node showed that:</p> <div class="panel" style="border-width: 1px;"><div class="panelContent"> <p>00000001:00000001:0.0:1369222356.453337:0:8789:0:(debug.c:444:libcfs_debug_mark_buffer()) ***************************************************<br/> 00000001:02000400:0.0:1369222356.453339:0:8789:0:(debug.c:445:libcfs_debug_mark_buffer()) DEBUG MARKER: fail_loc=0<br/> 00000001:00000001:0.0:1369222356.453345:0:8789:0:(debug.c:446:libcfs_debug_mark_buffer()) ***************************************************<br/> ......<br/> 00000080:00000001:0.0:1369222356.699798:0:8803:0:(dcache.c:96:ll_dcompare()) Process entered<br/> 00000080:00002000:0.0:1369222356.699799:0:8803:0:(dcache.c:119:ll_dcompare()) found name f.sanity.183(ffff88007a14f540) flags 0xce refc 0<br/> 00000080:00000001:0.0:1369222356.699800:0:8803:0:(dcache.c:128:ll_dcompare()) Process leaving (rc=0 : 0 : 0)<br/> 00000080:00000001:0.0:1369222356.699801:0:8803:0:(dcache.c:669:ll_revalidate_nd()) Process entered<br/> 00000080:00200000:0.0:1369222356.699801:0:8803:0:(dcache.c:671:ll_revalidate_nd()) VFS Op:name=f.sanity.183,flags=769<br/> <font color="red">00000080:00000001:0.0:1369222356.699802:0:8803:0:(dcache.c:684:ll_revalidate_nd()) Process leaving (rc=1 : 1 : 1)</font><br/> 00000080:00000001:0.0:1369222356.699803:0:8803:0:(file.c:2978:ll_inode_permission()) Process entered<br/> 00000080:00200000:0.0:1369222356.699803:0:8803:0:(file.c:3001:ll_inode_permission()) VFS Op:inode=144115205255725060/33554436(ffff88007b35c9f8), inode mode 41ed mask 3<br/> 00000080:00000001:0.0:1369222356.699804:0:8803:0:(file.c:3009:ll_inode_permission()) Process leaving (rc=0 : 0 : 0)<br/> 00000080:00000001:0.0:1369222356.699806:0:8803:0:(namei.c:622:ll_atomic_open()) Process entered<br/> 00000080:00200000:0.0:1369222356.699807:0:8803:0:(namei.c:627:ll_atomic_open()) VFS Op:name=f.sanity.183,dir=144115205255725060/33554436(ffff88007b35c9f8),file ffff88007adf5400,open_flags 8941,mode 81b6 opened 0<br/> 00000080:00000010:0.0:1369222356.699809:0:8803:0:(namei.c:629:ll_atomic_open()) kmalloced 'it': 72 at ffff88007a8bb420.<br/> 00000080:00000001:0.0:1369222356.699811:0:8803:0:(namei.c:496:ll_lookup_it()) Process entered<br/> 00000080:00200000:0.0:1369222356.699811:0:8803:0:(namei.c:503:ll_lookup_it()) VFS Op:name=f.sanity.183,dir=144115205255725060/33554436(ffff88007b35c9f8),intent=open|creat<br/> ......<br/> <font color="red">00000080:00000001:0.0:1369222356.702127:0:8803:0:(namei.c:568:ll_lookup_it()) Process leaving via out (rc=0 : 0 : 0x0)</font><br/> 00000100:00000001:0.0:1369222356.702128:0:8803:0:(client.c:2271:__ptlrpc_req_finished()) Process entered<br/> 00000100:00000040:0.0:1369222356.702130:0:8803:0:(client.c:2283:__ptlrpc_req_finished()) @@@ refcount now 2 req@ffff88007b35b400 x1435733615182304/t4294967313(4294967313) o101-&gt;lustre-MDT0000-mdc-ffff88007c342000@10.10.4.208@tcp:12/10 lens 568/600 e 0 to 0 dl 1369222363 ref 3 fl Complete:RP/4/0 rc 0/0<br/> 00000100:00000001:0.0:1369222356.702133:0:8803:0:(client.c:2290:__ptlrpc_req_finished()) Process leaving (rc=0 : 0 : 0)<br/> 00000080:00000001:0.0:1369222356.702135:0:8803:0:(dcache.c:289:ll_intent_release()) Process entered<br/> ......<br/> 00000080:00000001:0.0:1369222356.702158:0:8803:0:(dcache.c:304:ll_intent_release()) Process leaving<br/> 00000080:00000010:0.0:1369222356.702158:0:8803:0:(namei.c:688:ll_atomic_open()) kfreed 'it': 72 at ffff88007a8bb420.<br/> <font color="red">00000080:00000001:0.0:1369222356.702168:0:8803:0:(namei.c:690:ll_atomic_open()) Process leaving (rc=1 : 1 : 1)</font></p> </div></div> <p>The ll_atomic_open() failed with 1, which was returned from finish_no_open(). I'm still digging.</p> <p>Lustre Branch: b2_4<br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_4/27/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_4/27/</a><br/> Distro/Arch: RHEL6.4/x86_64 + FC18/x86_64 (Server + Client)</p> <p>The failure occurred regularly on Lustre b2_4 branch on FC18 client:<br/> <a href="https://maloo.whamcloud.com/test_sets/996b2278-fd79-11e2-9fdb-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/996b2278-fd79-11e2-9fdb-52540035b04c</a></p> <p>Lustre build: <a href="http://build.whamcloud.com/job/lustre-b2_4/44/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_4/44/</a> (2.4.1 RC1)<br/> Distro/Arch: RHEL6.4/x86_64 + FC18/x86_64 (Server + Client)</p> <p>sanity test 183 hit the same failure:<br/> <a href="https://maloo.whamcloud.com/test_sets/0cbde1d0-14ee-11e3-ac48-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/0cbde1d0-14ee-11e3-ac48-52540035b04c</a></p> <p>After failing like this, next unmount, that happens to be in test 223 crashes with reference count assertion.<br/> This happens to be <a href="https://jira.whamcloud.com/browse/LU-1480" title="failure on replay-single test_74: ASSERTION( cfs_atomic_read(&amp;d-&gt;ld_ref) == 0 ) failed: Refcount is 1" class="issue-link" data-issue-key="LU-1480"><del>LU-1480</del></a>.</p> <p>This bug is also observed in mainline kernel.</p> <p>Apparently this bug was introduced by commit 784cd144103871bd421c139c09bfbf4d5d29ca08 coming from patch <a href="http://review.whamcloud.com/4387" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4387</a> to support atomic_open.</p> <p>The problematic hunk is this:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>@@ -438,13 +443,21 @@ int ll_lookup_it_finish(struct ptlrpc_request *request, Also see bug 7198. */ } - *de = ll_splice_alias(inode, *de); + /* Only hash *de if it is unhashed (new dentry). + * Atoimc_open may passin hashed dentries for open. + */ + if (d_unhashed(*de)) + *de = ll_splice_alias(inode, *de); </pre> </div></div> <p>When this hits (as in, dentry IS hashed), two things happen:<br/> 1. we leak inode (and don't assign it to dentry, obviously)<br/> 2. we return NULL from ll_lookup_it, so it uses passed in dentry, but it is negative, and leads to that no_open branch, where we pass in de, that's NULL (probably should pass in dentry at all times?).</p> <p>Apparently this was fixed by <a href="http://review.whamcloud.com/8110" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8110</a></p> <p>I verified that the patch in <a href="http://review.whamcloud.com/8110" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8110</a> fixes the issue.</p> Duplicate LU-3373 Related LU-1480 Development Rank 1|hzvp27: Rank (Obsolete) 7889 Severity