https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3874 Lustre <p>parallel-scale-nfsv3 test iorfpp failed as follows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>** error ** ERROR in aiori-POSIX.c (line 235): seek failed. ERROR: Permission denied ** exiting ** ** error ** ERROR in aiori-POSIX.c (line 235): seek failed. ERROR: Permission denied ** exiting ** </pre> </div></div> <p>Maloo report: <a href="https://maloo.whamcloud.com/test_sets/6fc200da-14b3-11e3-8ae7-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/6fc200da-14b3-11e3-8ae7-52540035b04c</a></p> <p>This is a regression on Lustre b2_4 branch.</p> <br/> Lustre build: <a href="http://build.whamcloud.com/job/lustre-b2_4/44/">http://build.whamcloud.com/job/lustre-b2_4/44/</a> (2.4.1 RC1)<br/> Distro/Arch: SLES11SP2/x86_64 (client), RHEL6.4/x86_64 (server)<br/> LU-3874

parallel-scale-nfsv3 test_iorfpp: ERROR in aiori-POSIX.c (line 235): seek failed

Bug Blocker Closed Duplicate Oleg Drokin Jian Yu Wed, 4 Sep 2013 03:19:47 +0000 Fri, 4 Apr 2014 16:45:48 +0000 Mon, 23 Sep 2013 13:10:55 +0000 Lustre 2.5.0 0 7 <p>By searching on Maloo, I found the similar failure started occurring since Lustre b2_4 build #42:</p> <p>Lustre build: <a href="https://build.whamcloud.com/job/lustre-b2_4/42" class="external-link" target="_blank" rel="nofollow noopener">https://build.whamcloud.com/job/lustre-b2_4/42</a><br/> Distro/Arch: RHEL6.4/x86_64</p> <p>parallel-scale-nfsv3 test iorssf:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>** error ** ERROR in aiori-POSIX.c (line 256): transfer failed. ERROR: Permission denied ** exiting ** ** error ** ERROR in aiori-POSIX.c (line 256): transfer failed. ERROR: Permission denied ** exiting ** </pre> </div></div> <p>Maloo reports:<br/> <a href="https://maloo.whamcloud.com/test_sets/15807156-1355-11e3-a1df-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/15807156-1355-11e3-a1df-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/06ce48c2-13f5-11e3-9e61-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/06ce48c2-13f5-11e3-9e61-52540035b04c</a></p> <p>Fan Yong</p> <p>Is this related to the most recent <a href="https://jira.whamcloud.com/browse/LU-2904" title="parallel-scale-nfsv3: FAIL: setup nfs failed!" class="issue-link" data-issue-key="LU-2904"><del>LU-2904</del></a> fix?</p> <p>Peter</p> <p>I found the following failure on MDS side for each failure cases:</p> <p>==============================<br/> 08:08:02:Lustre: DEBUG MARKER: == parallel-scale-nfsv3 test iorfpp: iorfpp ========================================================== 08:07:55 (1378220875)<br/> 08:08:02:Lustre: DEBUG MARKER: lfs setstripe /mnt/lustre/d0.ior.fpp -c -1<br/> 08:08:02:LustreError: 12896:0:(lov_lock.c:674:lov_lock_enqueue()) ASSERTION( sublock-&gt;cll_conflict == ((void *)0) ) failed: <br/> 08:08:03:LustreError: 12896:0:(lov_lock.c:674:lov_lock_enqueue()) LBUG<br/> 08:08:03:Pid: 12896, comm: nfsd<br/> 08:08:04:<br/> 08:08:05:Call Trace:<br/> 08:08:05: <span class="error">&#91;&lt;ffffffffa06e7895&gt;&#93;</span> libcfs_debug_dumpstack+0x55/0x80 <span class="error">&#91;libcfs&#93;</span><br/> 08:08:06: <span class="error">&#91;&lt;ffffffffa06e7e97&gt;&#93;</span> lbug_with_loc+0x47/0xb0 <span class="error">&#91;libcfs&#93;</span><br/> 08:08:06: <span class="error">&#91;&lt;ffffffffa0c305ee&gt;&#93;</span> lov_lock_enqueue+0x76e/0x850 <span class="error">&#91;lov&#93;</span><br/> 08:08:06: <span class="error">&#91;&lt;ffffffffa082351c&gt;&#93;</span> cl_enqueue_try+0xfc/0x300 <span class="error">&#91;obdclass&#93;</span><br/> 08:08:06: <span class="error">&#91;&lt;ffffffffa082490f&gt;&#93;</span> cl_enqueue_locked+0x6f/0x1f0 <span class="error">&#91;obdclass&#93;</span><br/> 08:08:06: <span class="error">&#91;&lt;ffffffffa082557e&gt;&#93;</span> cl_lock_request+0x7e/0x270 <span class="error">&#91;obdclass&#93;</span><br/> 08:08:06: <span class="error">&#91;&lt;ffffffffa0cfd750&gt;&#93;</span> cl_glimpse_lock+0x180/0x490 <span class="error">&#91;lustre&#93;</span><br/> 08:08:07: <span class="error">&#91;&lt;ffffffffa0cfdfc5&gt;&#93;</span> cl_glimpse_size0+0x1a5/0x1d0 <span class="error">&#91;lustre&#93;</span><br/> 08:08:08: <span class="error">&#91;&lt;ffffffffa0cb1478&gt;&#93;</span> ll_inode_revalidate_it+0x198/0x1c0 <span class="error">&#91;lustre&#93;</span><br/> 08:08:09: <span class="error">&#91;&lt;ffffffffa0cb14e9&gt;&#93;</span> ll_getattr_it+0x49/0x170 <span class="error">&#91;lustre&#93;</span><br/> 08:08:09: <span class="error">&#91;&lt;ffffffffa0cb1647&gt;&#93;</span> ll_getattr+0x37/0x40 <span class="error">&#91;lustre&#93;</span><br/> 08:08:09: <span class="error">&#91;&lt;ffffffff8121d1a3&gt;&#93;</span> ? security_inode_getattr+0x23/0x30<br/> 08:08:10: <span class="error">&#91;&lt;ffffffff81186d81&gt;&#93;</span> vfs_getattr+0x51/0x80<br/> 08:08:11: <span class="error">&#91;&lt;ffffffff81182bc1&gt;&#93;</span> ? __fput+0x1a1/0x210<br/> 08:08:12: <span class="error">&#91;&lt;ffffffffa02f8d8d&gt;&#93;</span> encode_post_op_attr+0x5d/0xc0 <span class="error">&#91;nfsd&#93;</span><br/> 08:08:13: <span class="error">&#91;&lt;ffffffff81182c55&gt;&#93;</span> ? fput+0x25/0x30<br/> 08:08:13: <span class="error">&#91;&lt;ffffffffa02ef123&gt;&#93;</span> ? nfsd_write+0xf3/0x100 <span class="error">&#91;nfsd&#93;</span><br/> 08:08:13: <span class="error">&#91;&lt;ffffffffa02f9752&gt;&#93;</span> encode_wcc_data+0x72/0xd0 <span class="error">&#91;nfsd&#93;</span><br/> 08:08:13: <span class="error">&#91;&lt;ffffffffa02f987b&gt;&#93;</span> nfs3svc_encode_writeres+0x1b/0x80 <span class="error">&#91;nfsd&#93;</span><br/> 08:08:14: <span class="error">&#91;&lt;ffffffffa02e84e6&gt;&#93;</span> nfsd_dispatch+0x1a6/0x240 <span class="error">&#91;nfsd&#93;</span><br/> 08:08:14: <span class="error">&#91;&lt;ffffffffa0260614&gt;&#93;</span> svc_process_common+0x344/0x640 <span class="error">&#91;sunrpc&#93;</span><br/> 08:08:15: <span class="error">&#91;&lt;ffffffff81063410&gt;&#93;</span> ? default_wake_function+0x0/0x20<br/> 08:08:15: <span class="error">&#91;&lt;ffffffffa0260c50&gt;&#93;</span> svc_process+0x110/0x160 <span class="error">&#91;sunrpc&#93;</span><br/> 08:08:16: <span class="error">&#91;&lt;ffffffffa02e8b62&gt;&#93;</span> nfsd+0xc2/0x160 <span class="error">&#91;nfsd&#93;</span><br/> 08:08:16: <span class="error">&#91;&lt;ffffffffa02e8aa0&gt;&#93;</span> ? nfsd+0x0/0x160 <span class="error">&#91;nfsd&#93;</span><br/> 08:08:16: <span class="error">&#91;&lt;ffffffff81096a36&gt;&#93;</span> kthread+0x96/0xa0<br/> 08:08:17: <span class="error">&#91;&lt;ffffffff8100c0ca&gt;&#93;</span> child_rip+0xa/0x20<br/> 08:08:18: <span class="error">&#91;&lt;ffffffff810969a0&gt;&#93;</span> ? kthread+0x0/0xa0<br/> 08:08:19: <span class="error">&#91;&lt;ffffffff8100c0c0&gt;&#93;</span> ? child_rip+0x0/0x20<br/> ==============================</p> <p>I suspect it is related with the following patch, which has been new landed to Lustre build: <a href="https://build.whamcloud.com/job/lustre-b2_4/42" class="external-link" target="_blank" rel="nofollow noopener">https://build.whamcloud.com/job/lustre-b2_4/42</a></p> <p>==============================<br/> Commit 3781d4465c9fa72120f35084d4eb5edb55a0b66a by Oleg Drokin<br/> <a href="https://jira.whamcloud.com/browse/LU-3027" title="Failure on test suite parallel-scale test_write_disjoint: invalid file size 140329 instead of 160376 = 20047 * 8" class="issue-link" data-issue-key="LU-3027"><del>LU-3027</del></a> clio: Do not shrink sublock at cancel</p> <p>Shrinking sublock at ldlm lock cancel time means whoever happened<br/> to attach to this lock just before will reenqueue the wrong lock.</p> <p>Test-Parameters: envdefinitions=SLOW=yes,ONLY=write_disjoint \<br/> clientdistro=el6 serverdistro=el6 clientarch=x86_64 \<br/> serverarch=x86_64 testlist=parallel-scale</p> <p>Change-Id: I8f2de683812621fb2f8d761cf2aceebc12868d75<br/> Signed-off-by: Oleg Drokin &lt;oleg.drokin@intel.com&gt;<br/> Signed-off-by: Jian Yu &lt;jian.yu@intel.com&gt;<br/> Reviewed-on: <a href="http://review.whamcloud.com/7481" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7481</a><br/> Tested-by: Hudson<br/> Tested-by: Maloo &lt;whamcloud.maloo@gmail.com&gt;<br/> Reviewed-by: Jinshan Xiong &lt;jinshan.xiong@intel.com&gt;<br/> Reviewed-by: Bobi Jam &lt;bobijam@gmail.com&gt;<br/> ==============================</p> <p>Yujian, my suggestion is to revert above patch to verify.</p> <p>Lustre build: <a href="https://build.whamcloud.com/job/lustre-b2_4/44" class="external-link" target="_blank" rel="nofollow noopener">https://build.whamcloud.com/job/lustre-b2_4/44</a> (2.4.1 RC1)<br/> Distro/Arch: RHEL6.4/x86_64</p> <p>parallel-scale-nfsv3 test iorssf hit the same failure:<br/> <a href="https://maloo.whamcloud.com/test_sets/3098fb4c-15e0-11e3-9aab-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/3098fb4c-15e0-11e3-9aab-52540035b04c</a></p> <p>Lustre build: <a href="http://build.whamcloud.com/job/lustre-b2_4/44" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_4/44</a> (2.4.1 RC1)<br/> Distro/Arch: RHEL6.4/x86_64<br/> FSTYPE=zfs</p> <p>parallel-scale-nfsv3 test iorfpp hit the same failure:<br/> <a href="https://maloo.whamcloud.com/test_sets/a07679fc-155a-11e3-8938-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/a07679fc-155a-11e3-8938-52540035b04c</a></p> <p>I hit the same error when running parallel-scale-nfsv4 subtest iorssf on SLES11 SP2 client:</p> <p><a href="https://maloo.whamcloud.com/test_sets/5daf4960-174d-11e3-9d30-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/5daf4960-174d-11e3-9d30-52540035b04c</a></p> <p>MDS console:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>15:43:02:Lustre: DEBUG MARKER: == parallel-scale-nfsv4 test iorssf: iorssf ========================================================== 15:42:54 (1378507374) 15:43:02:Lustre: DEBUG MARKER: lfs setstripe /mnt/lustre/d0.ior.ssf -c -1 15:43:02:LustreError: 27445:0:(lov_lock.c:674:lov_lock_enqueue()) ASSERTION( sublock-&gt;cll_conflict == ((void *)0) ) failed: 15:43:02:LustreError: 27445:0:(lov_lock.c:674:lov_lock_enqueue()) LBUG 15:43:02:Pid: 27445, comm: nfsd 15:43:03: 15:43:03:Call Trace: 15:43:03: [&lt;ffffffffa0479895&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] 15:43:03: [&lt;ffffffffa0479e97&gt;] lbug_with_loc+0x47/0xb0 [libcfs] 15:43:03: [&lt;ffffffffa0a944ee&gt;] lov_lock_enqueue+0x76e/0x850 [lov] 15:43:04: [&lt;ffffffffa06033dc&gt;] cl_enqueue_try+0xfc/0x300 [obdclass] 15:43:05: [&lt;ffffffffa06047cf&gt;] cl_enqueue_locked+0x6f/0x1f0 [obdclass] 15:43:05: [&lt;ffffffffa060541e&gt;] cl_lock_request+0x7e/0x270 [obdclass] 15:43:06: [&lt;ffffffffa0b64ab0&gt;] cl_glimpse_lock+0x180/0x490 [lustre] 15:43:06: [&lt;ffffffffa0b65325&gt;] cl_glimpse_size0+0x1a5/0x1d0 [lustre] 15:43:06: [&lt;ffffffffa0b15765&gt;] ll_inode_revalidate_it+0x1a5/0x1d0 [lustre] 15:43:08: [&lt;ffffffffa05eddc4&gt;] ? keys_fini+0xe4/0x130 [obdclass] 15:43:08: [&lt;ffffffffa0b157d9&gt;] ll_getattr_it+0x49/0x170 [lustre] 15:43:08: [&lt;ffffffffa0b15937&gt;] ll_getattr+0x37/0x40 [lustre] 15:43:09: [&lt;ffffffff8121d1e3&gt;] ? security_inode_getattr+0x23/0x30 15:43:09: [&lt;ffffffff81186d81&gt;] vfs_getattr+0x51/0x80 15:43:09: [&lt;ffffffffa0303541&gt;] nfsd4_encode_fattr+0x181/0x1390 [nfsd] 15:43:09: [&lt;ffffffff810096f0&gt;] ? __switch_to+0xd0/0x320 15:43:09: [&lt;ffffffff8150e570&gt;] ? thread_return+0x4e/0x76e 15:43:09: [&lt;ffffffff81064e6a&gt;] ? __cond_resched+0x2a/0x40 15:43:10: [&lt;ffffffff8121c436&gt;] ? security_task_setgroups+0x16/0x20 15:43:10: [&lt;ffffffff8109fbc5&gt;] ? set_groups+0x25/0x190 15:43:11: [&lt;ffffffffa02f3926&gt;] ? nfsd_setuser+0x126/0x2c0 [nfsd] 15:43:11: [&lt;ffffffffa02eb992&gt;] ? nfsd_setuser_and_check_port+0x62/0xb0 [nfsd] 15:43:12: [&lt;ffffffffa03047b9&gt;] nfsd4_encode_getattr+0x69/0x80 [nfsd] 15:43:12: [&lt;ffffffffa02fcd35&gt;] nfsd4_encode_operation+0x75/0x180 [nfsd] 15:43:12: [&lt;ffffffffa02fad35&gt;] nfsd4_proc_compound+0x195/0x490 [nfsd] 15:43:13: [&lt;ffffffffa02e843e&gt;] nfsd_dispatch+0xfe/0x240 [nfsd] 15:43:13: [&lt;ffffffffa0260614&gt;] svc_process_common+0x344/0x640 [sunrpc] 15:43:15: [&lt;ffffffff81063410&gt;] ? default_wake_function+0x0/0x20 15:43:15: [&lt;ffffffffa0260c50&gt;] svc_process+0x110/0x160 [sunrpc] 15:43:15: [&lt;ffffffffa02e8b62&gt;] nfsd+0xc2/0x160 [nfsd] 15:43:17: [&lt;ffffffffa02e8aa0&gt;] ? nfsd+0x0/0x160 [nfsd] 15:43:17: [&lt;ffffffff81096a36&gt;] kthread+0x96/0xa0 15:43:17: [&lt;ffffffff8100c0ca&gt;] child_rip+0xa/0x20 15:43:18: [&lt;ffffffff810969a0&gt;] ? kthread+0x0/0xa0 15:43:18: [&lt;ffffffff8100c0c0&gt;] ? child_rip+0x0/0x20 15:43:18: 15:43:18:Kernel panic - not syncing: LBUG </pre> </div></div> <p>After patch <a href="http://review.whamcloud.com/7481" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7481</a> was reverted from Lustre b2_4 branch, the failure did not occur on Lustre 2.4.1 RC2.</p> <p>Duplicate of <a href="https://jira.whamcloud.com/browse/LU-3027" title="Failure on test suite parallel-scale test_write_disjoint: invalid file size 140329 instead of 160376 = 20047 * 8" class="issue-link" data-issue-key="LU-3027"><del>LU-3027</del></a></p> <p>Just FYI:<br/> I see <a href="https://jira.whamcloud.com/browse/LU-3027" title="Failure on test suite parallel-scale test_write_disjoint: invalid file size 140329 instead of 160376 = 20047 * 8" class="issue-link" data-issue-key="LU-3027"><del>LU-3027</del></a> clio was reverted from master (<a href="http://review.whamcloud.com/#/c/7749/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/7749/</a>) by Oleg to resolve this issue. Cray has continued carrying that patch in our 2.5 without any problems.</p> <p>I just tested my original reproducer for <a href="https://jira.whamcloud.com/browse/LU-3027" title="Failure on test suite parallel-scale test_write_disjoint: invalid file size 140329 instead of 160376 = 20047 * 8" class="issue-link" data-issue-key="LU-3027"><del>LU-3027</del></a>, which was fixed by the now reverted <a href="https://jira.whamcloud.com/browse/LU-3027" title="Failure on test suite parallel-scale test_write_disjoint: invalid file size 140329 instead of 160376 = 20047 * 8" class="issue-link" data-issue-key="LU-3027"><del>LU-3027</del></a> clio patch, on current master. (Reproducer is described in <a href="https://jira.whamcloud.com/browse/LU-3889" title=" LBUG: (osc_lock.c:497:osc_lock_upcall()) ASSERTION( lock-&gt;cll_state &gt;= CLS_QUEUING ) " class="issue-link" data-issue-key="LU-3889"><del>LU-3889</del></a>)<br/> That reproducer no longer hits any bugs, so it looks like that problem has been fixed by some other patch that's been landed.</p> Related LU-3027 Development Rank 1|hzvzv3: Rank (Obsolete) 10055 Severity