https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9367 Lustre <p>This issue was created by maloo for sarah_lw &lt;wei3.liu@intel.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.hpdd.intel.com/test_sets/45d48942-2507-11e7-9de9-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/45d48942-2507-11e7-9de9-5254006e85c2</a>.</p> <p>The sub-test test_cascading_rw failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>cascading_rw failed! 1 </pre> </div></div> <p>server/client: lustre-master #3558 ldiskfs el7</p> <p>test log</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>+ su mpiuser sh -c "/usr/lib64/compat-openmpi16/bin/mpirun --mca btl tcp,self --mca btl_tcp_if_include eth0 -mca boot ssh -machinefile /tmp/parallel-scale.machines -np 4 /usr/lib64/lustre/tests/cascading_rw -g -d /mnt/lustre/d0.cascading_rw -n 300 " -------------------------------------------------------------------------- A deprecated MCA parameter value was specified in an MCA parameter file. Deprecated MCA parameters should be avoided; they may disappear in future releases. Deprecated parameter: plm_rsh_agent -------------------------------------------------------------------------- -------------------------------------------------------------------------- A deprecated MCA parameter value was specified in an MCA parameter file. Deprecated MCA parameters should be avoided; they may disappear in future releases. Deprecated parameter: plm_rsh_agent -------------------------------------------------------------------------- -------------------------------------------------------------------------- A deprecated MCA parameter value was specified in an MCA parameter file. Deprecated MCA parameters should be avoided; they may disappear in future releases. Deprecated parameter: plm_rsh_agent -------------------------------------------------------------------------- -------------------------------------------------------------------------- A deprecated MCA parameter value was specified in an MCA parameter file. Deprecated MCA parameters should be avoided; they may disappear in future releases. Deprecated parameter: plm_rsh_agent -------------------------------------------------------------------------- -------------------------------------------------------------------------- A deprecated MCA parameter value was specified in an MCA parameter file. Deprecated MCA parameters should be avoided; they may disappear in future releases. Deprecated parameter: plm_rsh_agent -------------------------------------------------------------------------- -------------------------------------------------------------------------- A deprecated MCA parameter value was specified in an MCA parameter file. Deprecated MCA parameters should be avoided; they may disappear in future releases. Deprecated parameter: plm_rsh_agent -------------------------------------------------------------------------- /usr/lib64/lustre/tests/cascading_rw is running with 4 process(es) in DEBUG mode 23:47:45: Running test #/usr/lib64/lustre/tests/cascading_rw(iter 0) [trevis-55vm1:21694] *** Process received signal *** [trevis-55vm1:21694] Signal: Floating point exception (8) [trevis-55vm1:21694] Signal code: Integer divide-by-zero (1) [trevis-55vm1:21694] Failing at address: 0x4024c8 [trevis-55vm1:21694] [ 0] /lib64/libpthread.so.0(+0xf370) [0x7fdf9fad6370] [trevis-55vm1:21694] [ 1] /usr/lib64/lustre/tests/cascading_rw() [0x4024c8] [trevis-55vm1:21694] [ 2] /usr/lib64/lustre/tests/cascading_rw() [0x402be0] [trevis-55vm1:21694] [ 3] /usr/lib64/lustre/tests/cascading_rw() [0x40158e] [trevis-55vm1:21694] [ 4] /lib64/libc.so.6(__libc_start_main+0xf5) [0x7fdf9f727b35] [trevis-55vm1:21694] [ 5] /usr/lib64/lustre/tests/cascading_rw() [0x40169d] [trevis-55vm1:21694] *** End of error message *** [trevis-55vm1.trevis.hpdd.intel.com][[36239,1],2][btl_tcp_frag.c:215:mca_btl_tcp_frag_recv] mca_btl_tcp_frag_recv: readv failed: Connection reset by peer (104) [trevis-55vm2.trevis.hpdd.intel.com][[36239,1],1][btl_tcp_frag.c:215:mca_btl_tcp_frag_recv] mca_btl_tcp_frag_recv: readv failed: Connection reset by peer (104) -------------------------------------------------------------------------- mpirun noticed that process rank 0 with PID 21694 on node trevis-55vm1.trevis.hpdd.intel.com exited on signal 8 (Floating point exception). -------------------------------------------------------------------------- parallel-scale test_cascading_rw: @@@@@@ FAIL: cascading_rw failed! 1 Trace dump: = /usr/lib64/lustre/tests/test-framework.sh:4905:error() = /usr/lib64/lustre/tests/functions.sh:734:run_cascading_rw() = /usr/lib64/lustre/tests/parallel-scale.sh:130:test_cascading_rw() = /usr/lib64/lustre/tests/test-framework.sh:5181:run_one() = /usr/lib64/lustre/tests/test-framework.sh:5220:run_one_logged() = /usr/lib64/lustre/tests/test-framework.sh:5067:run_test() = /usr/lib64/lustre/tests/parallel-scale.sh:132:main() </pre> </div></div> LU-9367

parallel-scale test_cascading_rw: cascading_rw failed! 1

Bug Critical Resolved Fixed Zhenyu Xu Maloo Wed, 19 Apr 2017 18:12:28 +0000 Mon, 12 Jun 2017 22:39:51 +0000 Mon, 12 Jun 2017 22:39:51 +0000 Lustre 2.10.0 Lustre 2.10.0 0 8 <p>parallel-scale test_cascading_rw started failing on April 11, 2017 and has failed 59 times since that time. The failures are all for the 'full' test group, but no clear configuration is always failing; there are some interop failures, zfs and ldiskfs failures, and SLES11 SP4, CentOS 6.8 or 7 failures.</p> <p>The logs for the first three failures are at:<br/> <a href="https://testing.hpdd.intel.com/test_sets/b1cc400c-1e83-11e7-8920-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/b1cc400c-1e83-11e7-8920-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/d2e19c60-1eba-11e7-b742-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/d2e19c60-1eba-11e7-b742-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/70e2c49a-1e9b-11e7-9de9-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/70e2c49a-1e9b-11e7-9de9-5254006e85c2</a></p> <p>This looks like right after the PFL feature landed (April 8), so it probably makes sense for Bobijam to take a look at it.</p> <p>commit fafe6b4d4a6fa63cedff3bd44e6578009578b3d7 changes ll_lov_setstripe()</p> <div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader" style="border-bottom-width: 1px;"><b>static int ll_lov_setstripe(struct inode *inode, struct file *file, unsigned long arg)</b></div><div class="codeContent panelContent"> <pre class="code-java">@@ -1694,14 +1703,6 @@ <span class="code-keyword">static</span> <span class="code-object">int</span> ll_lov_setstripe(struct inode *inode, struct file *file, lum_size = rc; rc = ll_lov_setstripe_ea_info(inode, file, flags, klum, lum_size); - <span class="code-keyword">if</span> (rc == 0) { - __u32 gen; - - put_user(0, &amp;lum-&gt;lmm_stripe_count); - - ll_layout_refresh(inode, &amp;gen); - rc = ll_file_getstripe(inode, (struct lov_user_md __user *)arg); - } OBD_FREE(klum, lum_size); RETURN(rc); </pre> </div></div> <p>The thinking behind it is that ll_lov_setstripe() only uses the lum to set the file's stripe, while it shouldn't have the side effect to retrieve the instantiated stripe info back to the lum.</p> <p>And lustre/tests/mpi/cascading_rw.c just exploits the side effect of stripe retrieving during the stripe setting call.</p> <div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader" style="border-bottom-width: 1px;"><b>void rw_file(char *name, long stride, unsigned int seed)</b></div><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">if</span> (rank == 0) { remove_file_or_dir(filename); lum.lmm_magic = LOV_USER_MAGIC; lum.lmm_stripe_size = 0; lum.lmm_stripe_count = 0; lum.lmm_stripe_offset = -1; fd = open(filename, O_CREAT | O_RDWR | O_LOV_DELAY_CREATE, FILEMODE); <span class="code-keyword">if</span> (fd == -1) { sprintf(errmsg, <span class="code-quote">"open of file %s"</span>, filename); FAIL(errmsg); } rc = ioctl(fd, LL_IOC_LOV_SETSTRIPE, &amp;lum); <span class="code-keyword">if</span> (rc == -1) { sprintf(errmsg, <span class="code-quote">"ioctl SETSTRIPE of file %s"</span>, filename); FAIL(errmsg); } <span class="code-keyword">if</span> (close(fd) == -1) { sprintf(errmsg, <span class="code-quote">"close of file %s"</span>, filename); FAIL(errmsg); } } MPI_Barrier(MPI_COMM_WORLD); <span class="code-keyword">if</span> (stride &lt; 0) { <span class="code-keyword">if</span> (rank == 0) { srandom(seed); <span class="code-keyword">while</span> (stride &lt; page_size/2) { stride = random(); stride -= stride % 16; <span class="code-keyword">if</span> (stride &lt; 0) stride = -stride; stride %= 2 * lum.lmm_stripe_size; <span class="code-comment">// *DEVIDE BY ZERO EXCEPTION HERE* </span> } } MPI_Barrier(MPI_COMM_WORLD); </pre> </div></div> <p>We can fix the cascading_rw exploit by calling ioctl(fd, LL_IOC_LOV_GETSTRIPE, &amp;lum) after SETSTRIPE call so the lum.lmm_stripe_size is the real stripe_size instead of 0.</p> <p>Bobi Jam (bobijam@hotmail.com) uploaded a new patch: <a href="https://review.whamcloud.com/26915" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/26915</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9367" title="parallel-scale test_cascading_rw: cascading_rw failed! 1 " class="issue-link" data-issue-key="LU-9367"><del>LU-9367</del></a> mpi: get rid of SETSTRIPE side effect<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 7a493a89ea6bc5ea9f7429d9507891741905d4d8</p> <p>I'm not sure why you consider this to be a side effect? There has been code to explicitly call getstripe and return this to userspace forever. Why not just restore the call to ll_file_getstripe()?</p> <p>I am fine to restore the call to ll_file_getstripe(). And I always think the get/set methods to be with one direction data flow, and with that in mind, the ll_lov_setstripe() passes lum back to the caller is a side effect of that function to me.</p> <p>And</p> <p>#define LL_IOC_LOV_SETSTRIPE _IOW ('f', 154, long)<br/> #define LL_IOC_LOV_GETSTRIPE _IOW ('f', 155, long)</p> <p>shows that LL_IOC_LOV_SETSTRIPE should be WRITE ONLY ioctl(). And it seems that there is an error for LL_IOC_LOV_GETSTRIPE interface definition.</p> <p>This might be related to <a href="https://jira.whamcloud.com/browse/LU-9429" title="parallel-scale test_parallel_grouplock: test failed to respond and timed out" class="issue-link" data-issue-key="LU-9429">LU-9429</a>. The parallel_grouplock subtest started failing tag testing after 2017-04-05. In each case, parallel_grouplock times out after cascading_rw fails.</p> <p>tag 55 test (b3550): 2017-04-05 (parallel_grouplock 100% passing)<br/> several PFL landings: 2017-04-08<br/> tag 56 test (b3565): 2017-04-23 (parallel_grouplock 100% failing)</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/26915/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/26915/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9367" title="parallel-scale test_cascading_rw: cascading_rw failed! 1 " class="issue-link" data-issue-key="LU-9367"><del>LU-9367</del></a> llite: restore ll_file_getstripe in ll_lov_setstripe<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 364ec95f3688ac5cc3195f7f46d0d860844796f9</p> <p>Landed for 2.10</p> Related LU-9349 LU-9429 Development Rank 1|hzzan3: Rank (Obsolete) 9223372036854775807 Severity