https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-157 Lustre <p>metabench test failed on lustre client, can be reproduced.</p> <p>test log<br/> -----------<br/> <span class="error">&#91;03/23/2011 23:15:53&#93;</span> Leaving time_file_creation with proc_id = 11<br/> <span class="error">&#91;03/23/2011 23:15:53&#93;</span> Entering par_create_multidir to create 910 files in 1 dirs<br/> Removed 10000 files in 8.325 seconds<br/> <span class="error">&#91;client-5.lab.whamcloud.com:6909&#93;</span> *** An error occurred in MPI_Gather<br/> <span class="error">&#91;client-5.lab.whamcloud.com:6909&#93;</span> *** on communicator MPI COMMUNICATOR 14 CREATEE<br/> FROM 0<br/> <span class="error">&#91;client-5.lab.whamcloud.com:6909&#93;</span> *** MPI_ERR_TRUNCATE: message truncated<br/> <span class="error">&#91;client-5.lab.whamcloud.com:6909&#93;</span> *** MPI_ERRORS_ARE_FATAL (your MPI job will noo<br/> w abort)<br/> --------------------------------------------------------------------------<br/> mpirun has exited due to process rank 0 with PID 6909 on<br/> node client-5.lab.whamcloud.com exiting without calling "finalize". This may<br/> have caused other processes in the application to be<br/> terminated by signals sent by mpirun (as reported here).<br/> --------------------------------------------------------------------------</p> separated MDS and OSS, 3 clients LU-157

metabench failed on parallel-scale test

Bug Minor Resolved Fixed Michael MacDonald Sarah Liu Thu, 24 Mar 2011 12:55:14 +0000 Fri, 20 May 2011 16:14:00 +0000 Fri, 20 May 2011 16:14:00 +0000 Lustre 2.1.0 Lustre 1.8.6 Lustre 2.1.0 Lustre 1.8.6 0 6 <p>Ah sorry Yu Jian. This was the ticket that I meant to assign to you when I assigned LU158 to you. Could you please see what you can uncover about the failure? Thanks.</p> <p>Apparently Fan Yong is working on this one</p> <p>it seems this is a duplicate issue with <a href="https://jira.whamcloud.com/browse/LU-142" title="system hang when running replay-single or replay-dual with three clients" class="issue-link" data-issue-key="LU-142"><del>LU-142</del></a>, so close it.</p> <p>miss closing, reopen.this is not a duplicated with 161/142</p> <p>Fan Yong</p> <p>Are you able to look into this one or should I reassign it?</p> <p>Peter</p> <p>I will investigate it.</p> <p>After some painful debug (because of no familiar with MPI), I found some useful clue eventually. It is the incompatibility between openmpi and metabench installed on Toro nodes caused the failure. I compiled metabench from source code (tiny fix metabench code because of unknown parameter from MPI lib when start, not sure why) with openmpi-devel installed, then metabench test run successfully. So it is nothing related with Lustre. We need new MPI lib or metabench when deploying test nodes on Toro, but it is out of my control.</p> <p>Thanks yujian for the help of building test environment.</p> <p>thanks Fan Yong. Let's reassign this ticket to mjmac to sort out the Toro config</p> <p>fanyong, can you please provide more detail as to what the problem was, and how you fixed it? I would like to update the toolkit build so that future test installs work correctly.</p> <p>Currently, the openmpi installed on Toro nodes is openmpi-1.4-4.el5, the metabench is metabench-1.0-1.wc1, there are some incompatibility between them. I do not know the detailed reason, but you can try as following:</p> <p>1) install openmpi-devel on your test node<br/> 2) compile metabench from source code.<br/> 3) run parallel-scale with new metabench, it will report "Invalid Arg ?"<br/> 4) fix the metabench.c to ignore such unknown parameter "?", and recompile<br/> 5) then run parallel-scale again, it can pass.</p> <p>I have put the workable metabench under /tmp/metabench on Brent node, which can run on 2.6.18-194.17.1.el5. I am not sure how to fix it easily, maybe use "MPICH" or fix metabench.</p> <p>Branch: b1_8 (Revision: c5c2986be490b2fbceb4b38d6c983d279f4bbcf8)<br/> Distro/Arch: RHEL6/x86_64 (patchless client), RHEL5/x86_64 (server)<br/> Network: tcp</p> <p>&#35; rpm -qf /usr/lib64/openmpi/bin/mpirun<br/> openmpi-1.4.1-4.3.el6.x86_64</p> <p>&#35; rpm -qf /usr/bin/metabench<br/> metabench-1.0-1.wc1.x86_64</p> <p>The same failure occurred while running metabench test:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[client-13:9663] *** An error occurred in MPI_Gather [client-13:9663] *** on communicator MPI COMMUNICATOR 10 CREATE FROM 0 [client-13:9663] *** MPI_ERR_TRUNCATE: message truncated [client-13:9663] *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort) -------------------------------------------------------------------------- mpirun has exited due to process rank 0 with PID 9663 on node client-13 exiting without calling "finalize". This may have caused other processes in the application to be terminated by signals sent by mpirun (as reported here). -------------------------------------------------------------------------- </pre> </div></div> <p>Maloo report: <a href="https://maloo.whamcloud.com/test_sets/4673d0c8-6cb3-11e0-b32b-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/4673d0c8-6cb3-11e0-b32b-52540025f9af</a></p> <p>The real problem is that the receive buffer was defined as MPI_INT but the send buffer was defined as MPI_UNSIGNED_LONG. When compiled with gcc on x86_64, longs (8 bytes) don't fit into ints (4 bytes), hence the MPI_ERR_TRUNCATE error.</p> <p>I've committed a small patch which corrects this, and I'm waiting for RPMs to build across all platforms. I've already verified this on EL6/x86_64; please resolve the ticket when other platforms are verified in the normal course of testing. I'm confident that this issue is fixed, though, as it was a simple problem with a simple solution, once I understood the problem!</p> <p>For reference, here is an excerpt from the patch:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> MPI_SAFE(MPI_Gather(&amp;count,1,MPI_UNSIGNED_LONG, - count_buf,1,MPI_INT,proc0,*my_comm)); + count_buf,1,MPI_UNSIGNED_LONG,proc0,*my_comm)); </pre> </div></div> <p>I've applied this fix to all MPI_Gather instances with mismatched send/receive datatypes.</p> <p>Hi Mike, I ran cascading_rw on build lustre-master/rhel6-x86_64/#118, it continued running for almost two days and hasn't finished yet. Does this build contain the latest openmpi? </p> <p>I verified this bug on the latest last-master for RHEL5-x86_84, metabench passes on NFS3 but failed on NFS4. I think the failure is not related to mpi, so open a new ticket for tracking,<a href="https://jira.whamcloud.com/browse/LU-344" title="Test failure on test suite parallel-scale" class="issue-link" data-issue-key="LU-344"><del>LU-344</del></a>.</p> <p>Here are both results:<br/> <a href="https://maloo.whamcloud.com/test_sets/0df23b78-81de-11e0-b4df-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/0df23b78-81de-11e0-b4df-52540025f9af</a><br/> <a href="https://maloo.whamcloud.com/test_sets/3c243272-81e2-11e0-b4df-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/3c243272-81e2-11e0-b4df-52540025f9af</a></p> <p>I'm going to resolve this, as the original issue with bad code in metabench has been fixed. Please open new tickets for the other problems (e.g. cascading_rw).</p> Development Rank 1|hzw00v: Rank (Obsolete) 10083 Severity