https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6937 Lustre <p>This issue was created by maloo for sarah_lw &lt;wei3.liu@intel.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.hpdd.intel.com/test_sets/d07a7a8c-3799-11e5-9d53-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/d07a7a8c-3799-11e5-9d53-5254006e85c2</a>.</p> <p>The sub-test test_106 failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>uuid 0 not found in agent list on mds1 </pre> </div></div> <p>test log</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== sanity-hsm test 106: Copytool register/unregister ================================================= 13:35:57 (1438202157) CMD: onyx-30vm5 pkill -CONT -x lhsmtool_posix Purging archive on onyx-30vm5 CMD: onyx-30vm5 rm -rf /tmp/arc1/* Starting copytool agt1 on onyx-30vm5 CMD: onyx-30vm5 mkdir -p /tmp/arc1 CMD: onyx-30vm5 lhsmtool_posix --daemon --hsm-root /tmp/arc1 --bandwidth 1 /mnt/lustre2 &lt; /dev/null &gt; /logdir/test_logs/2015-07-29/lustre-master-el7-x86_64--full--1_7_1__3120__-70238999807760-034047/sanity-hsm.test_106.copytool_log.onyx-30vm5.log 2&gt;&amp;1 CMD: onyx-30vm5 PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/utils:/usr/lib64/qt-3.3/bin:/usr/lib64/openmpi/bin:/usr/bin:/bin:/usr/sbin:/sbin::/sbin:/usr/sbin:/sbin:/bin:/usr/sbin: NAME=autotest_config sh rpc.sh pgrep -fl lhsmtool_posix CMD: onyx-30vm5 PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests//usr/lib64/lustre/tests:/usr/lib64/lustre/tests:/usr/lib64/lustre/tests//usr/lib64/lustre/tests/utils:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/utils:/usr/lib64/qt-3.3/bin:/usr/lib64/openmpi/bin:/usr/bin:/bin:/usr/sbin:/sbin::/sbin:/usr/sbin:/sbin:/bin:/usr/sbin: NAME=autotest_config sh rpc.sh get_client_uuid lhsmtool_posix bash bash sh onyx-30vm5: error opening lhsmtool_posix: No such file or directory (2) onyx-30vm5: cannot get name for `lhsmtool_posix': No such file or directory onyx-30vm5: error: get_param: llite//uuid: Found no match CMD: onyx-30vm3 /usr/sbin/lctl get_param -n mdt.lustre-MDT000.hsm.agents | grep 0 onyx-30vm3: error: get_param: mdt/lustre-MDT000/hsm/agents: Found no match sanity-hsm test_106: @@@@@@ FAIL: uuid 0 not found in agent list on mds1 </pre> </div></div> <p>Info required for matching: sanity-hsm 106<br/> Info required for matching: sanity-hsm 403</p> client and server: lustre-master build # 3120 RHEL7 DNE LU-6937

sanity-hsm test_106 test_403: error opening lhsmtool_posix: No such file or directory

Bug Minor Resolved Fixed Bruno Faccini Maloo Fri, 31 Jul 2015 22:25:31 +0000 Fri, 4 Aug 2017 18:04:05 +0000 Wed, 11 May 2016 09:15:07 +0000 Lustre 2.8.0 Lustre 2.9.0 Lustre 2.9.0 0 10 <p>master, build# 3264, 2.7.64 tag<br/> Regression:EL7.1 Server/EL7.1 Client<br/> <a href="https://testing.hpdd.intel.com/test_sets/6a2f8cf8-9f37-11e5-ba94-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/6a2f8cf8-9f37-11e5-ba94-5254006e85c2</a></p> <p>Server: 2.5.5, b2_5_fe/62<br/> Client: Master, Build# 3266, Tag 2.7.64 , RHEL 7 <br/> <a href="https://testing.hpdd.intel.com/test_sets/0d1a952a-a05f-11e5-90cc-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/0d1a952a-a05f-11e5-90cc-5254006e85c2</a></p> <p>Another instance forEL7.1 Server/EL7.1 Client - ZFS<br/> Master, build# 3264<br/> <a href="https://testing.hpdd.intel.com/test_sets/3688a1f0-a135-11e5-83b8-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/3688a1f0-a135-11e5-83b8-5254006e85c2</a></p> <p>Another instance for EL7.1 Server/EL7.1 Client - DNE<br/> Master , Build# 3270<br/> <a href="https://testing.hpdd.intel.com/test_sets/57f8b5c6-a26d-11e5-bdef-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/57f8b5c6-a26d-11e5-bdef-5254006e85c2</a></p> <p>Another instance found for :<br/> Server: 2.7.1 , b2_7_fe/34<br/> Client: Master , Build# 3276 RHEL 7<br/> <a href="https://testing.hpdd.intel.com/test_sets/925ad49c-a5e2-11e5-a028-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/925ad49c-a5e2-11e5-a028-5254006e85c2</a></p> <p>Another instance found for interop : 2.7.1 Server/EL7 Client<br/> Server: 2.7.1, b2_7_fe/34<br/> Client: master, build# 3303, RHEL 7 <br/> <a href="https://testing.hpdd.intel.com/test_sets/2df919c4-bb03-11e5-9137-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/2df919c4-bb03-11e5-9137-5254006e85c2</a></p> <p>Another instance found for interop : 2.5.5 Server/EL7 Client<br/> Server: 2.5.5, b2_5_fe/62<br/> Client: master, build# 3303, RHEL 7<br/> <a href="https://testing.hpdd.intel.com/test_sets/60ff2eba-bb0a-11e5-87b4-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/60ff2eba-bb0a-11e5-87b4-5254006e85c2</a></p> <p>Encountered another instance for tag 2.7.66 for FULL - EL7.1 Server/EL7.1 Client , master , build# 3314.<br/> <a href="https://testing.hpdd.intel.com/test_sets/9f0a6890-ca88-11e5-84d3-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/9f0a6890-ca88-11e5-84d3-5254006e85c2</a></p> <p>Another instance for FULL - EL7.1 Server/EL7.1 Client - DNE, master, build# 3314<br/> <a href="https://testing.hpdd.intel.com/test_sets/911db61e-cac5-11e5-9609-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/911db61e-cac5-11e5-9609-5254006e85c2</a></p> <p>Another instance on master for FULL - EL7.1 Server/EL7.1 Client - ZFS, build# 3314<br/> <a href="https://testing.hpdd.intel.com/test_sets/c55bc81c-cb88-11e5-b49e-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/c55bc81c-cb88-11e5-b49e-5254006e85c2</a></p> <p>Another instance found for interop tag 2.7.66 - 2.7.1 Server/EL7 Client, build# 3316<br/> <a href="https://testing.hpdd.intel.com/test_sets/7a4d59a6-ccde-11e5-8b0e-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/7a4d59a6-ccde-11e5-8b0e-5254006e85c2</a></p> <p>Another instance found for interop tag 2.7.66 - 2.5.5 Server/EL7 Client, build# 3316<br/> <a href="https://testing.hpdd.intel.com/test_sets/1816312e-ccc8-11e5-b80c-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/1816312e-ccc8-11e5-b80c-5254006e85c2</a></p> <p>Another instance found for Full tag 2.7.66 - EL7.1 Server/EL7.1 Client, build# 3314<br/> <a href="https://testing.hpdd.intel.com/test_sets/9f0a6890-ca88-11e5-84d3-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/9f0a6890-ca88-11e5-84d3-5254006e85c2</a></p> <p>Another instance found for Full tag 2.7.66 -EL7.1 Server/EL7.1 Client - ZFS, build# 3314<br/> <a href="https://testing.hpdd.intel.com/test_sets/c55bc81c-cb88-11e5-b49e-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/c55bc81c-cb88-11e5-b49e-5254006e85c2</a></p> <p>Another instance found for Full tag 2.7.66 -EL7.1 Server/EL7.1 Client - DNE, build# 3314<br/> <a href="https://testing.hpdd.intel.com/test_sets/911db61e-cac5-11e5-9609-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/911db61e-cac5-11e5-9609-5254006e85c2</a></p> <p>Another instance found for interop - 2.7.1 Server/EL7 Client, tag 2.7.90. <br/> <a href="https://testing.hpdd.intel.com/test_sessions/3b9722f8-d2f8-11e5-bf08-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/3b9722f8-d2f8-11e5-bf08-5254006e85c2</a><br/> Another instance found for interop - 2.5.5 Server/EL7 Client, tag 2.7.90. <br/> <a href="https://testing.hpdd.intel.com/test_sessions/ba9d84fe-d300-11e5-be5c-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/ba9d84fe-d300-11e5-be5c-5254006e85c2</a></p> <p>Another instance: <a href="https://testing.hpdd.intel.com/test_sets/8f52b0de-fe26-11e5-b1c9-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/8f52b0de-fe26-11e5-b1c9-5254006e85c2</a></p> <p>lhsmtools_posix isn't found, It looks like a testing environment problem.</p> <p>This test is failing quite regularly, about 100 times in the past 4 weeks. Many of the failures are in "full" test runs, so I'm not sure if there is a different test configuration there (RHEL7 vs. RHEL6) or some other potential problem.</p> <p>Both sanity-hsm test_106 and test_403 also appear to be failing much more frequently on the PFL patches. I'm not sure why that is, but as many failures are on PFL patches as all other patches put together, so there may be something in the PFL patches that are triggering this problem more frequently.</p> <p>from the previous comments, it looks like this issue only affects RHEL7 </p> <p>This is likely due to a change in the output of pgrep. With the pgrep from CentOS 6.7:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># pgrep -V pgrep (procps version 3.2.8) # pgrep -fl lhsmtool_posix 24726 lhsmtool_posix --daemon --hsm-root /tmp/arc1/shsm --bandwidth 1 /mnt/lustre2 </pre> </div></div> <p>With a newer pgrep:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># pgrep -V pgrep from procps-ng 3.3.9 # pgrep -fl lhsmtool_posix 24735 lhsmtool_posix </pre> </div></div> <p>John L. Hammond (john.hammond@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/20093" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/20093</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6937" title="sanity-hsm test_106 test_403: error opening lhsmtool_posix: No such file or directory" class="issue-link" data-issue-key="LU-6937"><del>LU-6937</del></a> test: use ps in get_agent_uuid()<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 3b81057aadaf389e329320d1f8275f79425159b1</p> <p>Oleg Drokin (oleg.drokin@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/20096" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/20096</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6937" title="sanity-hsm test_106 test_403: error opening lhsmtool_posix: No such file or directory" class="issue-link" data-issue-key="LU-6937"><del>LU-6937</del></a> test: use ps in get_agent_uuid()<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: bc23e9b1983124e36939dbd1fecce15d010fda61</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/20093/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/20093/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6937" title="sanity-hsm test_106 test_403: error opening lhsmtool_posix: No such file or directory" class="issue-link" data-issue-key="LU-6937"><del>LU-6937</del></a> test: use ps in get_agent_uuid()<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: af4411d6afa93acccebd791b6e974fd84ca8eb33</p> <p>It's too bad that this was failing for so long on RHEL7 but we didn't fix it until it broke master testing... At least it is now fixed.</p> <p>Doug Oucharek (doug.s.oucharek@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/20341" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/20341</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6937" title="sanity-hsm test_106 test_403: error opening lhsmtool_posix: No such file or directory" class="issue-link" data-issue-key="LU-6937"><del>LU-6937</del></a> test: use ps in get_agent_uuid()<br/> Project: fs/lustre-release<br/> Branch: multi-rail<br/> Current Patch Set: 1<br/> Commit: e28e633b16bd7bee2994614872b2b65992367d0a</p> <p>Amir Shehata (amir.shehata@intel.com) merged in patch <a href="http://review.whamcloud.com/20341/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/20341/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6937" title="sanity-hsm test_106 test_403: error opening lhsmtool_posix: No such file or directory" class="issue-link" data-issue-key="LU-6937"><del>LU-6937</del></a> test: use ps in get_agent_uuid()<br/> Project: fs/lustre-release<br/> Branch: multi-rail<br/> Current Patch Set: <br/> Commit: 02e907afe1bc79fa32272b0afa7393d14bab35f8</p> <p>Jinshan Xiong (jinshan.xiong@intel.com) merged in patch <a href="http://review.whamcloud.com/20295/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/20295/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6937" title="sanity-hsm test_106 test_403: error opening lhsmtool_posix: No such file or directory" class="issue-link" data-issue-key="LU-6937"><del>LU-6937</del></a> test: use ps in get_agent_uuid()<br/> Project: fs/lustre-dev<br/> Branch: pfl<br/> Current Patch Set: <br/> Commit: fe9bd6cc3c02bad282f4dd7f8a1779764d3ca4ab</p> <p>Hit it on b2_8:<br/> <a href="https://testing.hpdd.intel.com/test_sets/750f521e-5aeb-11e6-aa74-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/750f521e-5aeb-11e6-aa74-5254006e85c2</a></p> <p>Still hit it on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/a5a1ecf0-7249-11e6-b08e-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/a5a1ecf0-7249-11e6-b08e-5254006e85c2</a></p> Related Development Rank 1|hzxjhz: Rank (Obsolete) 9223372036854775807 Severity