Whamcloud Community JIRA

Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us

9.4.14

940014

05-12-2023

[LU-1661] Test failure on test suite posix, subtest test_1 https://jira.whamcloud.com/browse/LU-1661 Lustre <p>This issue was created by maloo for sarah <sarah@whamcloud.com></p> <p>This issue relates to the following test suite run: <a href="https://maloo.whamcloud.com/test_sets/24c171c6-d2d6-11e1-8823-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/24c171c6-d2d6-11e1-8823-52540035b04c</a>.</p> <p>The sub-test test_1 failed with the following error:</p> <blockquote> <p>test failed to respond and timed out</p></blockquote> LU-1661

Test failure on test suite posix, subtest test_1

Bug Blocker Resolved Fixed Jian Yu Maloo Mon, 23 Jul 2012 19:17:40 +0000 Sun, 26 Aug 2012 11:33:15 +0000 Sun, 26 Aug 2012 11:33:15 +0000 Lustre 2.3.0 Lustre 2.3.0 Lustre 2.4.0 0 5 <p>Minh can you please comment?</p> <p>Unfortunately, the logs do not have any useful information. the "hung" is during mkfs on the loopback fs. I found a few instances of this failure. I am investigating this further</p> <p><a href="https://maloo.whamcloud.com/test_sets/query?utf8=%E2%9C%93&test_set%5Btest_set_script_id%5D=f60ca966-8b52-11e0-aab9-52540025f9af&test_set%5Bstatus%5D=TIMEOUT&test_set%5Bquery_bugs%5D=&test_session%5Btest_host%5D=&test_session%5Btest_group%5D=&test_session%5Buser_id%5D=&test_session%5Bquery_date%5D=&test_session%5Bquery_recent_period%5D=&test_node%5Bos_type_id%5D=&test_node%5Bdistribution_type_id%5D=&test_node%5Barchitecture_type_id%5D=&test_node%5Bfile_system_type_id%5D=&test_node%5Blustre_branch_id%5D=&test_node_network%5Bnetwork_type_id%5D=&commit=Update+results" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/query?utf8=%E2%9C%93&test_set%5Btest_set_script_id%5D=f60ca966-8b52-11e0-aab9-52540025f9af&test_set%5Bstatus%5D=TIMEOUT&test_set%5Bquery_bugs%5D=&test_session%5Btest_host%5D=&test_session%5Btest_group%5D=&test_session%5Buser_id%5D=&test_session%5Bquery_date%5D=&test_session%5Bquery_recent_period%5D=&test_node%5Bos_type_id%5D=&test_node%5Bdistribution_type_id%5D=&test_node%5Barchitecture_type_id%5D=&test_node%5Bfile_system_type_id%5D=&test_node%5Blustre_branch_id%5D=&test_node_network%5Bnetwork_type_id%5D=&commit=Update+results</a></p> <p>Yujian</p> <p>Could you please also look into this one while you are investigating <a href="https://jira.whamcloud.com/browse/LU-1580" title="POSIX: pathconf.32: pathconf("/dev/pts/0", _PC_NAME_MAX) did not give correct results" class="issue-link" data-issue-key="LU-1580"><del>LU-1580</del></a>?</p> <p>Thanks</p> <p>Peter</p> <blockquote><p>Unfortunately, the logs do not have any useful information. the "hung" is during mkfs on the loopback fs. I found a few instances of this failure. I am investigating this further</p></blockquote> <p>In fact, the test did not hang. It succeeded in running "setup_loop_dev", and went into "setup_posix_users", where the following issue occurred:</p> <div class="panel" style="border-width: 1px;"><div class="panelHeader" style="border-bottom-width: 1px;"><b>Syslog of client-27vm6</b></div><div class="panelContent"> <p>Jul 20 <font color="red">14:34:54</font> client-27vm6 kernel: Lustre: DEBUG MARKER: == posix test 1: build, install, run posix on ext4 and lustre, then compare ========================== 14:34:53 (1342820093)<br/> Jul 20 14:35:01 client-27vm6 rshd<span class="error">[10120]</span>: pam_unix(rsh:session): session closed for user root<br/> Jul 20 14:35:20 client-27vm6 xinetd<span class="error">[1915]</span>: EXIT: shell status=0 pid=10120 duration=0(sec)<br/> Jul 20 14:35:21 client-27vm6 kernel: loop: loaded (max 8 devices)<br/> Jul 20 14:35:21 client-27vm6 kernel: EXT4-fs (loop1): mounted filesystem with ordered data mode<br/> Jul 20 14:35:21 client-27vm6 xinetd<span class="error">[1915]</span>: START: shell pid=10294 from=10.10.4.159<br/> Jul 20 14:35:21 client-27vm6 rshd<span class="error">[10294]</span>: pam_rhosts_auth(rsh:auth): user root has a `+' user entry<br/> Jul 20 14:35:21 client-27vm6 rshd<span class="error">[10294]</span>: pam_rhosts_auth(rsh:auth): allowed to root@client-27vm6.lab.whamcloud.com as root<br/> Jul 20 14:35:21 client-27vm6 rshd<span class="error">[10294]</span>: pam_unix(rsh:session): session opened for user root by (uid=0)<br/> Jul 20 14:35:21 client-27vm6 rshd<span class="error">[10295]</span>: root@client-27vm6.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr<br/> /lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" USE_OFD= sh -c " error() { set +x; echo Error: \$2: \$1; echo XXRETCODE:\$1; exit \$1; } gid=\$(getent group vs<br/> xg0 | cut -d: -f3); if [ "x\$gid" != "x" ]; then [ \$gid -eq 1001 ] || \ error 1 \"inconsistent group ID: new: 1001, old: \$gid\"; e<br/> lse groupadd -g 1001 vsxg0 fi;");echo XXRETCODE:$?'<br/> Jul 20 14:35:21 client-27vm6 groupadd<span class="error">[10332]</span>: new group: name=vsxg0, GID=1001<br/> Jul 20 14:35:21 client-27vm6 rshd<span class="error">[10294]</span>: pam_unix(rsh:session): session closed for user root<br/> Jul 20 <font color="red">14:35:22 client-27vm6 xinetd<span class="error">[1915]</span>: EXIT: shell status=0 pid=10294 duration=1(sec)</font><br/> Jul 20 <font color="red">14:40:21 client-27vm6 xinetd<span class="error">[1915]</span>: START: shell pid=10367 from=10.10.4.159</font><br/> Jul 20 14:40:21 client-27vm6 rshd<span class="error">[10367]</span>: pam_rhosts_auth(rsh:auth): user root has a `+' user entry<br/> Jul 20 14:40:22 client-27vm6 rshd<span class="error">[10367]</span>: pam_rhosts_auth(rsh:auth): allowed to root@client-27vm6.lab.whamcloud.com as root<br/> Jul 20 14:40:22 client-27vm6 rshd<span class="error">[10367]</span>: pam_unix(rsh:session): session opened for user root by (uid=0)<br/> Jul 20 14:40:22 client-27vm6 rshd<span class="error">[10368]</span>: root@client-27vm6.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" USE_OFD= sh -c " error() { set +x; echo Error: \$2: \$1; echo XXRETCODE:\$1; exit \$1; } gid=\$(getent group vsxg1 | cut -d: -f3); if [ "x\$gid" != "x" ]; then [ \$gid -eq 1002 ] || \ error 1 \"inconsistent group ID: new: 1002, old: \$gid\"; else groupadd -g 1002 vsxg1 fi;");echo XXRETCODE:$?'<br/> Jul 20 14:40:22 client-27vm6 groupadd<span class="error">[10405]</span>: new group: name=vsxg1, GID=1002<br/> Jul 20 14:40:22 client-27vm6 rshd<span class="error">[10367]</span>: pam_unix(rsh:session): session closed for user root<br/> Jul 20 <font color="red">14:40:22 client-27vm6 xinetd<span class="error">[1915]</span>: EXIT: shell status=0 pid=10367 duration=1(sec)</font><br/> Jul 20 <font color="red">14:45:21 client-27vm6 xinetd<span class="error">[1915]</span>: START: shell pid=10440 from=10.10.4.159</font><br/> Jul 20 14:45:22 client-27vm6 rshd<span class="error">[10440]</span>: pam_rhosts_auth(rsh:auth): user root has a `+' user entry<br/> Jul 20 14:45:22 client-27vm6 rshd<span class="error">[10440]</span>: pam_rhosts_auth(rsh:auth): allowed to root@client-27vm6.lab.whamcloud.com as root<br/> Jul 20 14:45:22 client-27vm6 rshd<span class="error">[10440]</span>: pam_unix(rsh:session): session opened for user root by (uid=0)<br/> Jul 20 14:45:22 client-27vm6 rshd<span class="error">[10441]</span>: root@client-27vm6.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" USE_OFD= sh -c " error() { set +x; echo Error: \$2: \$1; echo XXRETCODE:\$1; exit \$1; } gid=\$(getent group vsxg2 | cut -d: -f3); if [ "x\$gid" != "x" ]; then [ \$gid -eq 1003 ] || \ error 1 \"inconsistent group ID: new: 1003, old: \$gid\"; else groupadd -g 1003 vsxg2 fi;");echo XXRETCODE:$?'<br/> Jul 20 14:45:22 client-27vm6 groupadd<span class="error">[10478]</span>: new group: name=vsxg2, GID=1003</p> <p><~snip~></p> <p>Jul 20 15:30:27 client-27vm6 groupadd<span class="error">[11144]</span>: new group: name=supp9, GID=1012<br/> Jul 20 15:30:27 client-27vm6 rshd<span class="error">[11106]</span>: pam_unix(rsh:session): session closed for user root<br/> Jul 20 <font color="red">15:30:27 client-27vm6 xinetd<span class="error">[1915]</span>: EXIT: shell status=0 pid=11106 duration=1(sec)</font><br/> Jul 20 <font color="red">15:35:28 client-27vm6 xinetd<span class="error">[1915]</span>: START: shell pid=11179 from=10.10.4.159</font> <br/> Jul 20 15:35:29 client-27vm6 rshd<span class="error">[11179]</span>: pam_rhosts_auth(rsh:auth): user root has a `+' user entry<br/> Jul 20 15:35:30 client-27vm6 rshd<span class="error">[11179]</span>: pam_rhosts_auth(rsh:auth): allowed to root@client-27vm6.lab.whamcloud.com as root<br/> Jul 20 15:35:30 client-27vm6 rshd<span class="error">[11179]</span>: pam_unix(rsh:session): session opened for user root by (uid=0)<br/> Jul 20 15:35:31 client-27vm6 rshd<span class="error">[11180]</span>: root@client-27vm6.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr<br/> /lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" USE_OFD= sh -c " error() { set +x; echo Error: \$2: \$1; echo XXRETCODE:\$1; exit \$1; } gid=\$(getent group su<br/> pp10 | cut -d: -f3); if [ "x\$gid" != "x" ]; then [ \$gid -eq 1013 ] || \ error 1 \"inconsistent group ID: new: 1013, old: \$gid\"; <br/> else groupadd -g 1013 supp10 fi;");echo XXRETCODE:$?'<br/> Jul 20 15:35:31 client-27vm6 groupadd<span class="error">[11217]</span>: new group: name=supp10, GID=1013<br/> Jul 20 15:35:32 client-27vm6 rshd<span class="error">[11179]</span>: pam_unix(rsh:session): session closed for user root<br/> Jul 20 15:35:32 client-27vm6 xinetd<span class="error">[1915]</span>: EXIT: shell status=0 pid=11179 duration=2(sec)<br/> Jul 20 15:35:38 client-27vm6 sshd<span class="error">[11221]</span>: Accepted publickey for autotest from 10.10.0.13 port 50279 ssh2<br/> Jul 20 15:35:39 client-27vm6 sshd<span class="error">[11221]</span>: pam_unix(sshd:session): session opened for user autotest by (uid=0)<br/> Jul 20 15:35:40 client-27vm6 sshd<span class="error">[11223]</span>: Received disconnect from 10.10.0.13: 11: disconnected by user<br/> Jul 20 15:35:40 client-27vm6 sshd<span class="error">[11221]</span>: pam_unix(sshd:session): session closed for user autotest<br/> Jul 20 15:35:42 client-27vm6 sshd<span class="error">[11226]</span>: Accepted publickey for autotest from 10.10.0.13 port 50281 ssh2<br/> Jul 20 15:35:42 client-27vm6 sshd<span class="error">[11226]</span>: pam_unix(sshd:session): session opened for user autotest by (uid=0)<br/> Jul 20 15:35:43 client-27vm6 sshd<span class="error">[11228]</span>: Received disconnect from 10.10.0.13: 11: disconnected by user<br/> Jul 20 15:35:44 client-27vm6 sshd<span class="error">[11226]</span>: pam_unix(sshd:session): session closed for user autotest<br/> Jul 20 15:35:48 client-27vm6 sshd<span class="error">[11231]</span>: Accepted publickey for autotest from 10.10.0.13 port 50282 ssh2<br/> Jul 20 15:35:49 client-27vm6 sshd<span class="error">[11231]</span>: pam_unix(sshd:session): session opened for user autotest by (uid=0)<br/> Jul 20 15:35:50 client-27vm6 xinetd<span class="error">[1915]</span>: START: shell pid=11235 from=10.10.4.159<br/> Jul 20 15:35:51 client-27vm6 rshd<span class="error">[11235]</span>: pam_rhosts_auth(rsh:auth): user root has a `+' user entry<br/> Jul 20 15:35:51 client-27vm6 rshd<span class="error">[11235]</span>: pam_rhosts_auth(rsh:auth): allowed to autotest@client-27vm6.lab.whamcloud.com as root<br/> Jul 20 15:35:52 client-27vm6 rshd<span class="error">[11235]</span>: pam_unix(rsh:session): session opened for user root by (uid=0)<br/> Jul 20 <font color="red">15:35:52</font> client-27vm6 rshd<span class="error">[11236]</span>: autotest@client-27vm6.lab.whamcloud.com as root: cmd='PATH=$PATH:/sbin:/usr/sbin;echo t <br class="atl-forced-newline" />> /proc/sysrq-trigger'<br/> Jul 20 15:35:53 client-27vm6 kernel: SysRq : Show State</p> </div></div> <p>There was 5 minutes interval related to xinetd<span class="error">[1915]</span> after adding each new group. Finally, autotest terminated the test after running 3600 seconds.</p> <p>After looking into the syslogs of other instances, I found the 5 minutes interval was related to add_group():</p> <div class="panel" style="border-width: 1px;"><div class="panelHeader" style="border-bottom-width: 1px;"><b>Syslog of client-19vm1</b></div><div class="panelContent"> <p>Jul 15 02:21:37 client-19vm1 kernel: Lustre: DEBUG MARKER: == posix test 1: build, install, run posix on ext4 and lustre, then compare ========================== 02:21:36 (1342344096)<br/> Jul 15 02:21:37 client-19vm1 rshd<span class="error">[8144]</span>: pam_unix(rsh:session): session closed for user root<br/> Jul 15 02:21:37 client-19vm1 xinetd<span class="error">[1921]</span>: EXIT: shell status=0 pid=8144 duration=0(sec)<br/> Jul 15 02:21:37 client-19vm1 kernel: loop: loaded (max 8 devices)<br/> Jul 15 02:22:00 client-19vm1 kernel: EXT4-fs (loop1): mounted filesystem with ordered data mode<br/> Jul 15 02:22:03 client-19vm1 xinetd<span class="error">[1921]</span>: START: shell pid=8316 from=10.10.4.220<br/> Jul 15 02:22:03 client-19vm1 rshd<span class="error">[8316]</span>: pam_rhosts_auth(rsh:auth): user root has a `+' user entry<br/> Jul 15 02:22:03 client-19vm1 rshd<span class="error">[8316]</span>: pam_rhosts_auth(rsh:auth): allowed to root@client-19vm1.lab.whamcloud.com as root<br/> Jul 15 02:22:03 client-19vm1 rshd<span class="error">[8316]</span>: pam_unix(rsh:session): session opened for user root by (uid=0)<br/> Jul 15 <font color="red">02:22:04</font> client-19vm1 rshd<span class="error">[8317]</span>: root@client-19vm1.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" USE_OFD= sh -c " error() { set +x; echo Error: \$2: \$1; echo XXRETCODE:\$1; exit \$1; } gid=\$(getent group vsxg0 | cut -d: -f3); if [ "x\$gid" != "x" ]; then [ \$gid -eq 1001 ] || \ error 1 \"inconsistent group ID: new: 1001, old: \$gid\"; else groupadd -g 1001 vsxg0 fi;");echo XXRETCODE:$?'<br/> Jul 15 <font color="red">02:27:04 client-19vm1 groupadd<span class="error">[8354]</span>: new group: name=vsxg0, GID=1001</font><br/> Jul 15 02:27:04 client-19vm1 rshd<span class="error">[8316]</span>: pam_unix(rsh:session): session closed for user root<br/> Jul 15 02:27:04 client-19vm1 xinetd<span class="error">[1921]</span>: EXIT: shell status=0 pid=8316 duration=304(sec)<br/> Jul 15 02:27:04 client-19vm1 xinetd<span class="error">[1921]</span>: START: shell pid=8389 from=10.10.4.220<br/> Jul 15 02:27:04 client-19vm1 rshd<span class="error">[8389]</span>: pam_rhosts_auth(rsh:auth): user root has a `+' user entry<br/> Jul 15 02:27:05 client-19vm1 rshd<span class="error">[8389]</span>: pam_rhosts_auth(rsh:auth): allowed to root@client-19vm1.lab.whamcloud.com as root<br/> Jul 15 02:27:05 client-19vm1 rshd<span class="error">[8389]</span>: pam_unix(rsh:session): session opened for user root by (uid=0)<br/> Jul 15 <font color="red">02:27:05</font> client-19vm1 rshd<span class="error">[8390]</span>: root@client-19vm1.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" USE_OFD= sh -c " error() { set +x; echo Error: \$2: \$1; echo XXRETCODE:\$1; exit \$1; } gid=\$(getent group vsxg1 | cut -d: -f3); if [ "x\$gid" != "x" ]; then [ \$gid -eq 1002 ] || \ error 1 \"inconsistent group ID: new: 1002, old: \$gid\"; else groupadd -g 1002 vsxg1 fi;");echo XXRETCODE:$?'<br/> Jul 15 <font color="red">02:32:04 client-19vm1 groupadd<span class="error">[8427]</span>: new group: name=vsxg1, GID=1002</font></p> <p><~snip~></p> </div></div> <p>Yujian commented "it seems it's related to the network issue of the test cluster. The issue has not occurred since 2012-07-21 (there are more than 30 POSIX test runs after that). " so I think that we can drop the priority of this one so it is not longer tracked as a blocker for 2.3</p> <p>Another instance:<br/> <a href="https://maloo.whamcloud.com/test_sets/a23346b8-e96d-11e1-a508-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/a23346b8-e96d-11e1-a508-52540035b04c</a></p> <div class="panel" style="border-width: 1px;"><div class="panelHeader" style="border-bottom-width: 1px;"><b>Syslog of fat-intel-3vm6</b></div><div class="panelContent"> <p>Aug 18 <font color="red">10:04:37</font> fat-intel-3vm6 rshd<span class="error">[2588]</span>: root@fat-intel-3vm6.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /<br/> usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" USE_OFD= MGSFSTYPE=ldiskfs MDSFSTYPE=ldiskfs OSTFSTYPE=ldiskfs FSTYPE=ldiskfs sh -c " error() { set +x; echo Error: <br/> \$2: \$1; echo XXRETCODE:\$1; exit \$1; } gid=\$(getent group vsxg0 | cut -d: -f3); if [ "x\$gid" != "x" ]; then [ \$gid -eq 1001 ] || \ <br/> error 1 \"inconsistent group ID: new: 1001, old: \$gid\"; else groupadd -g 1001 vsxg0 fi;");echo XXRETCODE:$?'<br/> Aug 18 <font color="red">10:09:37</font> fat-intel-3vm6 groupadd<span class="error">[2629]</span>: new group: name=vsxg0, GID=1001</p> </div></div> <p>The issue is that why it took 5 minutes to add one group on a remote node? Normally, the same operation only took less than 1 second:</p> <div class="panel" style="border-width: 1px;"><div class="panelHeader" style="border-bottom-width: 1px;"><b>Syslog of client-27vm2</b></div><div class="panelContent"> <p>Aug 3 <font color="red">14:02:49</font> client-27vm2 rshd<span class="error">[32413]</span>: root@client-27vm2.lab.whamcloud.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" USE_OFD= MGSFSTYPE=ldiskfs MDSFSTYPE=ldiskfs OSTFSTYPE=ldiskfs FSTYPE=ldiskfs sh -c " error() { set +x; echo Error: \$2: \$1; echo XXRETCODE:\$1; exit \$1; } gid=\$(getent group vsxg0 | cut -d: -f3); if [ "x\$gid" != "x" ]; then [ \$gid -eq 1001 ] || \ error 1 \"inconsistent group ID: new: 1001, old: \$gid\"; else groupadd -g 1001 vsxg0 fi;");echo XXRETCODE:$?'<br/> Aug 3 14:02:49 client-27vm2 kernel: [ $gid -eq 1001 ] || error 1 "inconsistent group ID: new: 1001, old:<br/> Aug 3 <font color="red">14:02:49</font> client-27vm2 groupadd<span class="error">[32450]</span>: new group: name=vsxg0, GID=1001</p> </div></div> <p>In addition, from the historical reports, I found the issue only occurred on RHEL5 clients. There were also passed instances on RHEL5 clients without the above network latency time issue.</p> <p>I can not manually reproduce the above issue on RHEL5 clients on Toro test nodes.</p> <p>To simplify the add_group() and add_user() in posix.cfg, I created the following patches to improve the setup_posix_users() in posix.cfg to use do_rpc_nodes to add groups and users on remote nodes.</p> <p>Patch for b2_3 branch: <a href="http://review.whamcloud.com/3771" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/3771</a><br/> Patch for master branch: <a href="http://review.whamcloud.com/3770" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/3770</a></p> <p>Landed for 2.3 and 2.4</p> Development Rank 1|hzv5sv: Rank (Obsolete) 4474 Severity