https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-12230 Lustre <p>parallel-scale-nfsv3 test_connectathon hangs for Ubuntu 18.04 clients only and hangs 100% of the time for Ubuntu 18.04 clients.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Test #6 - Try to lock the MAXEOF byte. Parent: 6.0 - F_TLOCK [7fffffffffffffff, 1] PASSED. Child: 6.1 - F_TEST [7ffffffffffffffe, 1] PASSED. Child: 6.2 - F_TEST [7ffffffffffffffe, 2] PASSED. Child: 6.3 - F_TEST [7ffffffffffffffe, ENDING] PASSED. Child: 6.4 - F_TEST [7fffffffffffffff, 1] PASSED. Child: 6.5 - F_TEST [7fffffffffffffff, 2] PASSED. Child: 6.6 - F_TEST [7fffffffffffffff, ENDING] PASSED. Child: 6.7 - F_TEST [8000000000000000, ENDING] PASSED. Child: 6.8 - F_TEST [8000000000000000, 1] PASSED. Child: 6.9 - F_TEST [8000000000000000,7fffffffffffffff] PASSED. Child: 6.10 - F_TEST [8000000000000000,8000000000000000] PASSED. Parent: 6.11 - F_ULOCK [7fffffffffffffff, 1] PASSED. Test #7 - Test parent/child mutual exclusion. Parent: 7.0 - F_TLOCK [ ffc, 9] PASSED. runtests: line 48: 1175 Segmentation fault (core dumped) $i $TESTARGS $NFSTESTDIR lock tests failed </pre> </div></div> <p>Looking at the logs for a recent failure, with logs at <a href="https://testing.whamcloud.com/test_sets/4ba5183a-6661-11e9-aeec-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/4ba5183a-6661-11e9-aeec-52540065bddc</a> , the console log for client 2 (vm9) shows the seg fault</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 2567.134886] Lustre: DEBUG MARKER: ./runtests -N 10 -l -f /mnt/lustre/d0.parallel-scale-nfs/d0.connectathon [ 2570.861240] tlocklfs[658]: segfault at 3cfa3a80 ip 00007f68b6289800 sp 00007ffea83150c8 error 4 in libc-2.27.so[7f68b61d8000+1e7000] [ 2571.788153] Lustre: DEBUG MARKER: /usr/sbin/lctl mark parallel-scale-nfsv3 test_connectathon: @@@@@@ FAIL: connectathon failed: 1 [ 2572.001779] Lustre: DEBUG MARKER: parallel-scale-nfsv3 test_connectathon: @@@@@@ FAIL: connectathon failed: 1 </pre> </div></div> <p>The console logs for all other nodes do not contain errors.</p> <p>There are several examples of this failure, but here are just a couple of additional links to logs<br/> <a href="https://testing.whamcloud.com/test_sets/0a36df6c-411a-11e9-8e92-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/0a36df6c-411a-11e9-8e92-52540065bddc</a><br/> <a href="https://testing.whamcloud.com/test_sets/7891c378-4e62-11e9-b98a-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/7891c378-4e62-11e9-b98a-52540065bddc</a><br/> <a href="https://testing.whamcloud.com/test_sets/2d506d0a-560b-11e9-b98a-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/2d506d0a-560b-11e9-b98a-52540065bddc</a></p> Ubuntu clients LU-12230

parallel-scale-nfsv3 test_connectathon times out with segfault

Bug Minor Resolved Fixed Alex Deiter James Nunez ubuntu ubuntu18 ubuntu20 Fri, 26 Apr 2019 19:26:19 +0000 Fri, 14 Jul 2023 20:29:17 +0000 Tue, 29 Nov 2022 14:41:26 +0000 Lustre 2.13.0 Lustre 2.12.1 Lustre 2.14.0 Lustre 2.12.4 Lustre 2.12.5 Lustre 2.12.6 Lustre 2.12.8 Lustre 2.15.0 0 8 <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [ 738.563521] Lustre: 30890:0:(osd_handler.c:1871:osd_trans_start()) lustre-OST0005: credits 3207 &gt; trans_max 2592 </pre> </div></div> <p>meaning transaction is too big for this journal.<br/> for that filesystem 2592 max blocks were calculated as 1/4 of 2592*4=10368 blocks in journal, which in turn 10368*4/1024=40MB - no any real setup has such a tiny journal. probably it makes sense to skip the test if target OST is too small.</p> <p>It looks like Ubuntu and CentOShave the same size servers for parallel-scale-nfsv3. For Ubuntu 18.04, <a href="https://testing.whamcloud.com/test_sets/4ba5183a-6661-11e9-aeec-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/4ba5183a-6661-11e9-aeec-52540065bddc</a> , we see</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1165900 7940 1054764 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 1933276 25788 1786248 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 1933276 25788 1786248 2% /mnt/lustre[OST:1] lustre-OST0002_UUID 1933276 25788 1786248 2% /mnt/lustre[OST:2] lustre-OST0003_UUID 1933276 25784 1786252 2% /mnt/lustre[OST:3] lustre-OST0004_UUID 1933276 25784 1786252 2% /mnt/lustre[OST:4] lustre-OST0005_UUID 1933276 25784 1786252 2% /mnt/lustre[OST:5] lustre-OST0006_UUID 1933276 25784 1786252 2% /mnt/lustre[OST:6] filesystem_summary: 13532932 180500 12503752 2% /mnt/lustre </pre> </div></div> <p>For CentOS, <a href="https://testing.whamcloud.com/test_sets/c0c429a4-6665-11e9-bd0e-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/c0c429a4-6665-11e9-bd0e-52540065bddc</a> , we have</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1165900 80736 981968 8% /mnt/lustre[MDT:0] lustre-OST0000_UUID 1933276 32116 1779920 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 1933276 29940 1782096 2% /mnt/lustre[OST:1] lustre-OST0002_UUID 1933276 28676 1783360 2% /mnt/lustre[OST:2] lustre-OST0003_UUID 1933276 30168 1781868 2% /mnt/lustre[OST:3] lustre-OST0004_UUID 1933276 29948 1782088 2% /mnt/lustre[OST:4] lustre-OST0005_UUID 1933276 34856 1777180 2% /mnt/lustre[OST:5] lustre-OST0006_UUID 1933276 29960 1782076 2% /mnt/lustre[OST:6] filesystem_summary: 13532932 215664 12468588 2% /mnt/lustre </pre> </div></div> <blockquote> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Lustre: 30890:0:(osd_handler.c:1871:osd_trans_start()) lustre-OST0005: credits 3207 &gt; trans_max 2592 </pre> </div></div> <p>meaning transaction is too big for this journal.</p></blockquote> <p>I also see this error frequently in local-node testing with striped directories. However, I <b>don't</b> think that this would actually cause the test to segfault in glibc. Instead, I suspect that this test is simply triggering some flock mode that we do not support and the error return code is not being handled gracefully for some reason.</p> <p>It would probably be useful to run this test with full debug, so that we can get a log of what the NFS server is requesting the Lustre client to do. Alternately (or also), manually running the test binary under gdb to catch the segfault would also be useful, especially in conjunction with the Lustre client debug logs to get the history of the just-completed request.</p> <p>James Nunez (jnunez@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/44134" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44134</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12230" title="parallel-scale-nfsv3 test_connectathon times out with segfault" class="issue-link" data-issue-key="LU-12230"><del>LU-12230</del></a> tests: test connectathon full debug<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: b42d3ff67c7a5d64ce4b02e34147309f9a8599dc</p> <p>Having a similar issue with <a href="https://testing.whamcloud.com/test_sets/daa7a9a4-d213-42fb-bb56-edbceff2c17e" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/daa7a9a4-d213-42fb-bb56-edbceff2c17e</a></p> <p>2.12.8, Ubuntu 18.04, parallel-scale-nfsv3, test_connectathon</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> Test #6 - Try to lock the MAXEOF <span class="code-object">byte</span>. Parent: 6.0 - F_TLOCK [7fffffffffffffff, 1] PASSED. Child: 6.1 - F_TEST [7ffffffffffffffe, 1] PASSED. Child: 6.2 - F_TEST [7ffffffffffffffe, 2] PASSED. Child: 6.3 - F_TEST [7ffffffffffffffe, ENDING] PASSED. Child: 6.4 - F_TEST [7fffffffffffffff, 1] PASSED. Child: 6.5 - F_TEST [7fffffffffffffff, 2] PASSED. Child: 6.6 - F_TEST [7fffffffffffffff, ENDING] PASSED. Child: 6.7 - F_TEST [8000000000000000, ENDING] PASSED. Child: 6.8 - F_TEST [8000000000000000, 1] PASSED. Child: 6.9 - F_TEST [8000000000000000,7fffffffffffffff] PASSED. Child: 6.10 - F_TEST [8000000000000000,8000000000000000] PASSED. Parent: 6.11 - F_ULOCK [7fffffffffffffff, 1] PASSED. Test #7 - Test parent/child mutual exclusion. Parent: 7.0 - F_TLOCK [ ffc, 9] PASSED. runtests: line 48: 15529 Segmentation fault (core dumped) $i $TESTARGS $NFSTESTDIR lock tests failed parallel-scale-nfsv3 test_connectathon: @@@@@@ FAIL: connectathon failed: 1 </pre> </div></div> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [ 1351.318605] Lustre: DEBUG MARKER: sh ./runtests -N 10 -l -f /mnt/lustre/d0.parallel-scale-nfs/d0.connectathon [ 1351.658038] tlocklfs[15529]: segfault at 36109a80 ip 00007f3a9645b5e0 sp 00007ffc088aa118 error 4 in libc-2.27.so[7f3a963aa000+1e7000] [ 1352.141971] Lustre: DEBUG MARKER: /usr/sbin/lctl mark parallel-scale-nfsv3 test_connectathon: @@@@@@ FAIL: connectathon failed: 1 [ 1352.360581] Lustre: DEBUG MARKER: parallel-scale-nfsv3 test_connectathon: @@@@@@ FAIL: connectathon failed: 1 </pre> </div></div> <p>Does <a href="https://jira.whamcloud.com/browse/LU-15143" title="journal_dirty_metadata failed: handle type 0 started at line 1926, credits 9/0, errcode -28" class="issue-link" data-issue-key="LU-15143"><del>LU-15143</del></a> fix this?</p> <p>"James Simmons &lt;jsimmons@infradead.org&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/46074" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/46074</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12230" title="parallel-scale-nfsv3 test_connectathon times out with segfault" class="issue-link" data-issue-key="LU-12230"><del>LU-12230</del></a> tests: validate connectathon<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 2aa99b4c4042ca088fee98bbedfb44acd768c64c</p> <p>I'm seeing in maloo:</p> <p>trevis-53vm4: Redirecting to /bin/systemctl restart nfs.service</p> <p>trevis-53vm4: Failed to restart nfs.service: Unit nfs.service not found. parallel-scale-nfsv3 : @@@@@@ FAIL: setup nfs failed! Trace dump: = /usr/lib64/lustre/tests/test-framework.sh:5919:error() = /usr/lib64/lustre/tests/parallel-scale-nfs.sh:66:main()</p> <p>Looks like a configuration issue.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=simmonsja" class="user-hover" rel="simmonsja">simmonsja</a> - I don't see the failure that you posted, but I do see a similar problem but the test recovers. See the suite_log at <a href="https://testing.whamcloud.com/test_sets/cd6bfcf2-b2ad-4de0-ae5a-8a62cea37819" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/cd6bfcf2-b2ad-4de0-ae5a-8a62cea37819</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>CMD: trevis-76vm2 { [[ -e /etc/SuSE-release ]] &amp;&amp; service nfsserver restart; } || service nfs restart || service nfs-server restart trevis-76vm2: Redirecting to /bin/systemctl restart nfs.service trevis-76vm2: Failed to restart nfs.service: Unit nfs.service not found. trevis-76vm2: Redirecting to /bin/systemctl restart nfs-server.service CMD: trevis-34vm5.trevis.whamcloud.com,trevis-34vm8 chkconfig --list rpcidmapd 2&gt;/dev/null | grep -q rpcidmapd &amp;&amp; service rpcidmapd restart || true Mounting NFS clients (version 3)... </pre> </div></div> <p>Please post links to failed test sessions if you continue to see this issue.</p> <p>The issue can be reproduced on Ubuntu distro:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> # sh ./runtests -N 2 -l -f /mnt/nfs # Test #6 - Try to lock the MAXEOF <span class="code-object">byte</span>. Parent: 6.0 - F_TLOCK [7fffffffffffffff, 1] PASSED. Child: 6.1 - F_TEST [7ffffffffffffffe, 1] PASSED. Child: 6.2 - F_TEST [7ffffffffffffffe, 2] PASSED. Child: 6.3 - F_TEST [7ffffffffffffffe, ENDING] PASSED. Child: 6.4 - F_TEST [7fffffffffffffff, 1] PASSED. Child: 6.5 - F_TEST [7fffffffffffffff, 2] PASSED. Child: 6.6 - F_TEST [7fffffffffffffff, ENDING] PASSED. Child: 6.7 - F_TEST [8000000000000000, ENDING] PASSED. Child: 6.8 - F_TEST [8000000000000000, 1] PASSED. Child: 6.9 - F_TEST [8000000000000000,7fffffffffffffff] PASSED. Child: 6.10 - F_TEST [8000000000000000,8000000000000000] PASSED. Parent: 6.11 - F_ULOCK [7fffffffffffffff, 1] PASSED. Test #7 - Test parent/child mutual exclusion. Parent: 7.0 - F_TLOCK [ ffc, 9] PASSED. Segmentation fault (core dumped) lock tests failed </pre> </div></div> <p>Packaging issue - will be fixed in DCO-9179</p> <p>Proposed patch has been successfully tested on Ubuntu 22.04:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> # dpkg -i connectathon_1.20-1_amd64.deb Selecting previously unselected <span class="code-keyword">package</span> connectathon. (Reading database ... 98334 files and directories currently installed.) Preparing to unpack connectathon_1.20-1_amd64.deb ... Unpacking connectathon (1.20-1) ... Setting up connectathon (1.20-1) ... # mount -t nfs 10.128.0.32:/mnt/exacloud on /mnt/exacloud type nfs (rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.128.0.32,mountvers=3,mountport=56089,mountproto=udp,local_lock=none,addr=10.128.0.32) # sh ./runtests -N 2 -l -f /mnt/exacloud ... Parent: 6.11 - F_ULOCK [7fffffffffffffff, 1] PASSED. Test #7 - Test parent/child mutual exclusion. Parent: 7.0 - F_TLOCK [ ffc, 9] PASSED. Parent: Wrote <span class="code-quote">'aaaa eh'</span> to testfile [ 4092, 7 ]. Parent: Now free child to run, should block on lock. Parent: Check data in file to insure child blocked. Parent: Read <span class="code-quote">'aaaa eh'</span> from testfile [ 4092, 7 ]. Parent: 7.1 - COMPARE [ ffc, 7] PASSED. Parent: Now unlock region so child will unblock. Parent: 7.2 - F_ULOCK [ ffc, 9] PASSED. Child: 7.3 - F_LOCK [ ffc, 9] PASSED. Child: Write child's version of the data and release lock. Parent: Now <span class="code-keyword">try</span> to regain lock, parent should block. Child: Wrote <span class="code-quote">'bebebebeb'</span> to testfile [ 4092, 9 ]. Child: 7.4 - F_ULOCK [ ffc, 9] PASSED. Parent: 7.5 - F_LOCK [ ffc, 9] PASSED. Parent: Check data in file to insure child unblocked. Parent: Read <span class="code-quote">'bebebebeb'</span> from testfile [ 4092, 9 ]. Parent: 7.6 - COMPARE [ ffc, 9] PASSED. Parent: 7.7 - F_ULOCK [ ffc, 9] PASSED. Test #10 - Make sure a locked region is split properly. </pre> </div></div> <p>Thank you for fixing this</p> <p>Proposed patch has been successfully tested on Ubuntu 18.04:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> $ sudo dpkg -i connectathon_1.20-1_amd64.deb Selecting previously unselected <span class="code-keyword">package</span> connectathon. (Reading database ... 217066 files and directories currently installed.) Preparing to unpack connectathon_1.20-1_amd64.deb ... Unpacking connectathon (1.20-1) ... Setting up connectathon (1.20-1) ... $ sh /opt/connectathon/runtests -a -f /mnt/test Test #7 - Test parent/child mutual exclusion. Parent: 7.0 - F_TLOCK [ ffc, 9] PASSED. Parent: Wrote <span class="code-quote">'aaaa eh'</span> to testfile [ 4092, 7 ]. Parent: Now free child to run, should block on lock. Parent: Check data in file to insure child blocked. Parent: Read <span class="code-quote">'aaaa eh'</span> from testfile [ 4092, 7 ]. Parent: 7.1 - COMPARE [ ffc, 7] PASSED. Parent: Now unlock region so child will unblock. Child: 7.3 - F_LOCK [ ffc, 9] PASSED. Child: Write child's version of the data and release lock. Parent: 7.2 - F_ULOCK [ ffc, 9] PASSED. Parent: Now <span class="code-keyword">try</span> to regain lock, parent should block. Child: Wrote <span class="code-quote">'bebebebeb'</span> to testfile [ 4092, 9 ]. Child: 7.4 - F_ULOCK [ ffc, 9] PASSED. Parent: 7.5 - F_LOCK [ ffc, 9] PASSED. Parent: Check data in file to insure child unblocked. Parent: Read <span class="code-quote">'bebebebeb'</span> from testfile [ 4092, 9 ]. Parent: 7.6 - COMPARE [ ffc, 9] PASSED. Parent: 7.7 - F_ULOCK [ ffc, 9] PASSED. ** PARENT pass 1 results: 49/49 pass, 1/1 warn, 0/0 fail (pass/total). ** CHILD pass 1 results: 64/64 pass, 0/0 warn, 0/0 fail (pass/total). Congratulations, you passed the locking tests! All tests completed </pre> </div></div> <p>Alex did git commit 1a8fe55b17ac2bc2195aaba446467ccdac67b564 resolve this bug? if so we can close this.</p> <p>Hello <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=simmonsja" class="user-hover" rel="simmonsja">simmonsja</a>,</p> <p>??Alex did git commit 1a8fe55b17ac2bc2195aaba446467ccdac67b564 resolve this bug? if so we can close this.<br/> ??</p> <p>I think yes.</p> <p>Thank you!</p> <p>James, this ticket was already marked resolved on Nov of last year. Did you mean to update a different ticket?</p> <p>I was seeing it as still open. </p> Duplicate Related LU-11948 LU-14294 LU-15781 Development Rank 1|i00fgn: Rank (Obsolete) 9223372036854775807 Severity