https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1940 Lustre <p>This issue was created by maloo for Oleg Drokin &lt;green@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://maloo.whamcloud.com/test_sets/d98c0f7a-fe88-11e1-a707-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/d98c0f7a-fe88-11e1-a707-52540035b04c</a>.</p> <p>The sub-test test_118c failed with the following error:</p> <blockquote> <p>Multiop fsync failed, rc=30</p></blockquote> <p>Info required for matching: sanity 118c</p> LU-1940

Test failure on test suite sanity, subtest test_118c

Bug Blocker Resolved Fixed Hongchao Zhang Maloo MB Fri, 14 Sep 2012 13:26:30 +0000 Fri, 15 Feb 2013 13:57:35 +0000 Fri, 15 Feb 2013 13:57:35 +0000 Lustre 2.4.0 Lustre 2.4.0 0 10 <p><a href="https://maloo.whamcloud.com/test_sets/807ed662-0273-11e2-ab94-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/807ed662-0273-11e2-ab94-52540035b04c</a></p> <p><a href="https://maloo.whamcloud.com/test_sets/ffd3d9f0-0e0b-11e2-bf2b-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/ffd3d9f0-0e0b-11e2-bf2b-52540035b04c</a></p> <p>Hit this failure three times last night on three different patches.</p> <p><a href="https://maloo.whamcloud.com/test_sets/8cf491dc-0e0a-11e2-91a3-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/8cf491dc-0e0a-11e2-91a3-52540035b04c</a></p> <p><a href="https://maloo.whamcloud.com/test_sets/b5a029e6-0e07-11e2-bf2b-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/b5a029e6-0e07-11e2-bf2b-52540035b04c</a></p> <p><a href="https://maloo.whamcloud.com/test_sets/8cf491dc-0e0a-11e2-91a3-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/8cf491dc-0e0a-11e2-91a3-52540035b04c</a></p> <p><a href="https://maloo.whamcloud.com/sub_tests/594d6ba4-17f8-11e2-a41f-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/sub_tests/594d6ba4-17f8-11e2-a41f-52540035b04c</a></p> <p><a href="https://maloo.whamcloud.com/test_sets/38aa57a4-1ea1-11e2-8b41-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/38aa57a4-1ea1-11e2-8b41-52540035b04c</a></p> <p>Note 30 is EROFS: </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>22:40:54:Lustre: DEBUG MARKER: == sanity test 118c: Fsync blocks on EROFS until dirty pages are flushed ============ 22:40:50 (1351143650) 22:41:05:LustreError: 11-0: an error occurred while communicating with 10.10.4.161@tcp. The ost_write operation failed with -30 22:41:05:LustreError: 2990:0:(osc_request.c:1689:osc_brw_redo_request()) @@@ redo for recoverable error -30 req@ffff880079ad4c00 x1416774108278040/t0(0) o4-&gt;lustre-OST0003-osc-ffff88007a8f6000@10.10.4.161@tcp:6/4 lens 488/192 e 0 to 0 dl 1351143702 ref 2 fl Interpret:R/0/0 rc -30/-30 22:41:05:LustreError: 11-0: an error occurred while communicating with 10.10.4.161@tcp. The ost_write operation failed with -30 22:41:05:LustreError: 11-0: an error occurred while communicating with 10.10.4.161@tcp. The ost_write operation failed with -30 22:41:05:LustreError: 11-0: an error occurred while communicating with 10.10.4.161@tcp. The ost_write operation failed with -30 22:41:06:LustreError: 11-0: an error occurred while communicating with 10.10.4.161@tcp. The ost_write operation failed with -30 22:41:06:LustreError: 2990:0:(osc_request.c:1931:brw_interpret()) lustre-OST0003-osc-ffff88007a8f6000: too many resent retries for object: 1118:0, rc = -30. 22:41:17:Lustre: DEBUG MARKER: /usr/sbin/lctl mark sanity test_118c: @@@@@@ FAIL: Multiop fsync failed, rc=30 22:41:17:Lustre: DEBUG MARKER: sanity test_118c: @@@@@@ FAIL: Multiop fsync failed, rc=30 </pre> </div></div> <p>We seem to be giving up (too many retries) before the system has become writable. EROFS is trying to write to a read only filesystem. It seems this test is to make sure we properly block in this condition. More investigation is needed. </p> <p>status update:</p> <p>it is still under investigation.</p> <p>this bug is caused by the resend limit in OSC (the default value is 10) for recoverable error(EIO, EROFS, ENOMEM,<br/> EAGAIN, EINPROGRESS), we can disable this check for this test to fix it.</p> <p>the patch is tracked at <a href="http://review.whamcloud.com/#change,4622" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,4622</a></p> <p>Landed for 2.4</p> <p><a href="https://maloo.whamcloud.com/test_sets/df9e51ca-3899-11e2-8c55-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/df9e51ca-3899-11e2-8c55-52540035b04c</a></p> <p>the new occurrence is still the resend count, normally there is 1s interval between the fail_loc=OBD_FAIL_OST_EROFS<br/> and fail_loc=0, but somehow the actual time is much longer than it.</p> <p>the extra patch is tracked at <a href="http://review.whamcloud.com/#change,4694" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,4694</a></p> <p>From December 30. 1 error out of the last 100 runs. </p> <p><a href="https://maloo.whamcloud.com/test_sets/90b395d0-5319-11e2-908e-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/90b395d0-5319-11e2-908e-52540035b04c</a></p> <p>Patch 4694 was landed for 2.4.0</p> Duplicate LU-2604 Development Rank 1|hzv44n: Rank (Obsolete) 4199 Severity