https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-15722 Lustre <p>IO write gets stuck on some test sanityn cases for 64K PAGE_SIZE, such as sanityn 16a, 16b, 71a etc.</p> <p> </p> <p>Kernel logs:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [ 1308.972770] Lustre: DEBUG MARKER: == sanityn test 71a: correct file map just after write operation is finished ========================================================== 01:56:46 (1638755806) [ 1309.958643] LustreError: 5126:0:(ofd_io.c:1401:ofd_commitrw_write()) lustre-OST0000: restart IO write too many times: 10000 [ 1310.704658] LustreError: 5126:0:(ofd_io.c:1401:ofd_commitrw_write()) lustre-OST0000: restart IO write too many times: 20000 [ 1312.183793] LustreError: 5126:0:(ofd_io.c:1401:ofd_commitrw_write()) lustre-OST0000: restart IO write too many times: 40000 [ 1312.186168] LustreError: 5126:0:(ofd_io.c:1401:ofd_commitrw_write()) Skipped 1 previous similar message [ 1314.448308] LustreError: 5126:0:(ofd_io.c:1401:ofd_commitrw_write()) lustre-OST0000: restart IO write too many times: 70000 [ 1314.538440] LustreError: 5126:0:(ofd_io.c:1401:ofd_commitrw_write()) Skipped 2 previous similar messages [ 1319.063438] LustreError: 5126:0:(ofd_io.c:1401:ofd_commitrw_write()) lustre-OST0000: restart IO write too many times: 120000 [ 1319.065847] LustreError: 5126:0:(ofd_io.c:1401:ofd_commitrw_write()) Skipped 4 previous similar messages [ 1327.150991] LustreError: 5126:0:(ofd_io.c:1401:ofd_commitrw_write()) lustre-OST0000: restart IO write too many times: 210000 [ 1327.153390] LustreError: 5126:0:(ofd_io.c:1401:ofd_commitrw_write()) Skipped 8 previous similar messages [ 1327.359397] Lustre: lustre-OST0001-osc-ffffcaf2f9e2e000: disconnect after 20s idle [ 1332.399266] Lustre: lustre-OST0001-osc-ffffcaf2fb2cf000: disconnect after 23s idle [ 1344.045735] LustreError: 5126:0:(ofd_io.c:1401:ofd_commitrw_write()) lustre-OST0000: restart IO write too many times: 400000 [ 1344.193155] LustreError: 5126:0:(ofd_io.c:1401:ofd_commitrw_write()) Skipped 18 previous similar messages [ 1349.278936] Lustre: ll_ost05_002: service thread pid 5101 was inactive <span class="code-keyword">for</span> 40.209 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace <span class="code-keyword">for</span> debugging purposes: [ 1349.278963] Pid: 5126, comm: ll_ost_io00_002 4.18.0-305.7.1.el8_lustre.aarch64 #1 SMP Mon Jul 19 08:24:26 UTC 2021 [ 1349.282879] Lustre: Skipped 1 previous similar message [ 1349.285101] Call Trace: [ 1349.286688] [&lt;0&gt;] __switch_to+0xbc/0x108 [ 1349.287534] [&lt;0&gt;] osd_trans_stop+0x66c/0xc58 [osd_ldiskfs] [ 1349.288692] [&lt;0&gt;] ofd_trans_stop+0x48/0x90 [ofd] [ 1349.289746] [&lt;0&gt;] ofd_commitrw_write+0x9c4/0x1e68 [ofd] [ 1349.290825] [&lt;0&gt;] ofd_commitrw+0x454/0xa88 [ofd] [ 1349.291909] [&lt;0&gt;] tgt_brw_write+0x1654/0x2db8 [ptlrpc] [ 1349.293054] [&lt;0&gt;] tgt_handle_request0+0xd0/0x978 [ptlrpc] [ 1349.294228] [&lt;0&gt;] tgt_request_handle+0x7c0/0x1a38 [ptlrpc] [ 1349.295423] [&lt;0&gt;] ptlrpc_server_handle_request+0x3bc/0x11e8 [ptlrpc] [ 1349.296802] [&lt;0&gt;] ptlrpc_main+0xd28/0x15f0 [ptlrpc] [ 1349.297808] [&lt;0&gt;] kthread+0x130/0x138 [ 1349.298560] [&lt;0&gt;] ret_from_fork+0x10/0x18 [ 1349.299465] Pid: 5101, comm: ll_ost05_002 4.18.0-305.7.1.el8_lustre.aarch64 #1 SMP Mon Jul 19 08:24:26 UTC 2021 [ 1349.301552] Call Trace: [ 1349.302051] [&lt;0&gt;] __switch_to+0xbc/0x108 [ 1349.302934] [&lt;0&gt;] ldlm_completion_ast+0x778/0xdf8 [ptlrpc] [ 1349.304076] [&lt;0&gt;] ldlm_cli_enqueue_local+0x204/0xb68 [ptlrpc] [ 1349.305282] [&lt;0&gt;] tgt_extent_lock+0x108/0x2d0 [ptlrpc] [ 1349.306308] [&lt;0&gt;] ofd_lock_unlock_region+0x74/0x1e8 [ofd] [ 1349.307370] [&lt;0&gt;] ofd_get_info_hdl+0xd30/0x1378 [ofd] [ 1349.308439] [&lt;0&gt;] tgt_handle_request0+0xd0/0x978 [ptlrpc] [ 1349.309637] [&lt;0&gt;] tgt_request_handle+0x7c0/0x1a38 [ptlrpc] [ 1349.310786] [&lt;0&gt;] ptlrpc_server_handle_request+0x3bc/0x11e8 [ptlrpc] [ 1349.312097] [&lt;0&gt;] ptlrpc_main+0xd28/0x15f0 [ptlrpc] [ 1349.313068] [&lt;0&gt;] kthread+0x130/0x138 [ 1349.313788] [&lt;0&gt;] ret_from_fork+0x10/0x18 [ 1376.839778] LustreError: 5126:0:(ofd_io.c:1401:ofd_commitrw_write()) lustre-OST0000: restart IO write too many times: 770000 [ 1376.842204] LustreError: 5126:0:(ofd_io.c:1401:ofd_commitrw_write()) Skipped 36 previous similar messages </pre> </div></div> Arm CentOS 8 server and client all-in-one test environment. <br/> <br/> LU-15722

IO write gets stuck on some sanityn test cases for 64K PAGE_SIZE

Bug Major Resolved Fixed Xinliang Liu Xinliang Liu arm arm-server Wed, 6 Apr 2022 07:32:07 +0000 Fri, 12 Jan 2024 04:29:40 +0000 Sat, 19 Aug 2023 12:07:22 +0000 Lustre 2.16.0 0 2 <p>Look into the source code and address that it goes into an infinite loop:</p> <p> </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [606895.151765] LustreError: 334886:0:(ofd_io.c:1389:ofd_commitrw_write()) lustre-OST0000: restart IO write too many times: 10000 [606895.207345] LustreError: 334886:0:(ofd_io.c:1389:ofd_commitrw_write()) Skipped 8 previous similar messages ofd_commitrw_write()-&gt; osd_write_commit()-&gt;osd_ldiskfs_map_inode_pages()-&gt;ldiskfs_map_blocks()-&gt;ofd_commitrw_write() </pre> </div></div> <p>The related loop source code part is:</p> <p> </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> 1219 ofd_commitrw_write(<span class="code-keyword">const</span> struct lu_env *env, struct obd_export *exp, .... 1288 retry: ... 1342 1343         <span class="code-keyword">if</span> (likely(!fake_write)) { ... 1346                 rc = osd_write_commit(env, o, lnb, niocount, th, oa-&gt;o_size); 1347                 <span class="code-keyword">if</span> (rc) { 1348                         restart = th-&gt;th_restart_tran; 1349                         GOTO(out_unlock, rc); 1350                 } 1351         } 1352 ..... 1384         <span class="code-keyword">if</span> (restart) { 1385                 retries++; 1386                 restart = 0; 1387                 <span class="code-keyword">if</span> (retries % 10000 == 0) 1388                         CERROR(<span class="code-quote">"%s: restart IO write too many times: %d\n"</span>, 1389                                 ofd_name(ofd), retries); 1390                 CDEBUG(D_INODE, <span class="code-quote">"retry transaction, retries:%d\n"</span>, 1391                        retries); 1392                 <span class="code-keyword">goto</span> retry; 1393         } 1079 <span class="code-keyword">static</span> <span class="code-object">int</span> osd_ldiskfs_map_inode_pages(struct inode *inode, ... 1143 cont_map: ... 1150                 <span class="code-keyword">if</span> (handle &amp;&amp; check_credits) { ... 1161                         <span class="code-keyword">if</span> (oh-&gt;oh_declared_ext &lt;= 0) { 1162                                 rc = osd_ldiskfs_map_write(inode, 1163                                         iobuf, osd, start_blocks, 1164                                         count, &amp;disk_size, user_size); 1165                                 <span class="code-keyword">if</span> (rc) 1166                                         GOTO(cleanup, rc); 1167                                 thandle-&gt;th_restart_tran = 1; 1168                                 GOTO(cleanup, rc = -EAGAIN); ... 1175                 } 1176                 rc = ldiskfs_map_blocks(handle, inode, &amp;map, create); ... 1224                 <span class="code-keyword">if</span> (rc == 0 &amp;&amp; total &lt; blen) { ... 1234                         <span class="code-keyword">goto</span> cont_map; 1235                 } </pre> </div></div> <p>And the cause is in function ldiskfs_map_blocks(), that for PAGE_SIZE &gt; block_size if a partial page allocation occurs first it will never get a full page(e.g. 64K PAGE_SIZE 16 blocks) allocated later for the page:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">  498 <span class="code-object">int</span> ldiskfs_map_blocks(handle_t *handle, struct inode *inode,  499                     struct ext4_map_blocks *map, <span class="code-object">int</span> flags)  500 {  ...  525         <span class="code-comment">/* Lookup extent status tree firstly */</span>  526         <span class="code-keyword">if</span> (ext4_es_lookup_extent(inode, map-&gt;m_lblk, &amp;es)) {  527                 <span class="code-keyword">if</span> (ext4_es_is_written(&amp;es) || ext4_es_is_unwritten(&amp;es)) {  528                         map-&gt;m_pblk = ext4_es_pblock(&amp;es) +  529                                         map-&gt;m_lblk - es.es_lblk;  530                         map-&gt;m_flags |= ext4_es_is_written(&amp;es) ?  531                                         EXT4_MAP_MAPPED : EXT4_MAP_UNWRITTEN;  532                         retval = es.es_len - (map-&gt;m_lblk - es.es_lblk); ...  550                 <span class="code-keyword">goto</span> found;  551         } 590 found: ...  608         <span class="code-keyword">if</span> (retval &gt; 0 &amp;&amp; map-&gt;m_flags &amp; EXT4_MAP_MAPPED)  614                 <span class="code-keyword">if</span> (!(flags &amp; EXT4_GET_BLOCKS_CONVERT_UNWRITTEN))  615                         <span class="code-keyword">return</span> retval; </pre> </div></div> <p> </p> <p>Below Lustre log and kernel trace logs is an example for page 1 partial block allocation first(a.k.a 14 of 16 blocks) and then ldiskfs_map_blocks for this page always keep 14 blocks allocated no way up to full page 16 blocks allocation. Because below log keep printing and no change any more.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> (osd_io.c:1107:osd_ldiskfs_map_inode_pages()) inode 109: map 2 pages from 1 (osd_io.c:1148:osd_ldiskfs_map_inode_pages()) pages[0, 1], clen=2: blen=32 (osd_io.c:507:osd_do_bio()) <span class="code-object">Process</span> entered (osd_io.c:571:osd_do_bio()) Write page 1 blocks_left_page=14: i=0, page_offset=0, nblocks=14 (osd_io.c:430:osd_bio_init()) <span class="code-object">Process</span> entered  6)               |  osd_ldiskfs_map_inode_pages [osd_ldiskfs]() {   6)   0.180 us    |    ldiskfs_inode_attach_jinode [ldiskfs]();   6)               |    ldiskfs_map_blocks [ldiskfs]() {   6)               |      ldiskfs_es_lookup_extent [ldiskfs]() {   6)               |        <span class="code-comment">/* ldiskfs_es_lookup_extent_enter: dev 253,1 ino 109 lblk 16 */</span>   6)               |        <span class="code-comment">/* ldiskfs_es_lookup_extent_exit: dev 253,1 ino 109 found 1 [0/30) 61112 W0x10 */</span>   6)   1.820 us    |      }   6)   0.440 us    |      ldiskfs_inode_block_valid [ldiskfs]();   6)   6.760 us    |    }   6) + 51.380 us   |    osd_ldiskfs_map_write.constprop.9 [osd_ldiskfs]();   6) + 75.360 us   |  } </pre> </div></div> <p> </p> <p>"xinliang &lt;xinliang.liu@linaro.org&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/47004" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/47004</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15722" title="IO write gets stuck on some sanityn test cases for 64K PAGE_SIZE" class="issue-link" data-issue-key="LU-15722"><del>LU-15722</del></a> osd-ldiskfs: fix IO write gets stuck for 64K PAGE_SIZE<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 40d76c4524e97862eb57b3821852b2383b2f1022</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/47004/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/47004/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15722" title="IO write gets stuck on some sanityn test cases for 64K PAGE_SIZE" class="issue-link" data-issue-key="LU-15722"><del>LU-15722</del></a> osd-ldiskfs: fix IO write gets stuck for 64K PAGE_SIZE<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 176ea3a4599ede8b1a0c91506dcd34bc162f2959</p> <p>"xinliang &lt;xinliang.liu@linaro.org&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/47563" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/47563</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15722" title="IO write gets stuck on some sanityn test cases for 64K PAGE_SIZE" class="issue-link" data-issue-key="LU-15722"><del>LU-15722</del></a> osd-ldiskfs: fix write stuck for 64K PAGE_SIZE<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 8fc89eaaffe26f407a635fef79131bb0cdb23470</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/47563/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/47563/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15722" title="IO write gets stuck on some sanityn test cases for 64K PAGE_SIZE" class="issue-link" data-issue-key="LU-15722"><del>LU-15722</del></a> osd-ldiskfs: fix write stuck for 64K PAGE_SIZE<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 48ccf2db8495344e1d0c39f8bed0969279e29db4</p> <p>Landed for 2.16</p> <p>"xinliang &lt;xinliang.liu@linaro.org&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/53655" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/53655</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15722" title="IO write gets stuck on some sanityn test cases for 64K PAGE_SIZE" class="issue-link" data-issue-key="LU-15722"><del>LU-15722</del></a> osd-ldiskfs: fix write stuck for 64K PAGE_SIZE<br/> Project: fs/lustre-release<br/> Branch: b2_15<br/> Current Patch Set: 1<br/> Commit: 0cf9f35f772b7aa3d967f82fdd426621648c933c</p> Related LU-10300 Development Rank 1|i02ml3: Rank (Obsolete) 9223372036854775807 Severity