https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-10239 Lustre <p>Our clients have max_rpcs_in_flight set to 16. Some of our clients hit this bug at times, which crashes these nodes:</p> <p> </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> 2017-10-27T20:22:40-05:00 node0748 kernel: [14998.118253] LustreError: 1708:0:(osc_cache.c:1931:try_to_add_extent_for_io()) extent ffff882e6c434000@{[60858 -&gt; 64953/64953], [1|0|+|lockdone|wSu|ffff882dc5cce4d0], [0|4096|+|-| (<span class="code-keyword">null</span>)|4096| (<span class="code-keyword">null</span>)]} The first extent to be fit in a RPC contains 17 chunks, which is over the limit 16. 2017-10-27T20:22:40-05:00 node0748 kernel: [14998.154188] LustreError: 1708:0:(osc_cache.c:1226:osc_extent_tree_dump0()) Dump object ffff882dc5cce4d0 extents at try_to_add_extent_for_io:1931, mppr: 4096. 2017-10-27T20:22:40-05:00 node0748 kernel: [14998.181833] LustreError: 1708:0:(osc_cache.c:1239:osc_extent_tree_dump0()) extent ffff882e6c434000@{[60858 -&gt; 64953/64953], [1|0|+|lockdone|wSu|ffff882dc5cce4d0], [0|4096|+|-| (<span class="code-keyword">null</span>)|4096| (<span class="code-keyword">null</span>)]}urgent 1. 2017-10-27T20:22:40-05:00 node0748 kernel: [14998.216119] LustreError: 1708:0:(osc_cache.c:1931:try_to_add_extent_for_io()) ASSERTION( data-&gt;erd_page_count != 0 || chunk_count &lt;= data-&gt;erd_max_chunks ) failed: 2017-10-27T20:22:40-05:00 node0748 kernel: [14998.244827] LustreError: 1708:0:(osc_cache.c:1931:try_to_add_extent_for_io()) LBUG 2017-10-27T20:22:40-05:00 node0748 kernel: [14998.259546] Pid: 1708, comm: ptlrpcd_00_86 2017-10-27T20:22:40-05:00 node0748 kernel: [14998.270365] 2017-10-27T20:22:40-05:00 node0748 kernel: [14998.270365] Call Trace: 2017-10-27T20:22:40-05:00 node0748 kernel: [14998.287055] [&lt;ffffffffa07537f3&gt;] libcfs_debug_dumpstack+0x53/0x80 [libcfs] 2017-10-27T20:22:40-05:00 node0748 kernel: [14998.301366] [&lt;ffffffffa0753861&gt;] lbug_with_loc+0x41/0xb0 [libcfs] 2017-10-27T20:22:40-05:00 node0748 kernel: [14998.314695] [&lt;ffffffffa0c69498&gt;] try_to_add_extent_for_io.isra.24+0xf58/0x12e0 [osc] 2017-10-27T20:22:40-05:00 node0748 kernel: [14998.328249] [&lt;ffffffffa0c6b9dd&gt;] osc_io_unplug0+0x3fd/0x1950 [osc] 2017-10-27T20:22:40-05:00 node0748 kernel: [14998.339730] [&lt;ffffffff810d2372&gt;] ? load_balance+0x192/0x990 2017-10-27T20:22:40-05:00 node0748 kernel: [14998.350013] [&lt;ffffffff810ce46c&gt;] ? dequeue_entity+0x11c/0x5d0 2017-10-27T20:22:40-05:00 node0748 kernel: [14998.360444] [&lt;ffffffffa0c6db30&gt;] osc_io_unplug+0x10/0x20 [osc] 2017-10-27T20:22:40-05:00 node0748 kernel: [14998.370874] [&lt;ffffffffa0c49441&gt;] brw_queue_work+0x31/0xd0 [osc] 2017-10-27T20:22:40-05:00 node0748 kernel: [14998.381510] [&lt;ffffffffa0a5e3d7&gt;] work_interpreter+0x37/0xf0 [ptlrpc] 2017-10-27T20:22:40-05:00 node0748 kernel: [14998.392538] [&lt;ffffffffa0a5b0b5&gt;] ptlrpc_check_set.part.23+0x425/0x1dd0 [ptlrpc] 2017-10-27T20:22:40-05:00 node0748 kernel: [14998.405132] [&lt;ffffffffa0a5cabb&gt;] ptlrpc_check_set+0x5b/0xe0 [ptlrpc] 2017-10-27T20:22:40-05:00 node0748 kernel: [14998.416411] [&lt;ffffffffa0a88a3b&gt;] ptlrpcd_check+0x4db/0x5d0 [ptlrpc] 2017-10-27T20:22:40-05:00 node0748 kernel: [14998.427367] [&lt;ffffffffa0a88d57&gt;] ptlrpcd+0x227/0x560 [ptlrpc] 2017-10-27T20:22:40-05:00 node0748 kernel: [14998.437727] [&lt;ffffffff810c4fd0&gt;] ? default_wake_function+0x0/0x20 2017-10-27T20:22:40-05:00 node0748 kernel: [14998.448479] [&lt;ffffffffa0a88b30&gt;] ? ptlrpcd+0x0/0x560 [ptlrpc] 2017-10-27T20:22:40-05:00 node0748 kernel: [14998.458669] [&lt;ffffffff810b064f&gt;] kthread+0xcf/0xe0 2017-10-27T20:22:40-05:00 node0748 kernel: [14998.467800] [&lt;ffffffff810b0580&gt;] ? kthread+0x0/0xe0 2017-10-27T20:22:40-05:00 node0748 kernel: [14998.477111] [&lt;ffffffff81696818&gt;] ret_from_fork+0x58/0x90 2017-10-27T20:22:40-05:00 node0748 kernel: [14998.486513] [&lt;ffffffff810b0580&gt;] ? kthread+0x0/0xe0 2017-10-27T20:22:40-05:00 node0748 kernel: [14998.495492] 2017-10-27T20:22:41-05:00 node0748 kernel: [14998.500760] Kernel panic - not syncing: LBUG 2017-10-27T20:22:41-05:00 node0748 kernel: [14998.509021] CPU: 203 PID: 1708 Comm: ptlrpcd_00_86 Tainted: P OE ------------ 3.10.0-514.6.1.el7.x86_64 #1 2017-10-27T20:22:41-05:00 node0748 kernel: [14998.524591] Hardware name: Penguin Computing Relion 1904GT/S7200AP, BIOS S72C610.86B.01.02.0001.112820162103 11/28/2016 2017-10-27T20:22:41-05:00 node0748 kernel: [14998.540144] ffffffffa0770ccc 00000000e8cd63f9 ffff882ed87a7958 ffffffff816862ac 2017-10-27T20:22:41-05:00 node0748 kernel: [14998.551919] ffff882ed87a79d8 ffffffff8167f6b3 ffffffff00000008 ffff882ed87a79e8   2017-10-27T20:22:40-05:00 node0748 kernel: [14998.118253] LustreError: 1708:0:(osc_cache.c:1931:try_to_add_extent_for_io()) extent ffff882e6c434000@{[60858 -&gt; 64953/64953], [1|0|+|lockdone|wSu|ffff882dc5cce4d0], [0|4096|+|-| (<span class="code-keyword">null</span>)|4096| (<span class="code-keyword">null</span>)]} The first extent to be fit in a RPC contains 17 chunks, which is over the limit 16. 2017-10-27T20:22:40-05:00 node0748 kernel: [14998.154188] LustreError: 1708:0:(osc_cache.c:1226:osc_extent_tree_dump0()) Dump object ffff882dc5cce4d0 extents at try_to_add_extent_for_io:1931, mppr: 4096. 2017-10-27T20:22:40-05:00 node0748 kernel: [14998.181833] LustreError: 1708:0:(osc_cache.c:1239:osc_extent_tree_dump0()) extent ffff882e6c434000@{[60858 -&gt; 64953/64953], [1|0|+|lockdone|wSu|ffff882dc5cce4d0], [0|4096|+|-| (<span class="code-keyword">null</span>)|4096| (<span class="code-keyword">null</span>)]}urgent 1. 2017-10-27T20:22:40-05:00 node0748 kernel: [14998.216119] LustreError: 1708:0:(osc_cache.c:1931:try_to_add_extent_for_io()) ASSERTION( data-&gt;erd_page_count != 0 || chunk_count &lt;= data-&gt;erd_max_chunks ) failed: 2017-10-27T20:22:40-05:00 node0748 kernel: [14998.244827] LustreError: 1708:0:(osc_cache.c:1931:try_to_add_extent_for_io()) LBUG 2017-10-27T20:22:40-05:00 node0748 kernel: [14998.259546] Pid: 1708, comm: ptlrpcd_00_86 2017-10-27T20:22:40-05:00 node0748 kernel: [14998.270365] 2017-10-27T20:22:40-05:00 node0748 kernel: [14998.270365] Call Trace: 2017-10-27T20:22:40-05:00 node0748 kernel: [14998.287055] [&lt;ffffffffa07537f3&gt;] libcfs_debug_dumpstack+0x53/0x80 [libcfs] 2017-10-27T20:22:40-05:00 node0748 kernel: [14998.301366] [&lt;ffffffffa0753861&gt;] lbug_with_loc+0x41/0xb0 [libcfs] 2017-10-27T20:22:40-05:00 node0748 kernel: [14998.314695] [&lt;ffffffffa0c69498&gt;] try_to_add_extent_for_io.isra.24+0xf58/0x12e0 [osc] 2017-10-27T20:22:40-05:00 node0748 kernel: [14998.328249] [&lt;ffffffffa0c6b9dd&gt;] osc_io_unplug0+0x3fd/0x1950 [osc] 2017-10-27T20:22:40-05:00 node0748 kernel: [14998.339730] [&lt;ffffffff810d2372&gt;] ? load_balance+0x192/0x990 2017-10-27T20:22:40-05:00 node0748 kernel: [14998.350013] [&lt;ffffffff810ce46c&gt;] ? dequeue_entity+0x11c/0x5d0 2017-10-27T20:22:40-05:00 node0748 kernel: [14998.360444] [&lt;ffffffffa0c6db30&gt;] osc_io_unplug+0x10/0x20 [osc] 2017-10-27T20:22:40-05:00 node0748 kernel: [14998.370874] [&lt;ffffffffa0c49441&gt;] brw_queue_work+0x31/0xd0 [osc] 2017-10-27T20:22:40-05:00 node0748 kernel: [14998.381510] [&lt;ffffffffa0a5e3d7&gt;] work_interpreter+0x37/0xf0 [ptlrpc] 2017-10-27T20:22:40-05:00 node0748 kernel: [14998.392538] [&lt;ffffffffa0a5b0b5&gt;] ptlrpc_check_set.part.23+0x425/0x1dd0 [ptlrpc] 2017-10-27T20:22:40-05:00 node0748 kernel: [14998.405132] [&lt;ffffffffa0a5cabb&gt;] ptlrpc_check_set+0x5b/0xe0 [ptlrpc] 2017-10-27T20:22:40-05:00 node0748 kernel: [14998.416411] [&lt;ffffffffa0a88a3b&gt;] ptlrpcd_check+0x4db/0x5d0 [ptlrpc] 2017-10-27T20:22:40-05:00 node0748 kernel: [14998.427367] [&lt;ffffffffa0a88d57&gt;] ptlrpcd+0x227/0x560 [ptlrpc] 2017-10-27T20:22:40-05:00 node0748 kernel: [14998.437727] [&lt;ffffffff810c4fd0&gt;] ? default_wake_function+0x0/0x20 2017-10-27T20:22:40-05:00 node0748 kernel: [14998.448479] [&lt;ffffffffa0a88b30&gt;] ? ptlrpcd+0x0/0x560 [ptlrpc] 2017-10-27T20:22:40-05:00 node0748 kernel: [14998.458669] [&lt;ffffffff810b064f&gt;] kthread+0xcf/0xe0 2017-10-27T20:22:40-05:00 node0748 kernel: [14998.467800] [&lt;ffffffff810b0580&gt;] ? kthread+0x0/0xe0 2017-10-27T20:22:40-05:00 node0748 kernel: [14998.477111] [&lt;ffffffff81696818&gt;] ret_from_fork+0x58/0x90 2017-10-27T20:22:40-05:00 node0748 kernel: [14998.486513] [&lt;ffffffff810b0580&gt;] ? kthread+0x0/0xe0 2017-10-27T20:22:40-05:00 node0748 kernel: [14998.495492] 2017-10-27T20:22:41-05:00 node0748 kernel: [14998.500760] Kernel panic - not syncing: LBUG 2017-10-27T20:22:41-05:00 node0748 kernel: [14998.509021] CPU: 203 PID: 1708 Comm: ptlrpcd_00_86 Tainted: P OE ------------ 3.10.0-514.6.1.el7.x86_64 #1 2017-10-27T20:22:41-05:00 node0748 kernel: [14998.524591] Hardware name: Penguin Computing Relion 1904GT/S7200AP, BIOS S72C610.86B.01.02.0001.112820162103 11/28/2016 2017-10-27T20:22:41-05:00 node0748 kernel: [14998.540144] ffffffffa0770ccc 00000000e8cd63f9 ffff882ed87a7958 ffffffff816862ac 2017-10-27T20:22:41-05:00 node0748 kernel: [14998.551919] ffff882ed87a79d8 ffffffff8167f6b3 ffffffff00000008 ffff882ed87a79e8   </pre> </div></div> <p> </p> <p>Seems to be related to <a href="https://jira.whamcloud.com/browse/LU-8680" title="replay-single test_20b: BUG: soft lockup - osc_makes_rpc()" class="issue-link" data-issue-key="LU-8680"><del>LU-8680</del></a></p> LU-10239

Lustre crash (client): The first extent to be fit in a RPC contains 17 chunks, which is over the limit 16.

Bug Critical Resolved Fixed Zhenyu Xu Murshid Azman Tue, 14 Nov 2017 08:15:57 +0000 Wed, 26 Feb 2020 15:40:24 +0000 Fri, 4 Jan 2019 05:03:27 +0000 Lustre 2.9.0 Lustre 2.13.0 1 5 <p>We just ran a job that blew up 1,600 nodes with this issue, so we're highly motivated to solve or work around it.</p> <p>My theory – which is admittedly not very deep – goes something like this:</p> <ul class="alternate" type="square"> <li>osc_io_submit() builds up a list of pending pages. It honours max_pages_per_rpc, but I don't see it making any effort to limit the number of chunks? (our max_pages_per_rpc on most nodes = 4096)</li> <li>osc_queue_sync_pages() walks the list of pending items, building up an extent.</li> <li>that extent gets added to the osc_object-&gt;oo_reading_exts</li> <li>osc_send_read_rpc() iterates over the oo_reading_exts extents</li> <li>each extent gets passed into try_to_add_extent_for_io()</li> <li>try_to_add_extent_for_io() sees that it has 17 chunks, greater than the 16 limit, and blows up</li> </ul> <p>Am I wrong? What prevents osc_io_submit() from building up a 4096-page I/O that contains more than 16 chunks?</p> <p>This is a 2.9.52 client, but I checked master and it didn't look like things were materially different in this area.</p> <p>I could see that cutting max_pages_per_rpc to 16 might work around it, but that would be an extremely harsh workaround, resulting in a lot of extremely tiny I/Os...</p> <p>Does anyone have a clue to spare, perhaps, to point us in the right direction?</p> <p>Bobijam - </p> <p>Would you please look into this issue?</p> <p>Thank you </p> <p>Bobi Jam (bobijam@hotmail.com) uploaded a new patch: <a href="https://review.whamcloud.com/30627" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30627</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10239" title="Lustre crash (client): The first extent to be fit in a RPC contains 17 chunks, which is over the limit 16." class="issue-link" data-issue-key="LU-10239"><del>LU-10239</del></a> osc: limit chunk size of write submit<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: cae710b2f9d2e257add0a572ebc999fd407a07aa</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/30627/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30627/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10239" title="Lustre crash (client): The first extent to be fit in a RPC contains 17 chunks, which is over the limit 16." class="issue-link" data-issue-key="LU-10239"><del>LU-10239</del></a> osc: limit chunk number of write submit<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 93ef6e7863b4e3aadfb3171f7f4958c5040446ec</p> <p>Landed for 2.13</p> <p>Hey, Any chances to backport this fix to 2.12.4?</p> <p>Hmm?</p> Related LU-8135 LU-11449 Development Rank 1|hzznrj: Rank (Obsolete) 9223372036854775807 Severity