https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-16441 Lustre <p>The following  crash hit our OSS servers for a production system.</p> <p><span class="error">&#91;3845523.531585&#93;</span> Call Trace: <br/> <span class="error">&#91;3845523.535134&#93;</span>  <span class="error">&#91;&lt;ffffffffb7d865c9&gt;&#93;</span> dump_stack+0x19/0x1b <br/> <span class="error">&#91;3845523.541350&#93;</span>  <span class="error">&#91;&lt;ffffffffb7d81905&gt;&#93;</span> bad_page.part.75+0xdc/0xf9 <br/> <span class="error">&#91;3845523.548082&#93;</span>  <span class="error">&#91;&lt;ffffffffb77c6fd2&gt;&#93;</span> free_pages_prepare+0x1f2/0x210 <br/> <span class="error">&#91;3845523.555139&#93;</span>  <span class="error">&#91;&lt;ffffffffb77c7309&gt;&#93;</span> __free_pages_ok+0x19/0xc0 <br/> <span class="error">&#91;3845523.561750&#93;</span>  <span class="error">&#91;&lt;ffffffffb77c73cb&gt;&#93;</span> free_compound_page+0x1b/0x20 <br/> <span class="error">&#91;3845523.568605&#93;</span>  <span class="error">&#91;&lt;ffffffffb7d8236f&gt;&#93;</span> __put_compound_page+0x25/0x28 <br/> <span class="error">&#91;3845523.575535&#93;</span>  <span class="error">&#91;&lt;ffffffffb7d824d8&gt;&#93;</span> put_compound_page+0x166/0x174 <br/> <span class="error">&#91;3845523.582456&#93;</span>  <span class="error">&#91;&lt;ffffffffb77cce06&gt;&#93;</span> put_page+0x56/0x60 <br/> <span class="error">&#91;3845523.588404&#93;</span>  <span class="error">&#91;&lt;ffffffffb7c4266f&gt;&#93;</span> skb_release_data+0x8f/0x150 <br/> <span class="error">&#91;3845523.595114&#93;</span>  <span class="error">&#91;&lt;ffffffffb7c42754&gt;&#93;</span> skb_release_all+0x24/0x30 <br/> <span class="error">&#91;3845523.601633&#93;</span>  <span class="error">&#91;&lt;ffffffffb7c42772&gt;&#93;</span> __kfree_skb+0x12/0x20 <br/> <span class="error">&#91;3845523.607795&#93;</span>  <span class="error">&#91;&lt;ffffffffb7cbecad&gt;&#93;</span> tcp_ack+0x60d/0x12f0 <br/> <span class="error">&#91;3845523.613856&#93;</span>  <span class="error">&#91;&lt;ffffffffb7cbff66&gt;&#93;</span> tcp_rcv_established+0x1d6/0x7a0 <br/> <span class="error">&#91;3845523.620857&#93;</span>  <span class="error">&#91;&lt;ffffffffb7cc83e3&gt;&#93;</span> ? tcp_v4_md5_lookup+0x13/0x20 <br/> <span class="error">&#91;3845523.627689&#93;</span>  <span class="error">&#91;&lt;ffffffffb7ccb04a&gt;&#93;</span> tcp_v4_do_rcv+0x10a/0x350 <br/> <span class="error">&#91;3845523.634166&#93;</span>  <span class="error">&#91;&lt;ffffffffb7c3e986&gt;&#93;</span> release_sock+0xa6/0x180 <br/> <span class="error">&#91;3845523.640454&#93;</span>  <span class="error">&#91;&lt;ffffffffb7cb609d&gt;&#93;</span> tcp_sendpage+0xdd/0x5c0 <br/> <span class="error">&#91;3845523.646746&#93;</span>  <span class="error">&#91;&lt;ffffffffc0b47bab&gt;&#93;</span> ksocknal_lib_send_kiov+0xdb/0x2e0 <span class="error">&#91;ksocklnd&#93;</span> <br/> <span class="error">&#91;3845523.654853&#93;</span>  <span class="error">&#91;&lt;ffffffffc0b48662&gt;&#93;</span> ? ksocknal_lib_send_iov+0xd2/0x140 <span class="error">&#91;ksocklnd&#93;</span> <br/> <span class="error">&#91;3845523.663040&#93;</span>  <span class="error">&#91;&lt;ffffffffc0b4112e&gt;&#93;</span> ksocknal_process_transmit+0x39e/0xc10 <span class="error">&#91;ksocklnd&#93;</span> <br/> <span class="error">&#91;3845523.671478&#93;</span>  <span class="error">&#91;&lt;ffffffffc0b45b80&gt;&#93;</span> ksocknal_scheduler+0x320/0xd50 <span class="error">&#91;ksocklnd&#93;</span> <br/> <span class="error">&#91;3845523.679304&#93;</span>  <span class="error">&#91;&lt;ffffffffb76c7080&gt;&#93;</span> ? wake_up_atomic_t+0x30/0x30 <br/> <span class="error">&#91;3845523.685989&#93;</span>  <span class="error">&#91;&lt;ffffffffc0b45860&gt;&#93;</span> ? ksocknal_recv+0x2a0/0x2a0 <span class="error">&#91;ksocklnd&#93;</span> <br/> <span class="error">&#91;3845523.693527&#93;</span>  <span class="error">&#91;&lt;ffffffffb76c5f91&gt;&#93;</span> kthread+0xd1/0xe0 <br/> <span class="error">&#91;3845523.699238&#93;</span>  <span class="error">&#91;&lt;ffffffffb76c5ec0&gt;&#93;</span> ? insert_kthread_work+0x40/0x40 <br/> <span class="error">&#91;3845523.706156&#93;</span>  <span class="error">&#91;&lt;ffffffffb7d99ddd&gt;&#93;</span> ret_from_fork_nospec_begin+0x7/0x21 <br/> <span class="error">&#91;3845523.713415&#93;</span>  <span class="error">&#91;&lt;ffffffffb76c5ec0&gt;&#93;</span> ? insert_kthread_work+0x40/0x40 <br/> <span class="error">&#91;3845523.720321&#93;</span> BUG: Bad page state in process socknal_sd05_02  pfn:282c3e3 <br/> <span class="error">&#91;3845523.727747&#93;</span> page:ffffeb1aa0b0f8c0 count:0 mapcount:-1 mapping:          (null) index:0x0 <br/> <span class="error">&#91;3845523.736648&#93;</span> page flags: 0x6fffff00008000(tail) <br/> <span class="error">&#91;3845523.741962&#93;</span> page dumped because: nonzero mapcount</p> Lustre 2.12.9 vanilla servers running in RHEL7.9 environment. Ethernet hardware is 200GiB cards. LU-16441

BUG: Bad page state in process socknal_* functions

Bug Critical Open Unresolved James A Simmons James A Simmons ORNL Tue, 3 Jan 2023 19:51:31 +0000 Mon, 9 Jan 2023 14:29:52 +0000 Lustre 2.16.0 Lustre 2.12.9 Lustre 2.15.1 0 4 <p>Both NVME and ceph ran into the same issue. Basically memory in the skbuff is being used after it is freed. I have a working prototype patch I'm testing right now.</p> <p>"James Simmons &lt;jsimmons@infradead.org&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/49545" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/49545</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16441" title="BUG: Bad page state in process socknal_* functions" class="issue-link" data-issue-key="LU-16441">LU-16441</a> ksocklnd: ensure a page is valid with sendpage_ok()<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 444ba4c1625268fce3b7a031f932360554f26983</p> Related Development Rank 1|i038zr: Rank (Obsolete) 9223372036854775807 Severity