https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1878 Lustre <p>Kernel OOPS on a client</p> <p>&lt;1&gt;<span class="error">&#91;1740431.005226&#93;</span> BUG: unable to handle kernel NULL pointer dereference at (null)<br/> &lt;1&gt;<span class="error">&#91;1740431.008694&#93;</span> IP: <span class="error">&#91;&lt;ffffffffa0a9dfda&gt;&#93;</span> ll_readahead+0x4a/0x1d20 <span class="error">&#91;lustre&#93;</span><br/> &lt;4&gt;<span class="error">&#91;1740431.008694&#93;</span> PGD 2cffabf5067 PUD 2cffabf4067 PMD 0<br/> &lt;0&gt;<span class="error">&#91;1740431.008694&#93;</span> Oops: 0000 <a href="#1" target="_blank" rel="noopener">1</a> SMP<br/> &lt;0&gt;<span class="error">&#91;1740431.008694&#93;</span> last sysfs file: /sys/devices/system/node/node63/meminfo<br/> <span class="error">&#91;archkdbcommon&#93;</span>kdb&gt; -bt</p> <p>Stack traceback for pid 839153<br/> 0xffff8acffbd72600 000000 000000 0 000 R 0xffff8acffbd72c90 *magic.exe<br/> <span class="error">&#91;&lt;ffffffffa0a9dfda&gt;&#93;</span> ll_readahead+0x4a/0x1d20 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0aa0ebb&gt;&#93;</span> ll_readpage+0x120b/0x1760 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffff810b5cf1&gt;&#93;</span> filemap_fault+0x241/0x3d0<br/> <span class="error">&#91;&lt;ffffffffa0aaebd7&gt;&#93;</span> ll_fault+0x3b7/0xdd0 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffff810ceae7&gt;&#93;</span> __do_fault+0x57/0x520<br/> <span class="error">&#91;&lt;ffffffff810d3459&gt;&#93;</span> handle_mm_fault+0x199/0x430<br/> <span class="error">&#91;&lt;ffffffff8139dcaf&gt;&#93;</span> do_page_fault+0x1bf/0x3e0<br/> <span class="error">&#91;&lt;ffffffff8139b5cf&gt;&#93;</span> page_fault+0x1f/0x30<br/> <span class="error">&#91;&lt;0000000000472d64&gt;&#93;</span> 0x472d64</p> LU-1878

NULL pointer in ll_readahead()

Bug Minor Resolved Won't Fix Bob Glossman Vladimir V. Saveliev Mon, 10 Sep 2012 17:00:56 +0000 Mon, 29 May 2017 05:23:12 +0000 Mon, 29 May 2017 05:23:12 +0000 Lustre 1.8.x (1.8.0 - 1.8.5) 0 3 <p>The BUG happened because ll_readhead() was called from ll_readpage() with<br/> mapping == NULL, so the line</p> <p> inode = mapping-&gt;host<br/> caused the oops.</p> <p>At the begining of ll_readpage() page-&gt;mapping was not NULL, because the line<br/> struct inode *inode = page-&gt;mapping-&gt;host;<br/> worked fine. Therefore, page-&gt;mapping changed to NULL while running ll_readpage().</p> <p>At first sight it looks like the page kept locked all the time at ll_readpage().<br/> However, the following race seems to be possible:</p> <p>Process 1:<br/> ll_readpage() called via handle_mm_fault()<del>&gt;__do_fault()</del>&gt;ll_fault()-&gt;filemap_fault():<br/> starts with locked page P and calls ll_issue_page_read() which queues the page for i/o.<br/> Once the page is queued it can be involved into group i/o triggered by another thread.</p> <p>Process 2:<br/> obd_trigger_group_io()<br/> triggers i/o for group of queued pages, including the page P. The P gets unlocked and uptodate as result of that.</p> <p>Process 3:<br/> shrink_slab()<del>&gt;llap_shrink_cache()</del>&gt;llap_shrink_cache_internal()<br/> kswapd shrinks slabs and calls ll_shrink_cache() (registered lustre's slab shrinker, which tries to shrink lustre pages.<br/> llap_shrink_cache_internal() locks the page P and calls ll_truncate_complete_page() which removes the page P from page cache (page-&gt;mapping becomes NULL).<br/> llap_shrink_cache_internal() avoids to ll_truncate_complete_page() dirty pages, but it does not care of busy pages: the ones with increase reference count.</p> <p>Process 1 continues with page-&gt;mapping == NULL and calls ll_readahead() passing NULL as 3rd argument:<br/> ll_readpage()<br/> ...<br/> /* We have just requested the actual page we want, see if we can tack</p> <ul> <li>on some readahead to that page's RPC before it is sent. */<br/> if (ll_i2sbi(inode)-&gt;ll_ra_info.ra_max_pages_per_file)<br/> ll_readahead(&amp;fd-&gt;fd_ras, exp, page-&gt;mapping, oig,<br/> fd-&gt;fd_flags);</li> </ul> <p>ll_readahead()<br/> {<br/> ...<br/> inode = mapping-&gt;host; &lt;&lt;&lt;== NULL pointer dereference happens here, as long as mapping is NULL<br/> ...</p> <p>Dump of stacktraces provided by HLRN (<a href="https://bugzilla.lustre.org/attachment.cgi?id=33148" class="external-link" target="_blank" rel="nofollow noopener">https://bugzilla.lustre.org/attachment.cgi?id=33148</a>) contains number of processes:</p> <ul class="alternate" type="square"> <li>handling pagefaults of lustre pages looking like:<br/> 0xffff8acffbd72600 839153 838417 1 343 R 0xffff8acffbd72c90 *magic.exe<br/> <span class="error">&#91;&lt;ffffffffa0a9dfda&gt;&#93;</span> ll_readahead+0x4a/0x1d20 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0aa0ebb&gt;&#93;</span> ll_readpage+0x120b/0x1760 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffff810b5cf1&gt;&#93;</span> filemap_fault+0x241/0x3d0<br/> <span class="error">&#91;&lt;ffffffffa0aaebd7&gt;&#93;</span> ll_fault+0x3b7/0xdd0 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffff810ceae7&gt;&#93;</span> __do_fault+0x57/0x520<br/> <span class="error">&#91;&lt;ffffffff810d3459&gt;&#93;</span> handle_mm_fault+0x199/0x430<br/> <span class="error">&#91;&lt;ffffffff8139dcaf&gt;&#93;</span> do_page_fault+0x1bf/0x3e0<br/> <span class="error">&#91;&lt;ffffffff8139b5cf&gt;&#93;</span> page_fault+0x1f/0x30</li> <li>processes which are entered lustre shrink code trying to allocate memory and kswapd's shrinking slabs with typical trace like:<br/> 0xffff881072408680 7239 2 0 965 D 0xffff881072408d10 kswapd56<br/> <span class="error">&#91;&lt;ffffffff81399094&gt;&#93;</span> thread_return+0x0/0x34c<br/> <span class="error">&#91;&lt;ffffffff810cf589&gt;&#93;</span> mapping_sleep+0x9/0x10<br/> <span class="error">&#91;&lt;ffffffff8139986a&gt;&#93;</span> __wait_on_bit_lock+0x4a/0xa0<br/> <span class="error">&#91;&lt;ffffffff81399939&gt;&#93;</span> out_of_line_wait_on_bit_lock+0x79/0xa0<br/> <span class="error">&#91;&lt;ffffffff810d2309&gt;&#93;</span> unmap_mapping_range+0x149/0x310<br/> <span class="error">&#91;&lt;ffffffffa0aae6c7&gt;&#93;</span> ll_teardown_mmaps+0x67/0x1c0 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0aa2072&gt;&#93;</span> llap_shrink_cache+0x5a2/0xa40 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0a83763&gt;&#93;</span> ll_shrink_cache+0x33/0x70 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffff810c3f8c&gt;&#93;</span> shrink_slab+0x12c/0x190<br/> <span class="error">&#91;&lt;ffffffff810c49d8&gt;&#93;</span> balance_pgdat+0x4c8/0x7b0<br/> <span class="error">&#91;&lt;ffffffff810c4de9&gt;&#93;</span> kswapd+0x129/0x180<br/> <span class="error">&#91;&lt;ffffffff81064366&gt;&#93;</span> kthread+0x96/0xa0</li> </ul> <p>which shows that the race described above is possible.<br/> The proposed fix is to avoid to ll_truncate_complete_page() pages having extra reference counters.</p> <p>llap_shrink_cache_internal(): avoid shrinking of pages which are in use</p> <p>Thanks for the patch Vladimir. Could you please upload it into gerrit so that we can review and test it? There are instructions on the whamcloud wiki or I can get an engineer to step you through the process if that helps.</p> <p>Hello, Peter</p> <p>That is what I was planning to do next. From <a href="http://wiki.whamcloud.com/display/PUB/Submitting+Changes" class="external-link" target="_blank" rel="nofollow noopener">http://wiki.whamcloud.com/display/PUB/Submitting+Changes</a> <br/> I realised that JIRA ticket needs to be created first. So, I created <br/> <a href="https://jira.whamcloud.com/browse/LU-1878" title="NULL pointer in ll_readahead()" class="issue-link" data-issue-key="LU-1878"><del>LU-1878</del></a>.</p> <p>I hope that my patch matches<br/> 2. Test and commit your patch locally using acceptance-small.sh and<br/> 3. The patch follows the Requirements for patch submission:<br/> and I was going to do 4. The patch has been uploaded to Gerrit. On <br/> that step I guess that I am to git push the patch somewhere.</p> <p>If I went wrong way or there is simple way, please get someone to help <br/> me.</p> <p>TIA<br/> Best regards,<br/> Vladimir</p> <p>Great! It sounds like you are on the right track. Once the patch is in gerrit then you should post the URL to the changeset as a comment in the JIRA ticket. Thanks!</p> <p><a href="http://review.whamcloud.com/3927" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/3927</a></p> <p>the patch is in gerrit (<a href="http://review.whamcloud.com/3927" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/3927</a>), however, I <br/> am not sure about inspectors, please advise.</p> <p>Excellent - thanks Vladimir! Don't worry about the inspectors - we'll organize that.</p> <p>Bob</p> <p>Could you please take care of landing this patch and check in particular its applicability to the master branch</p> <p>Thanks</p> <p>Peter</p> <p><a href="http://review.whamcloud.com/4026" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4026</a></p> <p>this is updated version accordingly to Jinshan Xiong's inspection.</p> <p>Close old ticket.</p> Bugzilla ID 24376.0 Development Epic client Rank 1|hzvyfz: Rank (Obsolete) 9764 Severity