https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3018 Lustre <p>During obdfilter-survey on OSS node or IOR write test from the client there are small reads during bulk writes which impact the performance of the OST.<br/> Based on the block trace and dumpe2fs output these reads are for ldiskfs block bitmaps.</p> <p>A patch will be uploaded shortly.</p> LU-3018

small reads occur during bulk writes hurting overall performance

Bug Major Resolved Won't Fix Cliff White Andrew Perepechko patch Fri, 22 Mar 2013 13:48:07 +0000 Fri, 9 May 2014 19:41:50 +0000 Fri, 9 May 2014 19:41:50 +0000 Lustre 2.4.0 0 5 <p><a href="http://review.whamcloud.com/#change,5811" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,5811</a></p> <p>The logs showing how bitmap pages are evicted:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [&lt;ffffffff81135c28&gt;] ? __remove_mapping+0xd8/0x160 [&lt;ffffffff81136b7d&gt;] ? shrink_page_list.clone.0+0x47d/0x5e0 [&lt;ffffffff81136fd0&gt;] ? shrink_inactive_list+0x2f0/0x730 [&lt;ffffffffa04e90fd&gt;] ? cfs_hash_rw_unlock+0x1d/0x30 [libcfs] [&lt;ffffffffa04e7ac4&gt;] ? cfs_hash_dual_bd_unlock+0x34/0x60 [libcfs] [&lt;ffffffff8113821f&gt;] ? shrink_zone+0x38f/0x510 [&lt;ffffffff8109cc99&gt;] ? ktime_get_ts+0xa9/0xe0 [&lt;ffffffff8113849e&gt;] ? do_try_to_free_pages+0xfe/0x520 [&lt;ffffffff81138abf&gt;] ? try_to_free_pages+0x9f/0x130 [&lt;ffffffff81139c40&gt;] ? isolate_pages_global+0x0/0x380 [&lt;ffffffff811301a7&gt;] ? __alloc_pages_nodemask+0x447/0x920 [&lt;ffffffff81164e2a&gt;] ? alloc_pages_current+0xaa/0x110 [&lt;ffffffff8111ccf7&gt;] ? __page_cache_alloc+0x87/0x90 [&lt;ffffffff8111db0f&gt;] ? find_or_create_page+0x4f/0xb0 [&lt;ffffffff81135c28&gt;] ? __remove_mapping+0xd8/0x160 [&lt;ffffffff81135cc6&gt;] ? remove_mapping+0x16/0x30 [&lt;ffffffff81134bf2&gt;] ? invalidate_inode_page+0x82/0xb0 [&lt;ffffffff81134efa&gt;] ? invalidate_mapping_pages+0xda/0x150 [&lt;ffffffff814fb8eb&gt;] ? _spin_unlock+0x2b/0x40 [&lt;ffffffff811a2ec0&gt;] ? shrink_icache_memory+0x1c0/0x2e0 [&lt;ffffffff811a2f95&gt;] ? shrink_icache_memory+0x295/0x2e0 [&lt;ffffffff811362ed&gt;] ? shrink_slab+0x14d/0x1b0 [&lt;ffffffff8113963d&gt;] ? balance_pgdat+0x5ad/0x810 [&lt;ffffffff81139c40&gt;] ? isolate_pages_global+0x0/0x380 [&lt;ffffffff811399e4&gt;] ? kswapd+0x144/0x3a0 [&lt;ffffffff81139deb&gt;] isolate_pages_global+0x1ab/0x380 [&lt;ffffffff81136d99&gt;] ? shrink_inactive_list+0xb9/0x730 [&lt;ffffffff81136e42&gt;] shrink_inactive_list+0x162/0x730 [&lt;ffffffffa04e90fd&gt;] ? cfs_hash_rw_unlock+0x1d/0x30 [libcfs] [&lt;ffffffffa04e7ac4&gt;] ? cfs_hash_dual_bd_unlock+0x34/0x60 [libcfs] [&lt;ffffffffa04e9c12&gt;] ? cfs_hash_lookup+0x82/0xa0 [libcfs] [&lt;ffffffffa06a20f5&gt;] ? cl_env_fetch+0x25/0x80 [obdclass] [&lt;ffffffff8113821f&gt;] shrink_zone+0x38f/0x510 [&lt;ffffffff811397a9&gt;] balance_pgdat+0x719/0x810 [&lt;ffffffff81139c40&gt;] ? isolate_pages_global+0x0/0x380 [&lt;ffffffff811399e4&gt;] kswapd+0x144/0x3a0 </pre> </div></div> <p>Note they are not passing through shrink_active_list();</p> <p>I_NEW is needed to avoid the following code path:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [&lt;ffffffff81134efa&gt;] ? invalidate_mapping_pages+0xda/0x150 [&lt;ffffffff814fb8eb&gt;] ? _spin_unlock+0x2b/0x40 [&lt;ffffffff811a2ec0&gt;] ? shrink_icache_memory+0x1c0/0x2e0 </pre> </div></div> <p>Xyratex-bug-id: MRP-691</p> <p>Would you mind if I update the patch to support SLES11 SP2 as well?</p> <p>Hello James!<br/> That would be very nice.<br/> Thank you.</p> <p>Using mark_page_accessed() is not enough to avoid page eviction.</p> <p>find_or_create_page() allocates a page and links it to the corresponding cpu buffer.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">struct page *find_or_create_page(struct address_space *mapping, pgoff_t index, gfp_t gfp_mask) { struct page *page; <span class="code-object">int</span> err; repeat: page = find_lock_page(mapping, index); <span class="code-keyword">if</span> (!page) { page = __page_cache_alloc(gfp_mask); <span class="code-keyword">if</span> (!page) <span class="code-keyword">return</span> NULL; /* * We want a regular kernel memory (not highmem or DMA etc) * allocation <span class="code-keyword">for</span> the radix tree nodes, but we need to honour * the context-specific requirements the caller has asked <span class="code-keyword">for</span>. * GFP_RECLAIM_MASK collects those requirements. */ err = add_to_page_cache_lru(page, mapping, index, (gfp_mask &amp; GFP_RECLAIM_MASK)); ... } </pre> </div></div> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">void __lru_cache_add(struct page *page, <span class="code-keyword">enum</span> lru_list lru) { struct pagevec *pvec = &amp;get_cpu_var(lru_add_pvecs)[lru]; page_cache_get(page); <span class="code-keyword">if</span> (!pagevec_add(pvec, page)) ____pagevec_lru_add(pvec, lru); put_cpu_var(lru_add_pvecs); } </pre> </div></div> <p>Note that ____pagevec_lru_add() which calls SetPageLRU() is only performed when the cpu buffer is full.</p> <p>make_page_accessed() activates the page only if it is on LRU. Otherwise, the page is marked or kept referenced:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">void mark_page_accessed(struct page *page) { <span class="code-keyword">if</span> (!PageActive(page) &amp;&amp; !PageUnevictable(page) &amp;&amp; PageReferenced(page) &amp;&amp; PageLRU(page)) { activate_page(page); ClearPageReferenced(page); } <span class="code-keyword">else</span> <span class="code-keyword">if</span> (!PageReferenced(page)) { SetPageReferenced(page); } } </pre> </div></div> <p>shrink_inactive_list() drains the buffer and evicts the pages even if mark_page_accessed() was called a lot of times:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-keyword">static</span> unsigned <span class="code-object">long</span> shrink_inactive_list(unsigned <span class="code-object">long</span> max_scan, struct zone *zone, struct scan_control *sc, <span class="code-object">int</span> priority, <span class="code-object">int</span> file) { LIST_HEAD(page_list); struct pagevec pvec; unsigned <span class="code-object">long</span> nr_scanned = 0; unsigned <span class="code-object">long</span> nr_reclaimed = 0; unsigned <span class="code-object">long</span> nr_dirty = 0; unsigned <span class="code-object">long</span> nr_writeback = 0; struct zone_reclaim_stat *reclaim_stat = get_reclaim_stat(zone, sc); <span class="code-keyword">while</span> (unlikely(too_many_isolated(zone, file, sc))) { congestion_wait(BLK_RW_ASYNC, HZ/10); <span class="code-comment">/* We are about to die and free our memory. Return now. */</span> <span class="code-keyword">if</span> (fatal_signal_pending(current)) <span class="code-keyword">return</span> SWAP_CLUSTER_MAX; } pagevec_init(&amp;pvec, 1); lru_add_drain(); ... } </pre> </div></div> <p>Can you please post any detailed performance data you have? What environment are you testing in and what results do you see? </p> <p>Andrew Perepechko, Any update?</p> <p>Keith Mannthey, there is a lot ongoing activity in LKML.</p> <p>That is excellent news. </p> <p>this ticket should be closed and the long-term solution backported from the vanilla kernel</p> <p>Closing ticket per Andrew</p> Development Rank 1|hzvlz3: Rank (Obsolete) 7341 Severity