https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4856 Lustre <p>The atomic_t used to count LRU entries is overflowing on systems with large memory configurations:</p> <p>LustreError: 22141:0:(osc_page.c:892:osc_lru_reserve()) ASSERTION(atomic_read(cli-&gt;cl_lru_left) &gt;= 0 ) failed:</p> <p>PID: 54214 TASK: ffff88fdef4e4100 CPU: 40 COMMAND: "cat"<br/> #3 <span class="error">&#91;ffff88fdf0823900&#93;</span> lbug_with_loc at ffffffffa07fedc3 <span class="error">&#91;libcfs&#93;</span><br/> #4 <span class="error">&#91;ffff88fdf0823920&#93;</span> osc_lru_reserve at ffffffffa0c2a28a <span class="error">&#91;osc&#93;</span><br/> #5 <span class="error">&#91;ffff88fdf08239a0&#93;</span> cl_page_alloc at ffffffffa09a7122 <span class="error">&#91;obdclass&#93;</span><br/> #6 <span class="error">&#91;ffff88fdf08239e0&#93;</span> cl_page_find0 at ffffffffa09a742d <span class="error">&#91;obdclass&#93;</span><br/> #7 <span class="error">&#91;ffff88fdf0823a40&#93;</span> lov_page_init_raid0 at ffffffffa0cc0f21 <span class="error">&#91;lov&#93;</span><br/> #8 <span class="error">&#91;ffff88fdf0823aa0&#93;</span> cl_page_alloc at ffffffffa09a7122 <span class="error">&#91;obdclass&#93;</span><br/> #9 <span class="error">&#91;ffff88fdf0823ae0&#93;</span> cl_page_find0 at ffffffffa09a742d <span class="error">&#91;obdclass&#93;</span><br/> #10 <span class="error">&#91;ffff88fdf0823b40&#93;</span> ll_cl_init at ffffffffa0d74123 <span class="error">&#91;lustre&#93;</span><br/> #11 <span class="error">&#91;ffff88fdf0823bd0&#93;</span> ll_readpage at ffffffffa0d74485 <span class="error">&#91;lustre&#93;</span><br/> #12 <span class="error">&#91;ffff88fdf0823c00&#93;</span> do_generic_file_read at ffffffff810fa39e<br/> #13 <span class="error">&#91;ffff88fdf0823c80&#93;</span> generic_file_aio_read at ffffffff810fad4c<br/> #14 <span class="error">&#91;ffff88fdf0823d40&#93;</span> vvp_io_read_start at ffffffffa0da2fb0 <span class="error">&#91;lustre&#93;</span><br/> #15 <span class="error">&#91;ffff88fdf0823da0&#93;</span> cl_io_start at ffffffffa09af979 <span class="error">&#91;obdclass&#93;</span><br/> #16 <span class="error">&#91;ffff88fdf0823dd0&#93;</span> cl_io_loop at ffffffffa09b3d33 <span class="error">&#91;obdclass&#93;</span><br/> #17 <span class="error">&#91;ffff88fdf0823e00&#93;</span> ll_file_io_generic at ffffffffa0d49c32 <span class="error">&#91;lustre&#93;</span><br/> #18 <span class="error">&#91;ffff88fdf0823e70&#93;</span> ll_file_aio_read at ffffffffa0d4a3b3 <span class="error">&#91;lustre&#93;</span><br/> #19 <span class="error">&#91;ffff88fdf0823ec0&#93;</span> ll_file_read at ffffffffa0d4aec3 <span class="error">&#91;lustre&#93;</span><br/> #20 <span class="error">&#91;ffff88fdf0823f10&#93;</span> vfs_read at ffffffff8115b237<br/> #21 <span class="error">&#91;ffff88fdf0823f40&#93;</span> sys_read at ffffffff8115b3a3</p> <p>In this case, the atomic_t (signed int) held:<br/> crash&gt; pd <b>(int</b>)0xffff943de11780fc<br/> $10 = -1506317746</p> <p>We've triggered this specific problem with configurations down to 11TB of physmem. A 10.5TB system can cat a small file without crashing.</p> <p>I noticed several other cases where page counts are handled using a signed int, and suspect anything more than 4TB is problematic. The kernel itself is consistently using unsigned long for page counts on all architectures.</p> LU-4856

osc_lru_reserve()) ASSERTION( atomic_read(cli->cl_lru_left) >= 0 ) failed

Bug Major Resolved Fixed Jian Yu Stephen Champion patch Thu, 3 Apr 2014 03:00:56 +0000 Thu, 1 Oct 2015 14:23:07 +0000 Tue, 30 Sep 2014 13:32:42 +0000 Lustre 2.5.0 Lustre 2.6.0 Lustre 2.4.2 Lustre 2.7.0 0 10 <p>we can use atomic64_t instead.</p> <p>I've been digging at this, trying to identify the changes required. To support large memory systems, all global accounting of pages needs to be done with 64 types. Just tracing usage of cfs_num_physpages (which cl_lru_left is derived from), the problem snowballs quickly, and affects almost every subsystem in Lustre.</p> <p>Some casting will be required, but it should not be a problem to use 32 bit counters for page vectors. I doubt any networks support 8 TB transactions yet.</p> <p>I have been working on a patch against master to address easily identified overflow hazards. This cascaded into lock management as well.</p> <p>I am about to give it a whirl on internal systems to make sure I didn't break anything, then allocate time on a system with 16T of memory to make sure it addresses the problem. I won't be able to run acceptance on the large system anytime soon, but will do some basic functionality testing.</p> <p>I will also need to cleanup for coding standards.</p> <p><a href="http://review.whamcloud.com/#/c/10537/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/10537/</a></p> <p>I setup an i686 build environment and worked through the initial errors.</p> <p>The kernel does not implement atomic64_add_unless on this arch, so I'll have to find a way around this problem. I will push the updated patch for feedback, but there will certainly be additional revisions, possibly major.</p> <p>Hello Stephen,<br/> Do you want us to keep this ticket open?</p> <p>Many thanks,<br/> ~ jfc.</p> <p>Yes please. The patch needs to have i686 build problems addressed, and I need to sync up with everyone who offered comments. I expect to get back to it during the week of Aug 5.</p> <p>I pushed a new revision of the patch this morning. I expected tests to start automatically - do I need to add Test-Parameters?</p> <p>I will testing on my own x86_64 / IB test environment today, but do not have a means to test i686.<br/> This has not yet been tested on a large memory system. We are starting that process.</p> <p>Hi Steve</p> <p>It's started testing now. Just higher than usual load on the test system. <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Peter</p> <p>Revision 4 of <a href="http://review.whamcloud.com/#/c/10537/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/10537/</a> is tested, working.</p> <ol> <li>rpm -q lustre-client<br/> lustre-client-2.6.51-3.0.101_0.35_default_gc69b1a0</li> <li>grep ^processor /proc/cpuinfo | wc -l<br/> 3072</li> <li>grep ^MemTotal /proc/meminfo<br/> MemTotal: 32825421388 kB</li> <li>mount -t lustre mds1-esa@tcp0:/esa-uv /mnt/esa-uv</li> <li>cd /mnt/esa-uv/schamp</li> <li>ls -l<br/> total 3145740<br/> <del>rw-r</del><del>r</del>- 1 schamp sgiemp_00 1073741824 Sep 4 14:36 foo.1<br/> <del>rw-r</del><del>r</del>- 1 schamp sgiemp_00 1073741824 Sep 4 15:20 foo.2</li> <li>cp foo.2 foo.3</li> <li>ls -l<br/> total 3145740<br/> <del>rw-r</del><del>r</del>- 1 schamp sgiemp_00 1073741824 Sep 4 14:36 foo.1<br/> <del>rw-r</del><del>r</del>- 1 schamp sgiemp_00 1073741824 Sep 4 15:20 foo.2<br/> <del>rw-r</del><del>r</del>- 1 root root 1073741824 Sep 4 18:24 foo.3</li> </ol> <p><a href="http://review.whamcloud.com/#/c/10537/5" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/10537/5</a> is confirmed as resolving this problem on a 32TB system.<br/> I also ran sanity and sanityn without serious failure.</p> <p>Stephen,<br/> Can you please review the comments on patch set 5?</p> <p>Thanks,<br/> ~ jfc.</p> <p>In the middle of it right now. Had to rebase to master again.</p> <p>I am hesitant to simply #define the lprocfs_.._long functions to _u64 functions, as sign conversion hazards might catch unsuspecting users. Seems like a great way to introduce very obscure bugs.</p> <p>I think I can eliminate the introduction of the long function by using the _64 functions in the cases my patch was using them.<br/> This does force 32 bit systems to unnecessarily use 64 bit types, but not in critical paths. This is what I have started on.</p> <p><a href="http://review.whamcloud.com/#/c/10537/7" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/10537/7</a> eliminates the lprocfs_..long functions entirely.</p> <p>Made it through autotest.<br/> ~ jfc.</p> <p>Landed for 2.7</p> <p>Here is the patch for master branch to resolve the issue that if "val" is larger than 2^32 on a 32-bit system, the code in proc_max_dirty_pages_in_mb() may truncate "val" when assigning it to obd_max_dirty_pages: <a href="http://review.whamcloud.com/12269/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12269/</a></p> <p>Hi </p> <p>We are seeing same issues with SLES 11 + SP3 and we using Lustre version 2.4.3</p> <p>luster client installed with 2048 core SGI UV1000 running:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> cat /etc/SuSE-release SUSE Linux Enterprise Server 11 (x86_64) VERSION = 11 PATCHLEVEL = 3 hungabee:~ # lsb_release -a LSB Version: core-2.0-noarch:core-3.2-noarch:core-4.0-noarch:core-2.0-x86_64:core-3.2-x86_64:core-4.0-x86_64:desktop-4.0-amd64:desktop-4.0-noarch:graphics-2.0-amd64:graphics-2.0-noarch:graphics-3.2-amd64:graphics-3.2-noarch:graphics-4.0-amd64:graphics-4.0-noarch Distributor ID: SUSE LINUX Description: SUSE Linux Enterprise Server 11 (x86_64) Release: 11 Codename: n/a hungabee:~ # rpm -qa | egrep "(lustre|ofed)" lustre-client-modules-2.4.3-3.0.101_0.29_default ofed-doc-1.5.4.1-0.11.5 ofed-1.5.4.1-0.11.5 ofed-kmp-trace-1.5.4.1_3.0.76_0.11-0.11.5 ofed-kmp-default-1.5.4.1_3.0.76_0.11-0.11.5 lustre-client-2.4.3-3.0.101_0.29_default </pre> </div></div> <p>So can we have a backport path for Lustre client v2.4.3 and is this patch included in 2.5.x branches if not then can we have backport patch for 2.5 branch too.</p> <p>Thank You,<br/> Manish</p> <p>Grégoire Pichon (gregoire.pichon@bull.net) uploaded a new patch: <a href="http://review.whamcloud.com/16697" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16697</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-4856" title="osc_lru_reserve()) ASSERTION( atomic_read(cli-&gt;cl_lru_left) &gt;= 0 ) failed" class="issue-link" data-issue-key="LU-4856"><del>LU-4856</del></a> misc: Reduce exposure to overflow on page counters.<br/> Project: fs/lustre-release<br/> Branch: b2_5<br/> Current Patch Set: 1<br/> Commit: f14f45c4e52246efe2c478b87c703705a30b3774</p> Related Development Rank 1|hzwj5z: Rank (Obsolete) 13394 Severity