https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1831 Lustre <p>Lustre: DEBUG MARKER: == sanityn test 15: test out-of-space with multiple writers ========================================== 18:15:33 (1346854533)<br/> LustreError: 7743:0:(filter_io_26.c:484:filter_direct_io()) ASSERTION( iobuf-&gt;dr_npages &gt; 0 ) failed: <br/> LustreError: 7743:0:(filter_io_26.c:484:filter_direct_io()) LBUG<br/> Pid: 7743, comm: ll_ost_io01_002</p> <p>Call Trace:<br/> <span class="error">&#91;&lt;00000000dfd5b7a1&gt;&#93;</span> libcfs_debug_dumpstack+0x51/0x80 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;00000000dfd5bf7b&gt;&#93;</span> lbug_with_loc+0x3b/0xa0 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;00000000e0e6e437&gt;&#93;</span> filter_do_bio+0x21c7/0x2a00 <span class="error">&#91;obdfilter&#93;</span><br/> <span class="error">&#91;&lt;00000000e0e7001f&gt;&#93;</span> filter_commitrw_write+0x13af/0x5010 <span class="error">&#91;obdfilter&#93;</span><br/> <span class="error">&#91;&lt;00000000dfd5ccd3&gt;&#93;</span> ? cfs_alloc+0x23/0xf0 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;00000000c109ea7a&gt;&#93;</span> ? cache_alloc_debugcheck_after.isra.40+0xca/0x180<br/> <span class="error">&#91;&lt;00000000dfd5ccd3&gt;&#93;</span> ? cfs_alloc+0x23/0xf0 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;00000000c109ed85&gt;&#93;</span> ? __kmalloc+0xb5/0x1a0<br/> <span class="error">&#91;&lt;00000000c10876a0&gt;&#93;</span> ? kzfree+0x30/0xc0<br/> <span class="error">&#91;&lt;00000000e0e623fc&gt;&#93;</span> filter_commitrw+0x29c/0x340 <span class="error">&#91;obdfilter&#93;</span><br/> <span class="error">&#91;&lt;00000000c13daa38&gt;&#93;</span> ? _spin_unlock+0x8/0x10<br/> <span class="error">&#91;&lt;00000000dff2ec4c&gt;&#93;</span> ? lprocfs_counter_add+0x14c/0x1e0 <span class="error">&#91;lvfs&#93;</span><br/> <span class="error">&#91;&lt;00000000e029443f&gt;&#93;</span> ost_brw_write+0x17fd/0x22e2 <span class="error">&#91;ost&#93;</span></p> local testing in vbox LU-1831

filter_direct_io()) ASSERTION( iobuf->dr_npages > 0 )

Bug Blocker Resolved Fixed Hongchao Zhang Alex Zhuravlev Wed, 5 Sep 2012 10:16:28 +0000 Mon, 10 Sep 2012 02:54:30 +0000 Mon, 10 Sep 2012 02:54:30 +0000 Lustre 2.3.0 Lustre 2.4.0 0 2 <p>I'm also able to hit this problem 100% of the time on my single-node testing system (real hardware), though I see it at the end of racer.sh when it is trying to unmount the filesystem.</p> <p>My first guess would be some recent change in either the RHEL 6.3 kernel bio layer which is breaking the assumption of this code, or a recent modification to nearby code. This code itself has been around for a very long time and I only started seeing the problem with the update to RHEL 6.3, but there was a window of 2 months or so where I didn't run any tests locally because the ldiskfs patches would build against the RHEL 6.2 kernel I had installed. </p> <p>I can make the node available for remote debugging if that is needed. However, it isn't ideal for a bug which crashes the node, since it cannot reboot automatically. </p> <p>I hit this quite often and can help to collect data.</p> <p>Besides this problem I also saw the module refcount of obdfilter is 2 after unmounting all OST targets.</p> <p>Bobijam</p> <p>Could you please look into this one?</p> <p>Thanks</p> <p>Peter</p> <p>I checked <tt>git log lustre/obdfilter/filter_io.c</tt> for changes that had been made recently to that code, and found the following commits:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>commit 859e5b2d20552f8df0ed2afda0f1a7c3c7d86678 Author: Hongchao Zhang &lt;hongchao.zhang@whamcloud.com&gt; Date: Mon Aug 20 15:40:48 2012 +0800 LU-657 obdfilter: fix bug in previous patch in the merged patch http://review.whamcloud.com/#change,3446, the usage of fsfilt_commit_wait is wrong, and it doesn't stop the journal firstly. Change-Id: I3a36edf7049466880c27c14bb7f99966aa75d4f1 Reviewed-on: http://review.whamcloud.com/3692 commit a9597791b658ff51474c06f419162d0a0bf03c65 Author: Hongchao Zhang &lt;hongchao.zhang@whamcloud.com&gt; Date: Tue Aug 7 08:43:11 2012 +0800 LU-657 obdfilter: commit pending journals if -ENOSPC in filter_preprw_write, if there is no enough space for this write operation, then commit the pending journals to get some more disk space and retry it. Change-Id: I46106b26443bb203eee6f01a0795b47be09170a6 Reviewed-on: http://review.whamcloud.com/3446 </pre> </div></div> <p>Reverting these two patches has allowed me to pass both racer.sh and sanityn.sh test_15 (OOS), which failed for me this morning.</p> <p>I suspect there is some kind of refcount problem in the retry code from this patch, and it is only hit when the new code is active when filesystem is nearly out of space.</p> <p>Hongchao</p> <p>Could you please look into this?</p> <p>Thanks</p> <p>Peter</p> <p>This looks like a duplicate of <a href="https://jira.whamcloud.com/browse/LU-1824" title="Test failure on test suite obdfilter-survey, subtest test_2a" class="issue-link" data-issue-key="LU-1824"><del>LU-1824</del></a>, which Yu Jian is already working on, and has a patch in <a href="http://review.whamcloud.com/3913" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/3913</a> ready for inspection.</p> Duplicate LU-1824 Development Rank 1|hzvglr: Rank (Obsolete) 6339 Severity