https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2598 Lustre <p>On execution of obdfilter-survey with rszlo="2048" and rszhi="2048" (non-default params) against a single OST, an LBUG is generated:</p> <p>LustreError: 3202:0:(filter_io_26.c:297:filter_iobuf_add_page()) ASSERTION( iobuf-&gt;dr_npages &lt; iobuf-&gt;dr_max_pages ) failed: <br/> LustreError: 3202:0:(filter_io_26.c:297:filter_iobuf_add_page()) LBUG<br/> LustreError: 3203:0:(filter_io_26.c:297:filter_iobuf_add_page()) ASSERTION( iobuf-&gt;dr_npages &lt; iobuf-&gt;dr_max_pages ) failed: <br/> LustreError: 3203:0:(filter_io_26.c:297:filter_iobuf_add_page()) LBUG<br/> Kernel panic - not syncing: LBUG<br/> Pid: 3202, comm: lctl Not tainted 2.6.32-279.2.1.el6_lustre.gc46c389.x86_64 #1<br/> Call Trace:<br/> <span class="error">&#91;&lt;ffffffff814fd57a&gt;&#93;</span> ? panic+0xa0/0x168<br/> <span class="error">&#91;&lt;ffffffffa0508e5b&gt;&#93;</span> ? lbug_with_loc+0x9b/0xb0 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0bef472&gt;&#93;</span> ? filter_iobuf_add_page+0x62/0x70 <span class="error">&#91;obdfilter&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0bf2762&gt;&#93;</span> ? filter_commitrw_write+0xa62/0x2e78 <span class="error">&#91;obdfilter&#93;</span><br/> <span class="error">&#91;&lt;ffffffff8112b690&gt;&#93;</span> ? __lru_cache_add+0x40/0x90<br/> <span class="error">&#91;&lt;ffffffffa0be6fa7&gt;&#93;</span> ? filter_preprw_write+0xc87/0x1cd0 <span class="error">&#91;obdfilter&#93;</span><br/> <span class="error">&#91;&lt;ffffffff81162710&gt;&#93;</span> ? cache_alloc_refill+0x1c0/0x240<br/> <span class="error">&#91;&lt;ffffffffa0be6252&gt;&#93;</span> ? filter_commitrw+0x272/0x290 <span class="error">&#91;obdfilter&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0be8d88&gt;&#93;</span> ? filter_preprw+0x68/0x80 <span class="error">&#91;obdfilter&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0be919e&gt;&#93;</span> ? filter_brw+0x3fe/0x740 <span class="error">&#91;obdfilter&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0a80f8d&gt;&#93;</span> ? echo_client_kbrw+0xd6d/0x1a50 <span class="error">&#91;obdecho&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0517894&gt;&#93;</span> ? cfs_hash_dual_bd_unlock+0x34/0x60 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa051b0c2&gt;&#93;</span> ? cfs_hash_del+0xa2/0x1d0 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0a827d7&gt;&#93;</span> ? echo_client_brw_ioctl+0x1f7/0x1380 <span class="error">&#91;obdecho&#93;</span><br/> <span class="error">&#91;&lt;ffffffff81212e39&gt;&#93;</span> ? security_capable+0x29/0x30<br/> <span class="error">&#91;&lt;ffffffffa0a86f38&gt;&#93;</span> ? echo_client_iocontrol+0x638/0x1d00 <span class="error">&#91;obdecho&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa050819b&gt;&#93;</span> ? cfs_set_ptldebug_header+0x2b/0xc0 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0509993&gt;&#93;</span> ? cfs_alloc+0x63/0x90 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa05c4cfa&gt;&#93;</span> ? obd_ioctl_getdata+0x13a/0x1160 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa05d83cf&gt;&#93;</span> ? class_handle_ioctl+0x12ff/0x1ed0 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffff8113ff34&gt;&#93;</span> ? handle_mm_fault+0x1e4/0x2b0<br/> <span class="error">&#91;&lt;ffffffffa05c42ab&gt;&#93;</span> ? obd_class_ioctl+0x4b/0x190 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffff8118dff2&gt;&#93;</span> ? vfs_ioctl+0x22/0xa0<br/> <span class="error">&#91;&lt;ffffffff81500a85&gt;&#93;</span> ? page_fault+0x25/0x30<br/> <span class="error">&#91;&lt;ffffffff8118e194&gt;&#93;</span> ? do_vfs_ioctl+0x84/0x580<br/> <span class="error">&#91;&lt;ffffffff8118e711&gt;&#93;</span> ? sys_ioctl+0x81/0xa0<br/> <span class="error">&#91;&lt;ffffffff8100b0f2&gt;&#93;</span> ? system_call_fastpath+0x16/0x1b</p> <p>The LBUG is consistent and reproducible on my test VM cluster using this command-line for obdfilter-survey:</p> <p>size="512" rszlo="2048" rszhi="2048" nobjlo="2" thrlo="2" nobjhi="32" thrhi="32" case="disk" rslt_loc="/root/obdres" obdfilter-survey</p> VirtualBox 4.2.6 VM<br/> CentOS 6.3 x86_64<br/> Kernel 2.6.32-279.2.1.el6_lustre.gc46c389.x86_64 <br/> Lustre Version: 2.1.3<br/> LU-2598

obdfilter-survey LBUG ASSERTION( iobuf->dr_npages < iobuf->dr_max_pages ) failed

Bug Major Resolved Fixed Jian Yu Malcolm Cowe LB Wed, 9 Jan 2013 21:00:47 +0000 Tue, 15 Oct 2013 22:06:49 +0000 Tue, 15 Oct 2013 22:06:49 +0000 Lustre 2.4.0 Lustre 2.1.3 Lustre 2.4.0 Lustre 2.1.5 Lustre 2.1.6 Lustre 2.5.0 0 9 <p>Yes, the maximum IO size is 1MB, but the code shouldn't crash if some larger IO size is specified. The code should return an error in this case, or handle the larger IO by submitting multiple IO requests. This may also be fixed by the 4MB RPC patch in <a href="https://jira.whamcloud.com/browse/LU-1431" title="Support for larger than 1MB sequential I/O RPCs" class="issue-link" data-issue-key="LU-1431"><del>LU-1431</del></a>.</p> <p>The <a href="http://review.whamcloud.com/4993" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4993</a> patch looks like it will resolve this problem in osd-ldiskfs/osd-io.c:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">- bio = bio_alloc(GFP_NOIO, max(BIO_MAX_PAGES, + bio = bio_alloc(GFP_NOIO, min(BIO_MAX_PAGES, </pre> </div></div> <p>If the patches for <a href="https://jira.whamcloud.com/browse/LU-1431" title="Support for larger than 1MB sequential I/O RPCs" class="issue-link" data-issue-key="LU-1431"><del>LU-1431</del></a> are not going to be landed on Lustre b2_1 branch, then we need cherry-pick the patch of <a href="http://review.whamcloud.com/1741" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/1741</a> for <a href="https://jira.whamcloud.com/browse/LU-844" title="LBUG: ASSERTION(iobuf-&gt;dr_npages &lt; iobuf-&gt;dr_max_pages) when run obdfilter_survey using rsz &gt;= 2M" class="issue-link" data-issue-key="LU-844"><del>LU-844</del></a> to b2_1 so as to get the following info while running obdfilter-survey with more than 1MB IO size:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Test disk case support maximum 1024KB IO data (rszhi=xxxx is too big) please use a smaller value. </pre> </div></div> <p>Here is the test result on the current master branch with rszhi=2048:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== obdfilter-survey test 1a: Object Storage Targets survey == 00:16:37 (1358756197) + NETTYPE=tcp rszlo=2048 rszhi=2048 nobjlo=2 thrlo=2 nobjhi=1 thrhi=4 size=512 case=disk rslt_loc=/tmp targets="10.10.4.209:lustre-OST0000 10.10.4.209:lustre-OST0001 10.10.4.209:lustre-OST0002 10.10.4.209:lustre-OST0003 10.10.4.209:lustre-OST0004 10.10.4.209:lustre-OST0005 10.10.4.209:lustre-OST0006" /usr/bin/obdfilter-survey Test disk case support maximum 1024KB IO data (rszhi=2048 is too big) please use a smaller value. Resetting fail_loc on all nodes...done. PASS 1a (1s) </pre> </div></div> <p>Maloo report: <a href="https://maloo.whamcloud.com/test_sets/2d9fdba8-63a3-11e2-824c-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/2d9fdba8-63a3-11e2-824c-52540035b04c</a></p> <p>The <a href="http://review.whamcloud.com/1741" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/1741</a> patch is at best a workaround of the real problem I now see. The check should be done in the kernel instead of the script, since kernels may be configured differently, and even if the script is changed it should not be possible to cause a kernel oops. </p> <p>On the latest master branch, after commenting out the "rszhi" check from obdfilter-survey, running the test with rszhi="2048" hit the following assertion failure:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LustreError: 16216:0:(ofd_internal.h:524:ofd_info_init()) ASSERTION( info-&gt;fti_exp == ((void *)0) ) failed: </pre> </div></div> <p>Yu Jian, can you please retest manually, now that <a href="http://review.whamcloud.com/1741" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/1741</a> has landed. Even better would be to write a small sanity test that runs a very short test manually with a huge blocksize (e.g. 32MB) to check that this no longer LASSERTs.</p> <blockquote><p>Yu Jian, can you please retest manually, now that <a href="http://review.whamcloud.com/1741" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/1741</a> has landed.</p></blockquote> <p>With <a href="http://review.whamcloud.com/1741" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/1741</a> on master branch, running obdfilter-survey with rszhi="2048" will always get:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Test disk case support maximum 1024KB IO data (rszhi=2048 is too big) please use a smaller value. </pre> </div></div> <p>Now that the 4MB RPC patch <a href="http://review.whamcloud.com/4993" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4993</a> has been landed on master branch, I commented out the change of <a href="http://review.whamcloud.com/1741" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/1741</a> and ran the obdfilter-survey test with rszhi="2048", it passed without any assertion failures.</p> <p>Lustre master build: <a href="http://build.whamcloud.com/job/lustre-master/1269/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-master/1269/</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>+ NETTYPE=tcp rszlo=2048 rszhi=2048 nobjlo=2 thrlo=2 nobjhi=32 thrhi=32 size=512 case=disk rslt_loc=/tmp targets="10.10.4.209:lustre-OST0000 10.10.4.209:lustre-OST0001" /usr/bin/obdfilter-survey Fri Feb 22 04:52:38 PST 2013 Obdfilter-survey for case=disk from client-12vm1 ost 2 sz 1048576K rsz 2048K obj 4 thr 4 write 38.53 [ 14.00, 24.00] rewrite 44.82 [ 19.99, 32.00] read 6816.48 SHORT ost 2 sz 1048576K rsz 2048K obj 4 thr 8 write 42.19 [ 12.00, 31.99] rewrite 50.90 [ 18.00, 31.99] read 6419.65 SHORT ost 2 sz 1048576K rsz 2048K obj 4 thr 16 write 44.38 [ 6.00, 33.99] rewrite 54.78 [ 15.99, 33.99] read 6656.54 SHORT ost 2 sz 1048576K rsz 2048K obj 4 thr 32 write 39.14 [ 0.00, 40.00] rewrite 51.76 [ 6.00, 41.99] read 6373.85 SHORT ost 2 sz 1048576K rsz 2048K obj 4 thr 64 write 44.55 [ 0.00, 47.99] rewrite 55.59 [ 6.00, 51.99] read 5994.27 SHORT ost 2 sz 1048576K rsz 2048K obj 8 thr 8 write 35.73 [ 8.00, 27.99] rewrite 46.46 [ 17.99, 27.99] read 6606.66 SHORT ost 2 sz 1048576K rsz 2048K obj 8 thr 16 write 39.02 [ 8.00, 31.99] rewrite 50.27 [ 10.00, 37.99] read 6695.94 SHORT ost 2 sz 1048576K rsz 2048K obj 8 thr 32 write 43.88 [ 0.00, 39.99] rewrite 50.43 [ 0.00, 39.99] read 6350.12 SHORT ost 2 sz 1048576K rsz 2048K obj 8 thr 64 write 46.22 [ 0.00, 63.98] rewrite 55.43 [ 0.00, 59.98] read 6055.04 SHORT ost 2 sz 1048576K rsz 2048K obj 16 thr 16 write 35.33 [ 0.00, 31.99] rewrite 44.84 [ 6.00, 32.00] read 6597.55 SHORT ost 2 sz 1048576K rsz 2048K obj 16 thr 32 write 39.48 [ 4.00, 35.99] rewrite 44.83 [ 0.00, 35.99] read 6348.42 SHORT ost 2 sz 1048576K rsz 2048K obj 16 thr 64 write 43.35 [ 0.00, 61.98] rewrite 52.14 [ 0.00, 49.99] read 6024.13 SHORT ost 2 sz 1048576K rsz 2048K obj 32 thr 32 write 36.63 [ 0.00, 37.99] rewrite 47.39 [ 8.00, 39.99] read 6045.66 SHORT ost 2 sz 1048576K rsz 2048K obj 32 thr 64 write 40.83 [ 0.00, 49.98] rewrite 50.22 [ 4.00, 51.99] read 5918.83 SHORT ost 2 sz 1048576K rsz 2048K obj 64 thr 64 write 39.22 [ 0.00, 45.99] rewrite 49.17 [ 0.00, 51.99] read 6107.23 SHORT done! </pre> </div></div> <p>Maloo report: <a href="https://maloo.whamcloud.com/test_sets/24a44ca0-7cf3-11e2-a108-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/24a44ca0-7cf3-11e2-a108-52540035b04c</a></p> <p>I'll change the limit of 1024 to 4096 in obdfilter-survey.</p> <blockquote><p>Even better would be to write a small sanity test that runs a very short test manually with a huge blocksize (e.g. 32MB) to check that this no longer LASSERTs.</p></blockquote> <p>OK, will do.</p> <blockquote> <p>I'll change the limit of 1024 to 4096 in obdfilter-survey.</p></blockquote> <p>Note that this would cause LBUG if new obdfilter-survey script is run against an old OST... It should be conditional upon the remote Lustre version being used.</p> <p>Patch for Lustre b2_1 branch is in <a href="http://review.whamcloud.com/5715" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5715</a>.</p> <p>Patch for Lustre master branch is in <a href="http://review.whamcloud.com/5783" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5783</a>.</p> <p>Hmm, "Note that this would cause LBUG if new obdfilter-survey script is run against an old OST... It should be conditional upon the remote Lustre version being used."</p> <p>Thanks! I kicked off a regression test yesterday and found my OSS crashed! Ah,<br/> a 2.1.4 server, pretty close, right? <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/sad.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Just want to be sure... This problem only affect sanity test-180c, right? It<br/> will not cause production server to crash, will it?</p> <p>yes, this will only happen if you run obdfilter, which you don't really do in production.</p> <blockquote><p>Just want to be sure... This problem only affect sanity test-180c, right? It will not cause production server to crash, will it?</p></blockquote> <p>The sanity test 180c on Lustre b2_1 branch needs to be improved to interoperate with the servers (version &lt; 2.1.5, and 2.2.0 &lt;= version &lt; 2.4.0) which do not have the patch fixing the assertion failure.</p> <p>Patch for Lustre b2_1 branch to resolve the interop issues: <a href="http://review.whamcloud.com/5902" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5902</a></p> <p>Patch for b2_4 at <a href="http://review.whamcloud.com/6394" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/6394</a>.</p> <p>Patches were landed on Lustre b2_1, b2_4 and master branches.</p> <p>Added 2.5.0 FixVersion</p> Related LU-1431 Development Rank 1|hzvevz: Rank (Obsolete) 6053 Severity