https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2557 Lustre <p>The "teardown failure" is handled with an assertion failure.</p> <p>The stack trace from the assertion failure call:<br/> <span class="error">&#91;&lt;ffffffffa06a1e61&gt;&#93;</span> osc_page_delete+0x2d1/0x2e0 <span class="error">&#91;osc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa02d33fd&gt;&#93;</span> cl_page_delete0+0xcd/0x4f0 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa02d3862&gt;&#93;</span> cl_page_delete+0x42/0x120 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0826e9d&gt;&#93;</span> ll_invalidatepage+0x8d/0x170 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa081e264&gt;&#93;</span> ll_page_mkwrite+0x7c4/0x840 <span class="error">&#91;lustre&#93;</span><br/> <span class="error">&#91;&lt;ffffffff810ef21b&gt;&#93;</span> __do_fault+0xbb/0x4c0<br/> <span class="error">&#91;&lt;ffffffff810f2c1b&gt;&#93;</span> handle_mm_fault+0x1db/0xe50<br/> <span class="error">&#91;&lt;ffffffff810289f7&gt;&#93;</span> do_page_fault+0x147/0x2c0<br/> <span class="error">&#91;&lt;ffffffff812d90df&gt;&#93;</span> page_fault+0x1f/0x30</p> <p>And extract from pid 617 Lustre traces:</p> <p>00000008:00020000:21.0:1353473247.507160:0:617:0:(osc_page.c:411:osc_page_delete()) page@ffff88080aa2f180<span class="error">&#91;1 ffff8801ff1e1508:51 ^ffff88080aa2f240_(null) 4 0 1 (null) (null) 0x0&#93;</span><br/> 00000008:00020000:21.0:1353473247.520125:0:617:0:(osc_page.c:411:osc_page_delete()) vvp-page@ffff880809658be0(0:0:0) vm@ffffea001bfd54d8 e000000000000e3 8:0 0 51 lru<br/> 00000008:00020000:21.0:1353473247.532044:0:617:0:(osc_page.c:411:osc_page_delete()) lov-page@ffff88080b1c4b58<br/> 00000008:00020000:21.0:1353473247.539101:0:617:0:(osc_page.c:411:osc_page_delete()) osc-page@ffff8808176944b0: 1&lt; 0x845fed 258 0 + - &gt; 2&lt; 208896 0 4096 0x0 0x520 | (null) ffff88041c1a6778 ffff8802072be888 &gt; 3&lt; + ffff8801fd445100 0 0 0 &gt; 4&lt; 0 0 8 31752192 - | - - + - &gt; 5&lt; - - + - | 0 - | 17 - -&gt;<br/> 00000008:00020000:21.0:1353473247.562323:0:617:0:(osc_page.c:411:osc_page_delete()) end page@ffff88080aa2f240<br/> 00000008:00020000:21.0:1353473247.569375:0:617:0:(osc_page.c:411:osc_page_delete()) Trying to teardown failed: -16<br/> 00000008:00040000:21.0:1353473247.576859:0:617:0:(osc_page.c:412:osc_page_delete()) ASSERTION( 0 ) failed: <br/> 00000008:00040000:21.0:1353473247.583736:0:617:0:(osc_page.c:412:osc_page_delete()) LBUG</p> <p>Customer commentary:<br/> &gt; The symptom is similar to <a href="https://jira.whamcloud.com/browse/LU-1442" title="File corrupt with 1MiB-aligned 4k regions of zeros" class="issue-link" data-issue-key="LU-1442"><del>LU-1442</del></a>/<a href="https://jira.whamcloud.com/browse/LU-1680" title="LBUG cl_lock.c:1949:discard_cb()) (ORI-726)" class="issue-link" data-issue-key="LU-1680"><del>LU-1680</del></a> but we have the patch and apparently the problem still exists.</p> <p>A dump is in:<br/> ftp.cray.com:/outbound/791555-osc_cache_writeback_range-assert.tar.bz2<br/> More are available on request.</p> <p>Reference: <a href="https://jira.whamcloud.com/browse/LU-1030" title="new IO engine" class="issue-link" data-issue-key="LU-1030"><del>LU-1030</del></a> &#8211; apparent source of problem</p> Lustre client 2.3 rc6, SLES11 2.6.32.59 or SLES11 3.0.38<br/> Lustre server 2.3 rc6, SLES11 2.6.32.59(direct-attached) or centos 2.6.32(external server) LU-2557

osc_page_delete()) Trying to teardown failed: -16 (EBUSY)

Bug Blocker Resolved Duplicate Hongchao Zhang Bruce Korb LB Wed, 2 Jan 2013 16:38:20 +0000 Mon, 25 Feb 2013 03:52:30 +0000 Mon, 25 Feb 2013 03:52:30 +0000 Lustre 2.3.0 Lustre 2.4.0 0 7 <p>The attached "clio disable CILR_PEEK optimization" patch does <em>not</em> seem to work. Vitaly's analysis of the issues falling out from <a href="https://jira.whamcloud.com/browse/LU-1030" title="new IO engine" class="issue-link" data-issue-key="LU-1030"><del>LU-1030</del></a> led him to believe that the peek optimization was the source of the trouble. The result is a different symptom.</p> <p>The new symptom is a hang. Two threads have the following stack trace and the data for one of the stack traces appears to have been corrupted. Upon request, I'll add those details, but meanwhile any help at all in understanding the root cause of the <a href="https://jira.whamcloud.com/browse/LU-1030" title="new IO engine" class="issue-link" data-issue-key="LU-1030"><del>LU-1030</del></a> patch issues would be <b>MOST</b> welcome.</p> <p>Thank you.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedHeader panelHeader" style="border-bottom-width: 1px;"><b>Stack Trace of Hung Process</b></div><div class="preformattedContent panelContent"> <pre>PID: 18325 TASK: ffff8803fe1381c0 CPU: 1 COMMAND: "fsx-linux-aio" #0 [ffff8803c51dd538] schedule at ffffffff8141e405 #1 [ffff8803c51dd610] schedule_timeout at ffffffff8141ee14 #2 [ffff8803c51dd690] cfs_waitq_timedwait at ffffffffa01a66c1 [libcfs] #3 [ffff8803c51dd6a0] osc_extent_wait at ffffffffa0725694 [osc] #4 [ffff8803c51dd810] osc_cache_wait_range at ffffffffa0726167 [osc] #5 [ffff8803c51dd9a0] osc_io_fsync_end at ffffffffa0717a79 [osc] #6 [ffff8803c51dd9e0] cl_io_end at ffffffffa03048b2 [obdclass] #7 [ffff8803c51dda10] lov_io_end_wrapper at ffffffffa07c26f9 [lov] #8 [ffff8803c51dda30] lov_io_fsync_end at ffffffffa07c407b [lov] #9 [ffff8803c51dda60] cl_io_end at ffffffffa03048b2 [obdclass] #10 [ffff8803c51dda90] cl_io_loop at ffffffffa0308c4a [obdclass] #11 [ffff8803c51ddac0] cl_sync_file_range at ffffffffa08aacbb [lustre] #12 [ffff8803c51ddb30] ll_writepages at ffffffffa08d0164 [lustre] #13 [ffff8803c51ddb70] do_writepages at ffffffff810cd631 #14 [ffff8803c51ddb80] __filemap_fdatawrite_range at ffffffff810c5b65 #15 [ffff8803c51ddbe0] do_sync_mapping_range at ffffffff8112ea01 #16 [ffff8803c51ddc10] __generic_file_aio_write at ffffffff810c6d6c #17 [ffff8803c51ddcc0] generic_file_aio_write at ffffffff810c6e42 #18 [ffff8803c51ddd00] vvp_io_write_start at ffffffffa08fd4f6 [lustre] #19 [ffff8803c51ddd40] cl_io_start at ffffffffa0304a1a [obdclass] #20 [ffff8803c51ddd70] cl_io_loop at ffffffffa0308c3c [obdclass] #21 [ffff8803c51ddda0] ll_file_io_generic at ffffffffa08a9cc5 [lustre] #22 [ffff8803c51dde10] ll_file_aio_write at ffffffffa08aa01b [lustre] #23 [ffff8803c51dde60] aio_rw_vect_retry at ffffffff811426d4 #24 [ffff8803c51dde90] aio_run_iocb at ffffffff81143ffa #25 [ffff8803c51ddec0] sys_io_submit at ffffffff81144e38 #26 [ffff8803c51ddf80] system_call_fastpath at ffffffff8100305b RIP: 00007fa64d7d06f7 RSP: 00007fff6d99f318 RFLAGS: 00010206 RAX: 00000000000000d1 RBX: ffffffff8100305b RCX: 000000000000002d RDX: 00007fff6d99f2c0 RSI: 0000000000000001 RDI: 00007fa64dbe6000 RBP: 0000000000006000 R8: 0000000000039000 R9: 0000000000000003 R10: 262b0c2b4d2b962b R11: 0000000000000202 R12: 00007fff6d99f770 R13: 0000000000404658 R14: 000000000003f000 R15: 000000000003f000 ORIG_RAX: 00000000000000d1 CS: 0033 SS: 002b</pre> </div></div> <p>this ticket could be related to the "write_one_page" call in ll_page_mkwrite0 if there is no lock to protect the write.<br/> ...<br/> if (result == 0 || result == -ENODATA) {<br/> ...<br/> if (vmpage-&gt;mapping == NULL) </p> { ... } <p> else if (result == -ENODATA) {<br/> write_one_page(vmpage, 1);<br/> lock_page(vmpage);<br/> if (vmpage-&gt;mapping != NULL) </p> { ll_invalidate_page(vmpage); LASSERT(vmpage-&gt;mapping == NULL); } <p> unlock_page(vmpage);<br/> }<br/> ...<br/> }<br/> ...</p> <p>if there is an extent covering this page is waiting to be flushed back to OST, "write_one_page" will put the page into the pending list.</p> <p>Hi Bruce, could you please try to test it without the "write_one_page" call? Thanks.</p> <p>Meaning that the call to "write_one_page(vmpage, 1)" did nothing useful, so just eliminate it entirely? OK...</p> <p>BTW, "noformat" inside of curly braces before and after a code snippet makes the code much more readable.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedHeader panelHeader" style="border-bottom-width: 1px;"><b>code snippet from ll_page_mkwrite0</b></div><div class="preformattedContent panelContent"> <pre>if (result == 0 || result == -ENODATA) { ... if (vmpage-&amp;gt;mapping == NULL) { ... } else if (result == -ENODATA) { write_one_page(vmpage, 1); lock_page(vmpage); if (vmpage-&amp;gt;mapping != NULL) { ll_invalidate_page(vmpage); LASSERT(vmpage-&amp;gt;mapping == NULL); } unlock_page(vmpage); } ... }</pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedHeader panelHeader" style="border-bottom-width: 1px;"><b>longer snippet from ll_page_mkwrite0</b></div><div class="preformattedContent panelContent"> <pre> if (result == 0 || result == -ENODATA) { lock_page(vmpage); if (vmpage-&gt;mapping == NULL) { unlock_page(vmpage); if (result == 0) result = -ENODATA; } else if (result == -ENODATA) { write_one_page(vmpage, 1); lock_page(vmpage); if (vmpage-&gt;mapping != NULL) { ll_invalidate_page(vmpage); LASSERT(vmpage-&gt;mapping == NULL); } unlock_page(vmpage);</pre> </div></div> <p>OK, I'm not understanding how the write_one_page is not within a "lock_page".<br/> In fact, it looks to me like the lock_page() call that follows it is wrong.<br/> I'm guessing that the thread is self-deadlocked.</p> <p>do not lock a page that is already locked</p> <p>the page is locked before calling "write_one_lock", and it will unlock the page after it exits, the modified codes like</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedHeader panelHeader" style="border-bottom-width: 1px;"><b>longer snippet from ll_page_mkwrite0</b></div><div class="preformattedContent panelContent"> <pre> if (result == 0 || result == -ENODATA) { lock_page(vmpage); if (vmpage-&gt;mapping == NULL) { unlock_page(vmpage); if (result == 0) result = -ENODATA; } else if (result == -ENODATA) { if (vmpage-&gt;mapping != NULL) { ll_invalidate_page(vmpage); LASSERT(vmpage-&gt;mapping == NULL); } unlock_page(vmpage);</pre> </div></div> <p>Sorry &#8211; should have read code before commenting. I knew Linus strongly favored locking and unlocking in the same function, ...</p> <p>Anyway, according to Cray, removing the "write_one_page" call still hangs:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Wally Wang updated LELUS-103: ----------------------------- Attachment: lelus103-sp1-nowrite-one-page.tar.bz2 After removing the write_one_page(), it still hangs. See attachment</pre> </div></div> <p>Hi, is the previouse patch(removing CILR_PEEK) still applied with removing the write_one_page?</p> <p>CF: <a href="https://jira.whamcloud.com/browse/LU-2720" title="osc_page_delete()) ASSERTION(0) failed" class="issue-link" data-issue-key="LU-2720"><del>LU-2720</del></a></p> <blockquote><p>Vitaly Fertman added a comment - 31/Jan/13 7:53 AM</p> <p><a href="http://review.whamcloud.com/5222" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5222</a></p></blockquote> <p>duplicate of <a href="https://jira.whamcloud.com/browse/LU-2720" title="osc_page_delete()) ASSERTION(0) failed" class="issue-link" data-issue-key="LU-2720"><del>LU-2720</del></a></p> Related LU-2720 Development Rank 1|hzvegf: Rank (Obsolete) 5983 Severity