https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1680 Lustre <p>Client LBUGS - This appears to be the same issue as ORI-726, but reporting separately just in case, as ORI is PPC issue.<br/> ------------</p> <p>Jul 25 17:27:40 ehyperion790 kernel: Lustre: DEBUG MARKER: == recovery-mds-scale test failover_ost: failover OST == 17:27:36 (1343262456)<br/> Jul 25 17:27:43 ehyperion790 kernel: LustreError: 13860:0:(cl_lock.c:1949:discard_cb()) ASSERTION( (!(page-&gt;cp_type == CPT_CACHEABLE) || (!PageDirty(cl_page_vmpage(env, page)))) ) failed:<br/> Jul 25 17:27:43 ehyperion790 kernel: LustreError: 13860:0:(cl_lock.c:1949:discard_cb()) ASSERTION( (!(page-&gt;cp_type == CPT_CACHEABLE) || (!PageDirty(cl_page_vmpage(env, page)))) ) failed:<br/> Jul 25 17:27:48 ehyperion790 kernel: LustreError: 13860:0:(cl_lock.c:1949:discard_cb()) LBUG<br/> Jul 25 17:27:48 ehyperion790 kernel: LustreError: 13860:0:(cl_lock.c:1949:discard_cb()) LBUG<br/> Jul 25 17:27:48 ehyperion790 kernel: Call Trace:<br/> Jul 25 17:27:51 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffffa034c905&gt;&#93;</span> libcfs_debug_dumpstack+0x55/0x80 <span class="error">&#91;libcfs&#93;</span><br/> Jul 25 17:27:54 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffffa034cf17&gt;&#93;</span> lbug_with_loc+0x47/0xb0 <span class="error">&#91;libcfs&#93;</span><br/> Jul 25 17:27:55 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffffa0519148&gt;&#93;</span> discard_cb+0x138/0x1f0 <span class="error">&#91;obdclass&#93;</span><br/> Jul 25 17:27:59 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffffa0516404&gt;&#93;</span> cl_page_gang_lookup+0x1f4/0x400 <span class="error">&#91;obdclass&#93;</span><br/> Jul 25 17:28:02 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffffa0519010&gt;&#93;</span> ? discard_cb+0x0/0x1f0 <span class="error">&#91;obdclass&#93;</span><br/> Jul 25 17:28:02 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffffa0519010&gt;&#93;</span> ? discard_cb+0x0/0x1f0 <span class="error">&#91;obdclass&#93;</span><br/> Jul 25 17:28:06 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffffa0518ece&gt;&#93;</span> cl_lock_discard_pages+0x11e/0x1f0 <span class="error">&#91;obdclass&#93;</span><br/> Jul 25 17:28:06 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffffa0832430&gt;&#93;</span> osc_lock_flush+0x110/0x200 <span class="error">&#91;osc&#93;</span><br/> Jul 25 17:28:10 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffffa0832579&gt;&#93;</span> osc_lock_cancel+0x59/0x1a0 <span class="error">&#91;osc&#93;</span><br/> Jul 25 17:28:12 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffffa0516c45&gt;&#93;</span> cl_lock_cancel0+0x75/0x160 <span class="error">&#91;obdclass&#93;</span><br/> Jul 25 17:28:14 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffffa05178ab&gt;&#93;</span> cl_lock_cancel+0x13b/0x140 <span class="error">&#91;obdclass&#93;</span><br/> Jul 25 17:28:17 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffffa083384a&gt;&#93;</span> osc_ldlm_blocking_ast+0x13a/0x380 <span class="error">&#91;osc&#93;</span><br/> Jul 25 17:28:19 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffffa062f480&gt;&#93;</span> ldlm_cancel_callback+0x60/0x100 <span class="error">&#91;ptlrpc&#93;</span><br/> Jul 25 17:28:23 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffffa063dc1b&gt;&#93;</span> ldlm_cli_cancel_local+0x7b/0x380 <span class="error">&#91;ptlrpc&#93;</span><br/> Jul 25 17:28:25 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffffa06404ff&gt;&#93;</span> ldlm_cli_cancel_list_local+0xef/0x1f0 <span class="error">&#91;ptlrpc&#93;</span><br/> Jul 25 17:28:26 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffffa064078a&gt;&#93;</span> ldlm_cancel_resource_local+0x18a/0x2a0 <span class="error">&#91;ptlrpc&#93;</span><br/> Jul 25 17:28:30 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffffa081a5cb&gt;&#93;</span> osc_destroy+0x12b/0x760 <span class="error">&#91;osc&#93;</span><br/> Jul 25 17:28:30 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffffa08bf63d&gt;&#93;</span> ? lov_set_add_req+0x2d/0x50 <span class="error">&#91;lov&#93;</span><br/> Jul 25 17:28:35 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffffa08b0235&gt;&#93;</span> lov_destroy+0x375/0xc40 <span class="error">&#91;lov&#93;</span><br/> Jul 25 17:28:36 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffff8115dd5c&gt;&#93;</span> ? transfer_objects+0x5c/0x80<br/> Jul 25 17:28:36 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffffa0691396&gt;&#93;</span> ? __req_capsule_get+0x176/0x750 <span class="error">&#91;ptlrpc&#93;</span><br/> Jul 25 17:28:36 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffffa09a8e8a&gt;&#93;</span> ll_objects_destroy+0x53a/0x1810 <span class="error">&#91;lustre&#93;</span><br/> Jul 25 17:28:36 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffffa0978811&gt;&#93;</span> ll_close_inode_openhandle+0x351/0x1050 <span class="error">&#91;lustre&#93;</span><br/> Jul 25 17:28:36 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffffa09796ba&gt;&#93;</span> ll_md_real_close+0x1aa/0x220 <span class="error">&#91;lustre&#93;</span><br/> Jul 25 17:28:36 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffffa097998b&gt;&#93;</span> ll_md_close+0x25b/0x760 <span class="error">&#91;lustre&#93;</span><br/> Jul 25 17:28:36 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffff814eec86&gt;&#93;</span> ? down_read+0x16/0x30<br/> Jul 25 17:28:36 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffffa0979fab&gt;&#93;</span> ll_file_release+0x11b/0x3e0 <span class="error">&#91;lustre&#93;</span><br/> Jul 25 17:28:36 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffff811781c5&gt;&#93;</span> __fput+0xf5/0x210<br/> Jul 25 17:28:36 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffff81178305&gt;&#93;</span> fput+0x25/0x30<br/> Jul 25 17:28:36 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffff81173d4d&gt;&#93;</span> filp_close+0x5d/0x90<br/> Jul 25 17:28:36 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffff8106c7bf&gt;&#93;</span> put_files_struct+0x7f/0xf0<br/> Jul 25 17:28:36 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffff8106c883&gt;&#93;</span> exit_files+0x53/0x70<br/> Jul 25 17:28:36 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffff8106e8f5&gt;&#93;</span> do_exit+0x185/0x870<br/> Jul 25 17:28:36 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffff8106f038&gt;&#93;</span> do_group_exit+0x58/0xd0<br/> Jul 25 17:28:36 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffff8106f0c7&gt;&#93;</span> sys_exit_group+0x17/0x20<br/> Jul 25 17:28:36 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffff8100b0f2&gt;&#93;</span> system_call_fastpath+0x16/0x1b<br/> Jul 25 17:28:36 ehyperion790 kernel:<br/> Jul 25 17:28:36 ehyperion790 kernel: Kernel panic - not syncing: LBUG<br/> -----------</p> <p>25 17:28:36 ehyperion790 kernel: Kernel panic - not syncing: LBUG<br/> Jul 25 17:28:36 ehyperion790 kernel: Call Trace:<br/> Jul 25 17:28:36 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffff814ec98a&gt;&#93;</span> ? panic+0x78/0x143<br/> Jul 25 17:28:36 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffffa034cf6b&gt;&#93;</span> ? lbug_with_loc+0x9b/0xb0 <span class="error">&#91;libcfs&#93;</span><br/> Jul 25 17:28:36 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffffa0519148&gt;&#93;</span> ? discard_cb+0x138/0x1f0 <span class="error">&#91;obdclass&#93;</span><br/> Jul 25 17:28:36 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffffa0516404&gt;&#93;</span> ? cl_page_gang_lookup+0x1f4/0x400 <span class="error">&#91;obdclass&#93;</span><br/> Jul 25 17:28:36 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffffa0519010&gt;&#93;</span> ? discard_cb+0x0/0x1f0 <span class="error">&#91;obdclass&#93;</span><br/> Jul 25 17:28:36 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffffa0519010&gt;&#93;</span> ? discard_cb+0x0/0x1f0 <span class="error">&#91;obdclass&#93;</span><br/> Jul 25 17:28:36 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffffa0518ece&gt;&#93;</span> ? cl_lock_discard_pages+0x11e/0x1f0 <span class="error">&#91;obdclass&#93;</span><br/> Jul 25 17:28:36 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffffa0832430&gt;&#93;</span> ? osc_lock_flush+0x110/0x200 <span class="error">&#91;osc&#93;</span><br/> Jul 25 17:28:36 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffffa0832579&gt;&#93;</span> ? osc_lock_cancel+0x59/0x1a0 <span class="error">&#91;osc&#93;</span><br/> Jul 25 17:28:36 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffffa0516c45&gt;&#93;</span> ? cl_lock_cancel0+0x75/0x160 <span class="error">&#91;obdclass&#93;</span><br/> Jul 25 17:28:36 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffffa05178ab&gt;&#93;</span> ? cl_lock_cancel+0x13b/0x140 <span class="error">&#91;obdclass&#93;</span><br/> Jul 25 17:28:36 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffffa083384a&gt;&#93;</span> ? osc_ldlm_blocking_ast+0x13a/0x380 <span class="error">&#91;osc&#93;</span><br/> Jul 25 17:28:36 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffffa062f480&gt;&#93;</span> ? ldlm_cancel_callback+0x60/0x100 <span class="error">&#91;ptlrpc&#93;</span><br/> Jul 25 17:28:36 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffffa063dc1b&gt;&#93;</span> ? ldlm_cli_cancel_local+0x7b/0x380 <span class="error">&#91;ptlrpc&#93;</span><br/> Jul 25 17:28:36 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffffa06404ff&gt;&#93;</span> ? ldlm_cli_cancel_list_local+0xef/0x1f0 <span class="error">&#91;ptlrpc&#93;</span><br/> Jul 25 17:28:36 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffffa064078a&gt;&#93;</span> ? ldlm_cancel_resource_local+0x18a/0x2a0 <span class="error">&#91;ptlrpc&#93;</span><br/> Jul 25 17:28:36 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffffa081a5cb&gt;&#93;</span> ? osc_destroy+0x12b/0x760 <span class="error">&#91;osc&#93;</span><br/> Jul 25 17:28:36 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffffa08bf63d&gt;&#93;</span> ? lov_set_add_req+0x2d/0x50 <span class="error">&#91;lov&#93;</span><br/> Jul 25 17:28:36 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffffa08b0235&gt;&#93;</span> ? lov_destroy+0x375/0xc40 <span class="error">&#91;lov&#93;</span><br/> Jul 25 17:28:36 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffff8115dd5c&gt;&#93;</span> ? transfer_objects+0x5c/0x80<br/> Jul 25 17:28:36 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffffa0691396&gt;&#93;</span> ? __req_capsule_get+0x176/0x750 <span class="error">&#91;ptlrpc&#93;</span><br/> Jul 25 17:28:36 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffffa09a8e8a&gt;&#93;</span> ? ll_objects_destroy+0x53a/0x1810 <span class="error">&#91;lustre&#93;</span><br/> Jul 25 17:28:36 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffffa0978811&gt;&#93;</span> ? ll_close_inode_openhandle+0x351/0x1050 <span class="error">&#91;lustre&#93;</span><br/> Jul 25 17:28:36 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffffa09796ba&gt;&#93;</span> ? ll_md_real_close+0x1aa/0x220 <span class="error">&#91;lustre&#93;</span><br/> Jul 25 17:28:36 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffffa097998b&gt;&#93;</span> ? ll_md_close+0x25b/0x760 <span class="error">&#91;lustre&#93;</span><br/> Jul 25 17:28:36 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffff814eec86&gt;&#93;</span> ? down_read+0x16/0x30<br/> Jul 25 17:28:36 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffffa0979fab&gt;&#93;</span> ? ll_file_release+0x11b/0x3e0 <span class="error">&#91;lustre&#93;</span><br/> Jul 25 17:28:36 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffff811781c5&gt;&#93;</span> ? __fput+0xf5/0x210<br/> Jul 25 17:28:36 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffff81178305&gt;&#93;</span> ? fput+0x25/0x30<br/> Jul 25 17:28:36 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffff81173d4d&gt;&#93;</span> ? filp_close+0x5d/0x90<br/> Jul 25 17:28:36 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffff8106c7bf&gt;&#93;</span> ? put_files_struct+0x7f/0xf0<br/> Jul 25 17:28:36 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffff8106c883&gt;&#93;</span> ? exit_files+0x53/0x70<br/> Jul 25 17:28:36 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffff8106e8f5&gt;&#93;</span> ? do_exit+0x185/0x870<br/> Jul 25 17:28:36 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffff8106f038&gt;&#93;</span> ? do_group_exit+0x58/0xd0<br/> Jul 25 17:28:36 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffff8106f0c7&gt;&#93;</span> ? sys_exit_group+0x17/0x20<br/> Jul 25 17:28:36 ehyperion790 kernel: <span class="error">&#91;&lt;ffffffff8100b0f2&gt;&#93;</span> ? system_call_fastpath+0x16/0x1b</p> LLNL Hyperion, running the OST failover portion of recovery-scale - failure occurs close to the start of the test. LU-1680

LBUG cl_lock.c:1949:discard_cb()) (ORI-726)

Bug Blocker Resolved Duplicate Jinshan Xiong Cliff White Thu, 26 Jul 2012 13:32:28 +0000 Sat, 6 Dec 2014 02:12:27 +0000 Wed, 5 Sep 2012 13:46:02 +0000 Lustre 2.3.0 0 5 <p>Recently I found a problem at <a href="https://jira.whamcloud.com/browse/LU-1442" title="File corrupt with 1MiB-aligned 4k regions of zeros" class="issue-link" data-issue-key="LU-1442"><del>LU-1442</del></a> where a page was wrongly marked as dirty but not in osc writing cache. That will definitely cause this kind of issue because osc_cache_writeback_range() won't clean up the whole lock region.</p> <p>Based on the situation that there is no enough log I'd like to try that patch first. The patch is located at: <a href="http://review.whamcloud.com/3447" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/3447</a></p> <p>This patch is landed to master and so should be in the next tag</p> <p>Hi Jinshan, What's the status of this? Is <a href="http://review.whamcloud.com/3447" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/3447</a> all that is needed to fix the assertion?</p> <p>Hi Prakash, yes, I think so.</p> <p>ok so let's mark this as a duplicate of <a href="https://jira.whamcloud.com/browse/LU-1442" title="File corrupt with 1MiB-aligned 4k regions of zeros" class="issue-link" data-issue-key="LU-1442"><del>LU-1442</del></a> then and reopen if that proves incorrect</p> <p>Correct me if I'm wrong, but I think <a href="https://jira.whamcloud.com/browse/LU-1442" title="File corrupt with 1MiB-aligned 4k regions of zeros" class="issue-link" data-issue-key="LU-1442"><del>LU-1442</del></a> is a separate issue. I thought Jinshan discovered the root cause of this issue when investigating <a href="https://jira.whamcloud.com/browse/LU-1442" title="File corrupt with 1MiB-aligned 4k regions of zeros" class="issue-link" data-issue-key="LU-1442"><del>LU-1442</del></a>, not that they are the same bug?</p> <p><a href="https://jira.whamcloud.com/browse/LU-1442" title="File corrupt with 1MiB-aligned 4k regions of zeros" class="issue-link" data-issue-key="LU-1442"><del>LU-1442</del></a> can definitely cause this issue, let me explain here:<br/> 1. from the description of lu-1442, one page can be wrongly marked as Dirty but not be put into OSC's write cache;<br/> 2. when the lock is canceled, we only cleanup the region of osc write cache covered by the lock, however, the page is not in osc write cache as I stated in 1;<br/> 3. when the page is discarded, we verify that there is no dirty page in the lock region by checking vmpage dirty flag. This will hit assertion.</p> <p>As a conclusion, I think these are different symptoms of the same issue.</p> <p>Thanks for the detailed explanation!</p> <p>Reopening. Somehow this ticket and ORI-726 were both marked "resolved" as duplicates of each other. One needs to stay open until the issue is resolved. <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Oh, I see, this should be resolved as duplicate of <a href="https://jira.whamcloud.com/browse/LU-1442" title="File corrupt with 1MiB-aligned 4k regions of zeros" class="issue-link" data-issue-key="LU-1442"><del>LU-1442</del></a>. I'll do that.</p> <p>Duplicate of <a href="https://jira.whamcloud.com/browse/LU-1442" title="File corrupt with 1MiB-aligned 4k regions of zeros" class="issue-link" data-issue-key="LU-1442"><del>LU-1442</del></a>.</p> <p>Sorry, my reading of the summary was too quick. I thought this was just a renamed ORI-726 so the patch could land to master, and that it was definitely related to PPC. I've removed the ppc label. </p> Duplicate LU-1442 Development Rank 1|hzvenj: Rank (Obsolete) 6015 Severity