https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2246 Lustre <p>This issue was created by maloo for sarah &lt;sarah@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://maloo.whamcloud.com/test_sets/34cdc80e-21c2-11e2-b552-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/34cdc80e-21c2-11e2-b552-52540035b04c</a>.</p> <p>The sub-test test_132 failed with the following error:</p> <blockquote> <p>test failed to respond and timed out</p></blockquote> <p>From MDS console log:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00:20:08:Lustre: lustre-MDT0000: haven't heard from client 5be596f3-4c31-36f1-0f15-cf978ff44af7 (at 192.168.4.23@o2ib) in 55 seconds. I think it's dead, and I am evicting it. exp ffff88030ddd2000, cur 1351495207 expire 1351495177 last 1351495152 00:20:08:LustreError: 25221:0:(mdd_device.c:1426:md_ucred()) ASSERTION( env-&gt;le_ses != ((void *)0) ) failed: 00:20:08:LustreError: 25221:0:(mdd_device.c:1426:md_ucred()) LBUG 00:20:08:Pid: 25221, comm: ll_evictor 00:20:08: 00:20:08:Call Trace: 00:20:08: [&lt;ffffffffa03f2905&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] 00:20:08: [&lt;ffffffffa03f2f17&gt;] lbug_with_loc+0x47/0xb0 [libcfs] 00:20:08: [&lt;ffffffffa0be7c2c&gt;] md_ucred+0x5c/0x60 [mdd] 00:20:08: [&lt;ffffffffa0bd03a6&gt;] mdd_xattr_sanity_check+0x36/0x1f0 [mdd] 00:20:08: [&lt;ffffffffa0bd71db&gt;] mdd_xattr_set+0x17b/0x620 [mdd] 00:20:08: [&lt;ffffffffa0bf0426&gt;] ? mdd_read_unlock+0x26/0x30 [mdd] 00:20:08: [&lt;ffffffffa0bd513b&gt;] ? mdd_xattr_get+0x13b/0x340 [mdd] 00:20:08: [&lt;ffffffffa0c4de73&gt;] mdt_som_attr_set+0x1b3/0x440 [mdt] 00:20:08: [&lt;ffffffffa0c4e24c&gt;] mdt_ioepoch_close_on_eviction+0x14c/0x170 [mdt] 00:20:08: [&lt;ffffffffa0f48e89&gt;] ? osd_key_init+0x119/0x680 [osd_ldiskfs] 00:20:08: [&lt;ffffffffa0c4eccb&gt;] mdt_ioepoch_close+0x2ab/0x3d0 [mdt] 00:20:08: [&lt;ffffffffa0c4f272&gt;] mdt_mfd_close+0x482/0x700 [mdt] 00:20:08: [&lt;ffffffffa0c1e01e&gt;] mdt_obd_disconnect+0x3ae/0x4f0 [mdt] 00:20:08: [&lt;ffffffffa056ed88&gt;] class_fail_export+0x248/0x580 [obdclass] 00:20:08: [&lt;ffffffffa0765e69&gt;] ping_evictor_main+0x249/0x640 [ptlrpc] 00:20:08: [&lt;ffffffff81060250&gt;] ? default_wake_function+0x0/0x20 00:20:08: [&lt;ffffffffa0765c20&gt;] ? ping_evictor_main+0x0/0x640 [ptlrpc] 00:20:08: [&lt;ffffffff8100c14a&gt;] child_rip+0xa/0x20 00:20:08: [&lt;ffffffffa0765c20&gt;] ? ping_evictor_main+0x0/0x640 [ptlrpc] 00:20:08: [&lt;ffffffffa0765c20&gt;] ? ping_evictor_main+0x0/0x640 [ptlrpc] 00:20:08: [&lt;ffffffff8100c140&gt;] ? child_rip+0x0/0x20 00:20:08: 00:20:08:Kernel panic - not syncing: LBUG 00:20:08:Pid: 25221, comm: ll_evictor Not tainted 2.6.32-279.5.1.el6_lustre.x86_64 #1 00:20:08:Call Trace: 00:20:08: [&lt;ffffffff814fd58a&gt;] ? panic+0xa0/0x168 00:20:08: [&lt;ffffffffa03f2f6b&gt;] ? lbug_with_loc+0x9b/0xb0 [libcfs] 00:20:08: [&lt;ffffffffa0be7c2c&gt;] ? md_ucred+0x5c/0x60 [mdd] 00:20:08: [&lt;ffffffffa0bd03a6&gt;] ? mdd_xattr_sanity_check+0x36/0x1f0 [mdd] 00:20:08: [&lt;ffffffffa0bd71db&gt;] ? mdd_xattr_set+0x17b/0x620 [mdd] 00:20:08: [&lt;ffffffffa0bf0426&gt;] ? mdd_read_unlock+0x26/0x30 [mdd] 00:20:08: [&lt;ffffffffa0bd513b&gt;] ? mdd_xattr_get+0x13b/0x340 [mdd] 00:20:08: [&lt;ffffffffa0c4de73&gt;] ? mdt_som_attr_set+0x1b3/0x440 [mdt] 00:20:08: [&lt;ffffffffa0c4e24c&gt;] ? mdt_ioepoch_close_on_eviction+0x14c/0x170 [mdt] 00:20:08: [&lt;ffffffffa0f48e89&gt;] ? osd_key_init+0x119/0x680 [osd_ldiskfs] 00:20:08: [&lt;ffffffffa0c4eccb&gt;] ? mdt_ioepoch_close+0x2ab/0x3d0 [mdt] 00:20:08: [&lt;ffffffffa0c4f272&gt;] ? mdt_mfd_close+0x482/0x700 [mdt] 00:20:08: [&lt;ffffffffa0c1e01e&gt;] ? mdt_obd_disconnect+0x3ae/0x4f0 [mdt] 00:20:08: [&lt;ffffffffa056ed88&gt;] ? class_fail_export+0x248/0x580 [obdclass] 00:20:08: [&lt;ffffffffa0765e69&gt;] ? ping_evictor_main+0x249/0x640 [ptlrpc] 00:20:08: [&lt;ffffffff81060250&gt;] ? default_wake_function+0x0/0x20 00:20:08: [&lt;ffffffffa0765c20&gt;] ? ping_evictor_main+0x0/0x640 [ptlrpc] 00:20:08: [&lt;ffffffff8100c14a&gt;] ? child_rip+0xa/0x20 00:20:08: [&lt;ffffffffa0765c20&gt;] ? ping_evictor_main+0x0/0x640 [ptlrpc] 00:20:08: [&lt;ffffffffa0765c20&gt;] ? ping_evictor_main+0x0/0x640 [ptlrpc] 00:20:08: [&lt;ffffffff8100c140&gt;] ? child_rip+0x0/0x20 </pre> </div></div> Both server and client are RHEL6 LU-2246

failure on sanity.sh test_132: ASSERTION( env->le_ses != ((void *)0) ) failed

Bug Major Resolved Fixed Hongchao Zhang Maloo Mon, 29 Oct 2012 13:50:58 +0000 Thu, 7 Jan 2016 23:44:28 +0000 Thu, 7 Jan 2016 23:44:28 +0000 Lustre 2.4.0 0 7 <p>this issue is related to SOM, and it can be reproduced easily, it will occur if a client is evicted while SOM is enabled,<br/> for ping_evictor threads have no lu_context "le_ses", which is designed for per-request.</p> <p>in this case, the client encountered ASSERTION of <a href="https://jira.whamcloud.com/browse/LU-1527" title="Assertion at: (cl_lock.c:2211:cl_lock_hold_add()) ASSERTION( lock-&gt;cll_state != CLS_FREEING )" class="issue-link" data-issue-key="LU-1527"><del>LU-1527</del></a> in test_132 and was evicted by MDT, which caused this ASSERTION.</p> <p>the initial patch is under test and will be attached soon</p> <p>the patch is tracked at <a href="http://review.whamcloud.com/#change,4512" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,4512</a></p> <p>this problem is related to SOM, which is not a mature feature, and the patch fixes some bugs in SOM, <br/> here are some notes about the issues fixed by the patch,</p> <p>1, the lu_env could not contain lu_context related to specific request (lu_env-&gt;le_ses), say, for eviction case,<br/> then the operations related to SOM should take it into account.<br/> 2, the AU (ll_som_update) rpc call could encounter an destroyed object(marked with LU_OBJECT_HEARD_BANSHEE) for the object<br/> was unlinked/closed but the a mfd is still held in MDT to wait the AU rpc, which cause deadlock for the AU rpc can't<br/> get the object by lu_object_find_at for it was marked as dead object.</p> <p>this issue can be dropped as a blocker, for it is caused by blocker <a href="https://jira.whamcloud.com/browse/LU-1527" title="Assertion at: (cl_lock.c:2211:cl_lock_hold_add()) ASSERTION( lock-&gt;cll_state != CLS_FREEING )" class="issue-link" data-issue-key="LU-1527"><del>LU-1527</del></a> and it is only related to unstable SOM feature</p> <p>the patch is updated again, and it tries to fix the following problems related to SOM, the ASSERTION issue<br/> "env-&gt;le_ses != ((void *)0))" will be fixed by <a href="http://review.whamcloud.com/2733" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/2733</a></p> <p>1, if there is no LSM, ll_som_update won't need to update SOM attributes on MDS (test_206 in sanity.sh)</p> <p>2, if the file isn't a regular one, ll_setattr_raw should not open the file (MF_EPOCH_OPEN), for it will leave the set_attr<br/> request and the corresponding obd_import &amp; obd_device won't be released (there are several MDC obd_device with ST state)</p> <p>3, mdt_som_au_close could be called during eviction, then there is no ptlrpc_request with it.</p> <p>4, during closing a file, MDT should not require the client to send AU if the file has been unlinked.</p> <p>Patch was merged.<br/> ~ jfc.</p> Related LU-1403 LU-1942 Development Rank 1|hzvazz: Rank (Obsolete) 5318 Severity