https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-10678 Lustre <p>Soak MDT was in normal operation. Sudden LBUG</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Feb 16 09:28:39 soak-8 kernel: LustreError: 2688:0:(osd_handler.c:2353:osd_read_lock()) ASSERTION( obj-&gt;oo_owner == ((void *)0) ) failed: Feb 16 09:28:39 soak-8 kernel: LustreError: 2688:0:(osd_handler.c:2353:osd_read_lock()) LBUG Feb 16 09:28:39 soak-8 kernel: Pid: 2688, comm: mdt00_028 Feb 16 09:28:39 soak-8 kernel: #012Call Trace: Feb 16 09:28:39 soak-8 kernel: [&lt;ffffffffc0dbc7ae&gt;] libcfs_call_trace+0x4e/0x60 [libcfs] Feb 16 09:28:39 soak-8 kernel: [&lt;ffffffffc0dbc83c&gt;] lbug_with_loc+0x4c/0xb0 [libcfs] Feb 16 09:28:39 soak-8 kernel: [&lt;ffffffffc140599a&gt;] osd_read_lock+0xda/0xe0 [osd_ldiskfs] Feb 16 09:28:40 soak-8 kernel: [&lt;ffffffffc1691287&gt;] lod_read_lock+0x37/0xd0 [lod] Feb 16 09:28:40 soak-8 kernel: [&lt;ffffffffc17125c7&gt;] mdd_read_lock+0x37/0xd0 [mdd] Feb 16 09:28:40 soak-8 kernel: [&lt;ffffffffc1715dcc&gt;] mdd_xattr_get+0x6c/0x390 [mdd] Feb 16 09:28:40 soak-8 kernel: [&lt;ffffffffc1593c3f&gt;] mdt_pack_acl2body+0x1af/0x800 [mdt] Feb 16 09:28:40 soak-8 kernel: [&lt;ffffffffc15beaf9&gt;] mdt_finish_open+0x289/0x690 [mdt] Feb 16 09:28:40 soak-8 kernel: [&lt;ffffffffc15c120b&gt;] mdt_reint_open+0x230b/0x3260 [mdt] Feb 16 09:28:40 soak-8 kernel: [&lt;ffffffffc0f27d2e&gt;] ? upcall_cache_get_entry+0x20e/0x8f0 [obdclass] Feb 16 09:28:40 soak-8 kernel: [&lt;ffffffffc15a4b43&gt;] ? ucred_set_jobid+0x53/0x70 [mdt] Feb 16 09:28:40 soak-8 kernel: [&lt;ffffffffc15b5400&gt;] mdt_reint_rec+0x80/0x210 [mdt] Feb 16 09:28:40 soak-8 kernel: [&lt;ffffffffc1594f8b&gt;] mdt_reint_internal+0x5fb/0x9c0 [mdt] Feb 16 09:28:40 soak-8 kernel: [&lt;ffffffffc15a1437&gt;] mdt_intent_reint+0x157/0x420 [mdt] Feb 16 09:28:40 soak-8 kernel: [&lt;ffffffffc15980b2&gt;] mdt_intent_opc+0x442/0xad0 [mdt] Feb 16 09:28:41 soak-8 kernel: [&lt;ffffffffc1144470&gt;] ? lustre_swab_ldlm_intent+0x0/0x20 [ptlrpc] Feb 16 09:28:41 soak-8 kernel: [&lt;ffffffffc159fc63&gt;] mdt_intent_policy+0x1a3/0x360 [mdt] Feb 16 09:28:41 soak-8 kernel: [&lt;ffffffffc10f4202&gt;] ldlm_lock_enqueue+0x382/0x8f0 [ptlrpc] Feb 16 09:28:41 soak-8 kernel: [&lt;ffffffffc111c753&gt;] ldlm_handle_enqueue0+0x8f3/0x13e0 [ptlrpc] Feb 16 09:28:41 soak-8 kernel: [&lt;ffffffffc11444f0&gt;] ? lustre_swab_ldlm_request+0x0/0x30 [ptlrpc] Feb 16 09:28:41 soak-8 kernel: [&lt;ffffffffc11a2202&gt;] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 16 09:28:41 soak-8 kernel: [&lt;ffffffffc11aa405&gt;] tgt_request_handle+0x925/0x13b0 [ptlrpc] Feb 16 09:28:41 soak-8 kernel: [&lt;ffffffffc114e58e&gt;] ptlrpc_server_handle_request+0x24e/0xab0 [ptlrpc] Feb 16 09:28:41 soak-8 kernel: [&lt;ffffffffc114b448&gt;] ? ptlrpc_wait_event+0x98/0x340 [ptlrpc] Feb 16 09:28:41 soak-8 kernel: [&lt;ffffffff810c6440&gt;] ? default_wake_function+0x0/0x20 Feb 16 09:28:41 soak-8 kernel: [&lt;ffffffffc1151d42&gt;] ptlrpc_main+0xa92/0x1e40 [ptlrpc] Feb 16 09:28:41 soak-8 kernel: [&lt;ffffffffc11512b0&gt;] ? ptlrpc_main+0x0/0x1e40 [ptlrpc] Feb 16 09:28:41 soak-8 kernel: [&lt;ffffffff810b252f&gt;] kthread+0xcf/0xe0 Feb 16 09:28:42 soak-8 kernel: [&lt;ffffffff810b2460&gt;] ? kthread+0x0/0xe0 Feb 16 09:28:42 soak-8 kernel: [&lt;ffffffff816b8798&gt;] ret_from_fork+0x58/0x90 Feb 16 09:28:42 soak-8 kernel: [&lt;ffffffff810b2460&gt;] ? kthread+0x0/0xe0 Feb 16 09:28:42 soak-8 kernel: </pre> </div></div> Soak stress cluster - Lustre version=2.10.57_58_gf24340c. LU-10678

LBUG: osd_handler.c:2353:osd_read_lock()) ASSERTION( obj->oo_owner == ((void *)0) ) failed:

Bug Major Resolved Not a Bug Yang Sheng Cliff White soak Fri, 16 Feb 2018 16:08:26 +0000 Wed, 27 Oct 2021 16:58:18 +0000 Tue, 11 Jun 2019 15:37:43 +0000 Lustre 2.11.0 1 15 <p>Yang Sheng</p> <p>Can you please advise</p> <p>Peter</p> <p>Hi,</p> <p>We have seen this a couple of times on 2.7.21.2 recently.</p> <p>The crashdump collection has failed on the first occurence, and I am waiting for confirmation about the second occurence.</p> <p>Would you be interested in a dump if we get one ?</p> <p>Hi, Sebastien,</p> <p>This is helpful if got a crash dump. TIA.</p> <p>Thanks,<br/> YangSheng</p> <p>Unfortunately, the dump collection failed because the crashkernel=auto parameter does not reserve enough memory for our configuration. I have requested this to be adjusted. Let's hope we can get a dump at next crash.</p> <p> </p> <p>Regards,</p> <p> </p> <p>Sebastien.</p> <p>Hi,</p> <p>I have this LBUG on server running : lustre-el7.3-2.7.21.3-255.ddn20.g10dd357.el7.x86_64</p> <p><span class="error">&#91;75345.251740&#93;</span> LustreError: 7714:0:(osd_handler.c:1751:osd_object_read_lock()) ASSERTION( obj-&gt;oo_owner == ((void *)0) ) failed: <br/> <span class="error">&#91;75345.265763&#93;</span> LustreError: 7714:0:(osd_handler.c:1751:osd_object_read_lock()) LBUG </p> <p>Do I need to open a Jira ticket for this one, or we can use this one ? Seem to be similar, but I preffer to ask.</p> <p>I will try to get the crash file this week and the whole dmesg.</p> <p> </p> <p>Regards,</p> <p>Johann</p> <p>We have a few similar crashes on version 2.10.1, kernel 3.10.0-693.2.2.el7_lustre.x86_64.</p> <p>We have an incomplete kernel dump and I uploaded to ftp.whamcloud.com in /uploads/<a href="https://jira.whamcloud.com/browse/LU-10678" title="LBUG: osd_handler.c:2353:osd_read_lock()) ASSERTION( obj-&gt;oo_owner == ((void *)0) ) failed:" class="issue-link" data-issue-key="LU-10678"><del>LU-10678</del></a> directory. Not sure if this helps.</p> <p>Thanks.</p> <p> </p> <p>Lixin Liu</p> <p>Simon Fraser University</p> <p> </p> <p>We have one complete vmcore from an MDS running kernel 3.10.0-693.11.1.el7 and lustre 2.7.21.2.</p> <p>I have uploaded it to ftp.whamcloud.com/uploads/<a href="https://jira.whamcloud.com/browse/LU-10678" title="LBUG: osd_handler.c:2353:osd_read_lock()) ASSERTION( obj-&gt;oo_owner == ((void *)0) ) failed:" class="issue-link" data-issue-key="LU-10678"><del>LU-10678</del></a>/vmcore-3.10.0-693.11.1.el7.x86_64_lustre-2.7.21.2</p> <p> </p> <p>Hi, Sebastien,</p> <p>Looks like you use a non-standard combination of lustre &amp; kernel? 2.7.21.2 should use 3.10.0.514.xx kernel. Can you provide debuginfo rpms?</p> <p>Thanks,<br/> YangSheng</p> <p>Hi Yang Sheng,</p> <p>I have just uploaded the corresponding lustre and kernel debuginfo packages to the same directory.</p> <p>Regards,</p> <p> </p> <p>Sebastien.</p> <p>Also seen in server running  lustre version 2.11.0.201 and kernel version 3.10.0-693.21.1.x3.1.11.x86_64 </p> <p>Apr 25 11:49:02 hostname-n03 kernel: Pid: 26722, comm: mdt03_004<br/> Apr 25 11:49:02 hostname-n03 kernel: IEC: 026000003: LASSERT: { "pid": "26722", "ext_pid": "0", "filename": "osd_handler.c", "line": "2382", "func_name": "osd_read_lock", "assert_info": "( obj-&gt;oo_owner == ((void *)0) ) failed: " }<br/> Apr 25 11:49:02 hostname-n03 kernel: IEC: 026000004: LBUG: { "pid": "26722", "ext_pid": "0", "filename": "osd_handler.c", "line": "2382", "func_name": "osd_read_lock" }<br/> Apr 25 11:49:02 hostname-n03 kernel:<br/> Apr 25 11:49:02 hostname-n03 kernel: <span class="error">&#91;&lt;ffffffffc0a407ae&gt;&#93;</span> libcfs_call_trace+0x4e/0x60 <span class="error">&#91;libcfs&#93;</span><br/> Apr 25 11:49:02 hostname-n03 kernel: <span class="error">&#91;&lt;ffffffffc0a4083c&gt;&#93;</span> lbug_with_loc+0x4c/0xb0 <span class="error">&#91;libcfs&#93;</span><br/> Apr 25 11:49:02 hostname-n03 kernel: <span class="error">&#91;&lt;ffffffffc12d4800&gt;&#93;</span> ? mdd_xattr_get+0x0/0x5c0 <span class="error">&#91;mdd&#93;</span><br/> Apr 25 11:49:02 hostname-n03 kernel: <span class="error">&#91;&lt;ffffffffc15569ca&gt;&#93;</span> osd_read_lock+0xda/0xe0 <span class="error">&#91;osd_ldiskfs&#93;</span><br/> Apr 25 11:49:02 hostname-n03 kernel: <span class="error">&#91;&lt;ffffffffc141256a&gt;&#93;</span> lod_read_lock+0x3a/0xd0 <span class="error">&#91;lod&#93;</span><br/> Apr 25 11:49:02 hostname-n03 kernel: <span class="error">&#91;&lt;ffffffffc12ce82a&gt;&#93;</span> mdd_read_lock+0x3a/0xd0 <span class="error">&#91;mdd&#93;</span><br/> Apr 25 11:49:02 hostname-n03 kernel: <span class="error">&#91;&lt;ffffffffc12d4870&gt;&#93;</span> mdd_xattr_get+0x70/0x5c0 <span class="error">&#91;mdd&#93;</span><br/> Apr 25 11:49:02 hostname-n03 kernel: <span class="error">&#91;&lt;ffffffffc1413b0b&gt;&#93;</span> ? lod_attr_get+0xab/0x130 <span class="error">&#91;lod&#93;</span><br/> Apr 25 11:49:02 hostname-n03 kernel: <span class="error">&#91;&lt;ffffffffc1370490&gt;&#93;</span> mdt_get_som+0x90/0x210 <span class="error">&#91;mdt&#93;</span><br/> Apr 25 11:49:02 hostname-n03 kernel: <span class="error">&#91;&lt;ffffffffc133ea95&gt;&#93;</span> mdt_attr_get_complex+0x955/0xb10 <span class="error">&#91;mdt&#93;</span><br/> Apr 25 11:49:02 hostname-n03 kernel: <span class="error">&#91;&lt;ffffffffc1368248&gt;&#93;</span> mdt_reint_open+0x898/0x3190 <span class="error">&#91;mdt&#93;</span><br/> Apr 25 11:49:02 hostname-n03 kernel: <span class="error">&#91;&lt;ffffffffc0c41261&gt;&#93;</span> ? upcall_cache_get_entry+0x211/0x8d0 <span class="error">&#91;obdclass&#93;</span><br/> Apr 25 11:49:02 hostname-n03 kernel: <span class="error">&#91;&lt;ffffffffc0c46f0e&gt;&#93;</span> ? lu_ucred+0x1e/0x30 <span class="error">&#91;obdclass&#93;</span><br/> Apr 25 11:49:02 hostname-n03 kernel: <span class="error">&#91;&lt;ffffffffc134ccc5&gt;&#93;</span> ? mdt_ucred+0x15/0x20 <span class="error">&#91;mdt&#93;</span><br/> Apr 25 11:49:02 hostname-n03 kernel: <span class="error">&#91;&lt;ffffffffc134d551&gt;&#93;</span> ? mdt_root_squash+0x21/0x430 <span class="error">&#91;mdt&#93;</span><br/> Apr 25 11:49:02 hostname-n03 kernel: <span class="error">&#91;&lt;ffffffffc135dd93&gt;&#93;</span> mdt_reint_rec+0x83/0x210 <span class="error">&#91;mdt&#93;</span><br/> Apr 25 11:49:02 hostname-n03 kernel: <span class="error">&#91;&lt;ffffffffc133d1bb&gt;&#93;</span> mdt_reint_internal+0x5fb/0x9c0 <span class="error">&#91;mdt&#93;</span><br/> Apr 25 11:49:02 hostname-n03 kernel: <span class="error">&#91;&lt;ffffffffc1349737&gt;&#93;</span> mdt_intent_reint+0x157/0x420 <span class="error">&#91;mdt&#93;</span><br/> Apr 25 11:49:02 hostname-n03 kernel: <span class="error">&#91;&lt;ffffffffc1340315&gt;&#93;</span> mdt_intent_opc+0x455/0xae0 <span class="error">&#91;mdt&#93;</span><br/> Apr 25 11:49:03 hostname-n03 kernel: <span class="error">&#91;&lt;ffffffffc0e69d10&gt;&#93;</span> ? lustre_swab_ldlm_intent+0x0/0x20 <span class="error">&#91;ptlrpc&#93;</span><br/> Apr 25 11:49:03 hostname-n03 kernel: <span class="error">&#91;&lt;ffffffffc1347f63&gt;&#93;</span> mdt_intent_policy+0x1a3/0x360 <span class="error">&#91;mdt&#93;</span><br/> Apr 25 11:49:03 hostname-n03 kernel: <span class="error">&#91;&lt;ffffffffc0e1af9e&gt;&#93;</span> ldlm_lock_enqueue+0x34e/0xa50 <span class="error">&#91;ptlrpc&#93;</span><br/> Apr 25 11:49:03 hostname-n03 kernel: <span class="error">&#91;&lt;ffffffffc0a5674e&gt;&#93;</span> ? cfs_hash_add+0xbe/0x1a0 <span class="error">&#91;libcfs&#93;</span><br/> Apr 25 11:49:03 hostname-n03 kernel: <span class="error">&#91;&lt;ffffffffc0e43843&gt;&#93;</span> ldlm_handle_enqueue0+0x8f3/0x13e0 <span class="error">&#91;ptlrpc&#93;</span><br/> Apr 25 11:49:03 hostname-n03 kernel: <span class="error">&#91;&lt;ffffffffc0e69d90&gt;&#93;</span> ? lustre_swab_ldlm_request+0x0/0x30 <span class="error">&#91;ptlrpc&#93;</span><br/> Apr 25 11:49:03 hostname-n03 kernel: <span class="error">&#91;&lt;ffffffffc0ec8572&gt;&#93;</span> tgt_enqueue+0x62/0x210 <span class="error">&#91;ptlrpc&#93;</span><br/> Apr 25 11:49:03 hostname-n03 kernel: <span class="error">&#91;&lt;ffffffffc0ece8ba&gt;&#93;</span> tgt_request_handle+0x92a/0x13b0 <span class="error">&#91;ptlrpc&#93;</span><br/> Apr 25 11:49:03 hostname-n03 kernel: <span class="error">&#91;&lt;ffffffffc0e73f13&gt;&#93;</span> ptlrpc_server_handle_request+0x253/0xab0 <span class="error">&#91;ptlrpc&#93;</span><br/> Apr 25 11:49:03 hostname-n03 kernel: <span class="error">&#91;&lt;ffffffffc0e71aa5&gt;&#93;</span> ? ptlrpc_wait_event+0xa5/0x360 <span class="error">&#91;ptlrpc&#93;</span><br/> Apr 25 11:49:03 hostname-n03 kernel: <span class="error">&#91;&lt;ffffffff810c7c92&gt;&#93;</span> ? default_wake_function+0x12/0x20<br/> Apr 25 11:49:03 hostname-n03 kernel: <span class="error">&#91;&lt;ffffffff810bdc4b&gt;&#93;</span> ? __wake_up_common+0x5b/0x90<br/> Apr 25 11:49:03 hostname-n03 kernel: <span class="error">&#91;&lt;ffffffffc0e77862&gt;&#93;</span> ptlrpc_main+0xab2/0x1f70 <span class="error">&#91;ptlrpc&#93;</span><br/> Apr 25 11:49:03 hostname-n03 kernel: <span class="error">&#91;&lt;ffffffffc0e76db0&gt;&#93;</span> ? ptlrpc_main+0x0/0x1f70 <span class="error">&#91;ptlrpc&#93;</span><br/> Apr 25 11:49:03 hostname-n03 kernel: <span class="error">&#91;&lt;ffffffff810b4031&gt;&#93;</span> kthread+0xd1/0xe0<br/> Apr 25 11:49:03 hostname-n03 kernel: <span class="error">&#91;&lt;ffffffff810b3f60&gt;&#93;</span> ? kthread+0x0/0xe0<br/> Apr 25 11:49:03 hostname-n03 kernel: <span class="error">&#91;&lt;ffffffff816c155d&gt;&#93;</span> ret_from_fork+0x5d/0xb0<br/> Apr 25 11:49:03 hostname-n03 kernel: <span class="error">&#91;&lt;ffffffff810b3f60&gt;&#93;</span> ? kthread+0x0/0xe0<br/> Apr 25 11:49:03 hostname-n03 kernel:<br/> Apr 25 11:49:03 hostname-n03 kernel: Kernel panic - not syncing: LBUG</p> <p>We, at Cray, encountered a bunch of similar crashes which we associated with the broken rwsem implementation in certain RHEL7 kernels.</p> <p><a href="https://access.redhat.com/solutions/3393611" class="external-link" target="_blank" rel="nofollow noopener">https://access.redhat.com/solutions/3393611</a></p> <p>Resolution<br/> Red Hat Enterprise Linux 7.6<br/> The issue was fixed in kernel-3.10.0-957.12.1.el7 from Errata RHSA-2019:0818<br/> Red Hat Enterprise Linux 7.4.z (EUS)<br/> The issue was fixed in kernel-3.10.0-693.47.2.el7 from Errata RHSA-2019:1170</p> <p>Hi, Andrew,</p> <p>Thanks for the info. It is really a tricky one.</p> <p>Thanks,<br/> YangSheng</p> <p>Red Hat bug not Lustre bug</p> Duplicate LU-11786 LU-12508 Related LU-15156 Development Rank 1|hzzsxj: Rank (Obsolete) 9223372036854775807