https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9142 Lustre <p>sleeping in invalid context at ldiskfs/ext4_jbd2.c:259 followed by LBUG:</p> <p>2017-02-21 17:51:20 <span class="error">&#91;25522.565474&#93;</span> BUG: sleeping function called from invalid context at /builddir/build/BUILD/lustre-2.8.0_8.chaos/ldiskfs/ext4_jbd2.c:259<br/> 2017-02-21 17:51:20 <span class="error">&#91;25522.577368&#93;</span> in_atomic(): 1, irqs_disabled(): 0, pid: 15066, name: mdt_rdpg00_003<br/> 2017-02-21 17:51:20 <span class="error">&#91;25522.584754&#93;</span> CPU: 6 PID: 15066 Comm: mdt_rdpg00_003 Tainted: G IOE ------------ 3.10.0-514.6.1.1chaos.ch6.x86_64 #1<br/> 2017-02-21 17:51:20 <span class="error">&#91;25522.596123&#93;</span> Hardware name: Dell Inc. PowerEdge R710/00NH4P, BIOS 3.0.0 01/31/2011<br/> 2017-02-21 17:51:20 <span class="error">&#91;25522.603591&#93;</span> ffff88060710df70 00000000b9ff74be ffff880313ab3640 ffffffff8169c685<br/> 2017-02-21 17:51:20 <span class="error">&#91;25522.611015&#93;</span> ffff880313ab3650 ffffffff810c0199 ffff880313ab36b0 ffffffffa0e5c048<br/> 2017-02-21 17:51:20 <span class="error">&#91;25522.618435&#93;</span> 0000000000000002 0000000000000000 00000000b9ff74be ffff880313ab3698<br/> 2017-02-21 17:51:20 <span class="error">&#91;25522.625855&#93;</span> Call Trace:<br/> 2017-02-21 17:51:20 <span class="error">&#91;25522.628305&#93;</span> <span class="error">&#91;&lt;ffffffff8169c685&gt;&#93;</span> dump_stack+0x19/0x1b<br/> 2017-02-21 17:51:20 <span class="error">&#91;25522.633436&#93;</span> <span class="error">&#91;&lt;ffffffff810c0199&gt;&#93;</span> __might_sleep+0xd9/0x100<br/> 2017-02-21 17:51:20 <span class="error">&#91;25522.638922&#93;</span> <span class="error">&#91;&lt;ffffffffa0e5c048&gt;&#93;</span> __ldiskfs_handle_dirty_metadata+0x38/0x230 <span class="error">&#91;ldiskfs&#93;</span><br/> 2017-02-21 17:51:20 <span class="error">&#91;25522.646829&#93;</span> <span class="error">&#91;&lt;ffffffff810b4745&gt;&#93;</span> ? wake_up_bit+0x25/0x30<br/> 2017-02-21 17:51:20 <span class="error">&#91;25522.652228&#93;</span> <span class="error">&#91;&lt;ffffffffa0ea8bf2&gt;&#93;</span> ldiskfs_getblk+0x142/0x210 <span class="error">&#91;ldiskfs&#93;</span><br/> 2017-02-21 17:51:20 <span class="error">&#91;25522.658754&#93;</span> <span class="error">&#91;&lt;ffffffffa0ea8ce7&gt;&#93;</span> ldiskfs_bread+0x27/0xe0 <span class="error">&#91;ldiskfs&#93;</span><br/> 2017-02-21 17:51:20 <span class="error">&#91;25522.665025&#93;</span> <span class="error">&#91;&lt;ffffffffa0f3e6e9&gt;&#93;</span> osd_ldiskfs_write_record+0x169/0x360 <span class="error">&#91;osd_ldiskfs&#93;</span><br/> 2017-02-21 17:51:20 <span class="error">&#91;25522.672764&#93;</span> <span class="error">&#91;&lt;ffffffffa0f3ea22&gt;&#93;</span> osd_write+0x142/0x570 <span class="error">&#91;osd_ldiskfs&#93;</span><br/> 2017-02-21 17:51:20 <span class="error">&#91;25522.679206&#93;</span> <span class="error">&#91;&lt;ffffffffa11ac779&gt;&#93;</span> lod_sub_object_write+0x1d9/0x420 <span class="error">&#91;lod&#93;</span><br/> 2017-02-21 17:51:20 <span class="error">&#91;25522.685900&#93;</span> <span class="error">&#91;&lt;ffffffffa1194054&gt;&#93;</span> lod_write+0x24/0x30 <span class="error">&#91;lod&#93;</span><br/> 2017-02-21 17:51:20 <span class="error">&#91;25522.691493&#93;</span> <span class="error">&#91;&lt;ffffffffa0a47129&gt;&#93;</span> dt_record_write+0x39/0x120 <span class="error">&#91;obdclass&#93;</span><br/> 2017-02-21 17:51:20 <span class="error">&#91;25522.698150&#93;</span> <span class="error">&#91;&lt;ffffffffa0c92ee0&gt;&#93;</span> tgt_add_reply_data+0x240/0x430 <span class="error">&#91;ptlrpc&#93;</span><br/> 2017-02-21 17:51:20 <span class="error">&#91;25522.704964&#93;</span> <span class="error">&#91;&lt;ffffffffa0c950fc&gt;&#93;</span> tgt_last_rcvd_update+0x77c/0xc80 <span class="error">&#91;ptlrpc&#93;</span><br/> 2017-02-21 17:51:20 <span class="error">&#91;25522.711952&#93;</span> <span class="error">&#91;&lt;ffffffffa0c9914d&gt;&#93;</span> tgt_txn_stop_cb+0x19d/0x490 <span class="error">&#91;ptlrpc&#93;</span><br/> 2017-02-21 17:51:20 <span class="error">&#91;25522.718473&#93;</span> <span class="error">&#91;&lt;ffffffffa071b639&gt;&#93;</span> ? start_this_handle+0x1a9/0x480 <span class="error">&#91;jbd2&#93;</span><br/> 2017-02-21 17:51:20 <span class="error">&#91;25522.725182&#93;</span> <span class="error">&#91;&lt;ffffffffa0a4861e&gt;&#93;</span> dt_txn_hook_stop+0x7e/0xd0 <span class="error">&#91;obdclass&#93;</span><br/> 2017-02-21 17:51:20 <span class="error">&#91;25522.731793&#93;</span> <span class="error">&#91;&lt;ffffffffa0f27eb8&gt;&#93;</span> osd_trans_stop+0x138/0x5a0 <span class="error">&#91;osd_ldiskfs&#93;</span><br/> 2017-02-21 17:51:20 <span class="error">&#91;25522.738692&#93;</span> <span class="error">&#91;&lt;ffffffffa0cb4429&gt;&#93;</span> top_trans_stop+0x99/0x970 <span class="error">&#91;ptlrpc&#93;</span><br/> 2017-02-21 17:51:20 <span class="error">&#91;25522.745054&#93;</span> <span class="error">&#91;&lt;ffffffffa0a450d3&gt;&#93;</span> ? lu_context_init+0xd3/0x200 <span class="error">&#91;obdclass&#93;</span><br/> 2017-02-21 17:51:20 <span class="error">&#91;25522.751836&#93;</span> <span class="error">&#91;&lt;ffffffffa1180399&gt;&#93;</span> lod_trans_stop+0x259/0x340 <span class="error">&#91;lod&#93;</span><br/> 2017-02-21 17:51:20 <span class="error">&#91;25522.758041&#93;</span> <span class="error">&#91;&lt;ffffffffa0cb414f&gt;&#93;</span> ? top_trans_start+0x70f/0x950 <span class="error">&#91;ptlrpc&#93;</span><br/> 2017-02-21 17:51:20 <span class="error">&#91;25522.764738&#93;</span> <span class="error">&#91;&lt;ffffffffa0f1b2c5&gt;&#93;</span> ? osd_trans_create+0x75/0x390 <span class="error">&#91;osd_ldiskfs&#93;</span><br/> 2017-02-21 17:51:20 <span class="error">&#91;25522.771882&#93;</span> <span class="error">&#91;&lt;ffffffffa0a614ce&gt;&#93;</span> ? lu_ucred+0x1e/0x30 <span class="error">&#91;obdclass&#93;</span><br/> 2017-02-21 17:51:20 <span class="error">&#91;25522.777975&#93;</span> <span class="error">&#91;&lt;ffffffffa12043bf&gt;&#93;</span> ? mdd_changelog_data_store+0x2f/0x330 <span class="error">&#91;mdd&#93;</span><br/> 2017-02-21 17:51:20 <span class="error">&#91;25522.785135&#93;</span> <span class="error">&#91;&lt;ffffffffa0cb0ec7&gt;&#93;</span> ? top_trans_create+0x87/0x200 <span class="error">&#91;ptlrpc&#93;</span><br/> 2017-02-21 17:51:20 <span class="error">&#91;25522.791832&#93;</span> <span class="error">&#91;&lt;ffffffffa120eafd&gt;&#93;</span> mdd_trans_stop+0x1d/0x25 <span class="error">&#91;mdd&#93;</span><br/> 2017-02-21 17:51:20 <span class="error">&#91;25522.797834&#93;</span> <span class="error">&#91;&lt;ffffffffa1205327&gt;&#93;</span> mdd_close+0x117/0xc90 <span class="error">&#91;mdd&#93;</span><br/> 2017-02-21 17:51:20 <span class="error">&#91;25522.803588&#93;</span> <span class="error">&#91;&lt;ffffffffa10d5e29&gt;&#93;</span> mdt_mfd_close+0x249/0x620 <span class="error">&#91;mdt&#93;</span><br/> 2017-02-21 17:51:20 <span class="error">&#91;25522.809681&#93;</span> <span class="error">&#91;&lt;ffffffffa10db573&gt;&#93;</span> mdt_close_internal+0x103/0x470 <span class="error">&#91;mdt&#93;</span><br/> 2017-02-21 17:51:20 <span class="error">&#91;25522.816207&#93;</span> <span class="error">&#91;&lt;ffffffffa10dbb00&gt;&#93;</span> mdt_close+0x220/0x780 <span class="error">&#91;mdt&#93;</span><br/> 2017-02-21 17:51:20 <span class="error">&#91;25522.821982&#93;</span> <span class="error">&#91;&lt;ffffffffa0ca0445&gt;&#93;</span> tgt_request_handle+0x915/0x1320 <span class="error">&#91;ptlrpc&#93;</span><br/> 2017-02-21 17:51:20 <span class="error">&#91;25522.828876&#93;</span> <span class="error">&#91;&lt;ffffffffa0c4ce9b&gt;&#93;</span> ptlrpc_server_handle_request+0x21b/0xa90 <span class="error">&#91;ptlrpc&#93;</span><br/> 2017-02-21 17:51:20 <span class="error">&#91;25522.836534&#93;</span> <span class="error">&#91;&lt;ffffffffa0931748&gt;&#93;</span> ? lc_watchdog_touch+0x68/0x180 <span class="error">&#91;libcfs&#93;</span><br/> 2017-02-21 17:51:20 <span class="error">&#91;25522.843341&#93;</span> <span class="error">&#91;&lt;ffffffffa0c4aa6b&gt;&#93;</span> ? ptlrpc_wait_event+0xab/0x350 <span class="error">&#91;ptlrpc&#93;</span><br/> 2017-02-21 17:51:20 <span class="error">&#91;25522.850150&#93;</span> <span class="error">&#91;&lt;ffffffffa0c50f40&gt;&#93;</span> ptlrpc_main+0xa90/0x1db0 <span class="error">&#91;ptlrpc&#93;</span><br/> 2017-02-21 17:51:20 <span class="error">&#91;25522.856439&#93;</span> <span class="error">&#91;&lt;ffffffffa0c504b0&gt;&#93;</span> ? ptlrpc_register_service+0xe40/0xe40 <span class="error">&#91;ptlrpc&#93;</span><br/> 2017-02-21 17:51:20 <span class="error">&#91;25522.863823&#93;</span> <span class="error">&#91;&lt;ffffffff810b360f&gt;&#93;</span> kthread+0xcf/0xe0<br/> 2017-02-21 17:51:20 <span class="error">&#91;25522.868693&#93;</span> <span class="error">&#91;&lt;ffffffff810b3540&gt;&#93;</span> ? kthread_create_on_node+0x140/0x140<br/> 2017-02-21 17:51:20 <span class="error">&#91;25522.875212&#93;</span> <span class="error">&#91;&lt;ffffffff816ad498&gt;&#93;</span> ret_from_fork+0x58/0x90<br/> 2017-02-21 17:51:20 <span class="error">&#91;25522.880601&#93;</span> <span class="error">&#91;&lt;ffffffff810b3540&gt;&#93;</span> ? kthread_create_on_node+0x140/0x140<br/> 2017-02-21 17:51:20 <span class="error">&#91;25522.887133&#93;</span> LNetError: 15066:0:(lib-lnet.h:293:lnet_msg_alloc()) ASSERTION( !(((current_thread_info()-&gt;preempt_count) &amp; ((((1UL &lt;&lt; (10))-1) &lt;&lt; ((0 + 8) + 8)) | (((1UL &lt;&lt; (8))-1) &lt;&lt; (0 + 8)) | (((1UL &lt;&lt; (1))-1) &lt;&lt; (((0 + 8) + 8) + 10))))) || (((sizeof(*msg))) &lt;= (2 &lt;&lt; 12) &amp;&amp; (((((( gfp_t)0x10u) | (( gfp_t)0x40u)))) &amp; ((( gfp_t)0x20u)))) != 0 ) failed:<br/> 2017-02-21 17:51:20 <span class="error">&#91;25522.918175&#93;</span> LNetError: 15066:0:(lib-lnet.h:293:lnet_msg_alloc()) LBUG</p> servers and clients:<br/> lustre: 2.8.0_8.chaos<br/> kernel: 3.10.0-514.6.1.1chaos.ch6.x86_64<br/> RHEL 7.3 LU-9142

(lib-lnet.h:293:lnet_msg_alloc()) LBUG

Bug Major Resolved Not a Bug Amir Shehata Ruth Klundt Wed, 22 Feb 2017 16:11:47 +0000 Fri, 3 Mar 2017 00:11:46 +0000 Tue, 28 Feb 2017 17:49:43 +0000 Lustre 2.8.0 0 3 <p>The I/O load was IOR, on a smallish test cluster ~200 nodes total.</p> <p>Amir</p> <p>Could you please advise on this one?</p> <p>Thanks</p> <p>Peter</p> <p>Is this crash on the server or the client?</p> <p>If the server, has the kernel been patched?</p> <p> it's the MDS, and I'll check on the kernel. It is also from LLNL. </p> <p>The kernel has the following patches in the src rpm:</p> <p>ACPI-APEI-GHES-Further-instrumentation-refinements.patch<br/> ACPI-APEI-GHES-GHES-platform-driver-instrumentation.patch<br/> edac-Enable-MCE-and-amd64-error-injection.patch<br/> Export-mutex-lockdep-symbols-to-non-GPL-modules.patch<br/> Fix-missed-REQ_TYPE_SPECIAL-REQ_TYPE_DRV_PRIV-in-nbd-driver.patch<br/> fs-drop_caches-report-amount-of-dropped-pagecache.patch<br/> fs-Enhance-proc-sys-vm-drop_caches.patch<br/> fs-nfs-support-16-groups-in-nfs.patch<br/> IB-hfi1-Add-static-PCIe-Gen3-CTLE-tuning.patch<br/> IB-rdmavt-Only-put-mmap_info-ref-if-it-exists.patch<br/> ipmi-Disable-kipmid-by-default.patch<br/> mlx4-Add-huge-TLB-support-for-CQs-and-QPs.patch<br/> mm-Export-symbols-needed-by-DI-MMAP.patch<br/> net-9p-add-privport-option-to-9p-tcp-transport.patch<br/> net-9p-Initialize-opts-privport-as-it-should-be.patch<br/> net-9p-Remove-ib_get_dma_mr-calls.patch<br/> Revert-scsi-sas-provide-stub-implementation-for-scsi_is_sas_rphy.patch<br/> Revert-scsi-sas-remove-is_sas_attached.patch<br/> Revert-scsi-ses-use-scsi_is_sas_rphy-instead-of-is_sas_attached.patch</p> <p>This doesn't look like a lustre patched kernel. </p> <p>I would expect the following patches to be applied:<br/> raid5-mmp-unplug-dev-3.7.patch<br/> dev_read_only-3.7.patch<br/> blkdev_tunables-3.8.patch<br/> jbd2-fix-j_list_lock-unlock-3.10-rhel7.patch</p> <p>According to Andreas LLNL doesn't patch the kernel for lustre anymore because they use ZFS.</p> <p>Out of those patches jdb2-fix is the real bug fixer.</p> <p>And according to the crash log you have listed in the ticket this could be your problem.</p> <p>Please make sure to patch the kernel properly for the server nodes.</p> <p>Thanks for the diagnosis, you can close this bug. </p> <p>Issue due to a non lustre patched kernel.</p> <p>FYI, the jbd2 patch would appear to be obsolete for rhel7.3, it was generated against 3.10.0-327.36.1. The kernel I'm currently looking at is 3.10.0-514, and there is only one spin_lock before the if-else block, and one unlock after. The patch shows one spin_lock within each of the if-else branches.</p> <p>I'm surprised if that patch is not failing, even against rhel stock kernels.</p> <p>sorry ignore the last comment, I was not looking in a redhat kernel. </p> <p>Also note that as far as I can tell 2.8 only supports the following RHEL 7.x:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">3.10-rhel7.series 3.10.0-327.3.1.el7 (RHEL 7.2) </pre> </div></div> <p>full list of kernel version support for 2.8.0</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">2.6-rhel6.series 2.6.32-431.29.2.el6 (RHEL 6.5) 2.6-rhel6.series 2.6.32-504.30.3.el6 (RHEL 6.6) 2.6-rhel6.series 2.6.32-573.12.1.el6 (RHEL 6.7) 3.10-rhel7.series 3.10.0-327.3.1.el7 (RHEL 7.2) 3.0-sles11sp3.series 3.0.101-0.47.71 (SLES11 SP3) 3.0-sles11sp3.series 3.0.101-68 (SLES11 SP4) </pre> </div></div> Development Rank 1|hzz4n3: Rank (Obsolete) 9223372036854775807 Severity