https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6447 Lustre <p>Running on RHEL7.1 with CONFIG_DEBUG_SLEEP_ATOMIC enabled caught this gem:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Mar 27 21:12:13 centos6-16 kernel: BUG: sleeping function called from invalid context at mm/slab.c:3054 Mar 27 21:12:13 centos6-16 kernel: in_atomic(): 1, irqs_disabled(): 0, pid: 19599, name: mdt00_001 Mar 27 21:12:13 centos6-16 kernel: 1 lock held by mdt00_001/19599: Mar 27 21:12:13 centos6-16 kernel: #0: (&amp;cache-&gt;uc_upcall_rwlock){......}, at: [&lt;ffffffffa0b15ad1&gt;] mdt_identity_do_upcall+0x91/0x470 [mdt] Mar 27 21:12:13 centos6-16 kernel: CPU: 3 PID: 19599 Comm: mdt00_001 Tainted: GF W O-------------- 3.10.0-debug #5 Mar 27 21:12:13 centos6-16 kernel: Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 Mar 27 21:12:13 centos6-16 kernel: ffff8800808945c0 000000006b10abfb ffff88008b1eb808 ffffffff816ccb68 Mar 27 21:12:13 centos6-16 kernel: ffff88008b1eb820 ffffffff810a8fd9 0000000000000000 ffff88008b1eb8b8 Mar 27 21:12:13 centos6-16 kernel: ffffffff811bdeda ffff880074b48000 0000000000000246 0000000000000000 Mar 27 21:12:13 centos6-16 kernel: Call Trace: Mar 27 21:12:13 centos6-16 kernel: [&lt;ffffffff816ccb68&gt;] dump_stack+0x19/0x1b Mar 27 21:12:13 centos6-16 kernel: [&lt;ffffffff810a8fd9&gt;] __might_sleep+0xe9/0x110 Mar 27 21:12:13 centos6-16 kernel: [&lt;ffffffff811bdeda&gt;] __kmalloc_track_caller+0x11a/0x620 Mar 27 21:12:13 centos6-16 kernel: [&lt;ffffffffa0b15b0b&gt;] ? mdt_identity_do_upcall+0xcb/0x470 [mdt] Mar 27 21:12:13 centos6-16 kernel: [&lt;ffffffff81182f91&gt;] kstrdup+0x31/0x60 Mar 27 21:12:13 centos6-16 kernel: [&lt;ffffffffa0b15b0b&gt;] mdt_identity_do_upcall+0xcb/0x470 [mdt] Mar 27 21:12:13 centos6-16 kernel: [&lt;ffffffffa02de48f&gt;] upcall_cache_get_entry+0x2af/0x8e0 [obdclass] Mar 27 21:12:13 centos6-16 kernel: [&lt;ffffffffa050ef87&gt;] ? lustre_msg_buf+0x17/0x60 [ptlrpc] Mar 27 21:12:13 centos6-16 kernel: [&lt;ffffffffa0536882&gt;] ? __req_capsule_get+0x162/0x710 [ptlrpc] Mar 27 21:12:13 centos6-16 kernel: [&lt;ffffffffa05116cf&gt;] ? lustre_pack_reply_flags+0x6f/0x1e0 [ptlrpc] Mar 27 21:12:13 centos6-16 kernel: [&lt;ffffffffa0b16437&gt;] mdt_identity_get+0x17/0x50 [mdt] Mar 27 21:12:13 centos6-16 kernel: [&lt;ffffffffa0af921a&gt;] mdt_init_ucred_reint+0x23a/0x380 [mdt] Mar 27 21:12:13 centos6-16 kernel: [&lt;ffffffffa0ae9dcb&gt;] mdt_reint_internal+0x24b/0x760 [mdt] Mar 27 21:12:13 centos6-16 kernel: [&lt;ffffffffa0aea442&gt;] mdt_intent_reint+0x162/0x400 [mdt] Mar 27 21:12:13 centos6-16 kernel: [&lt;ffffffffa0af447a&gt;] mdt_intent_policy+0x57a/0xbe0 [mdt] Mar 27 21:12:13 centos6-16 kernel: [&lt;ffffffffa04c6f66&gt;] ldlm_lock_enqueue+0x326/0x900 [ptlrpc] Mar 27 21:12:13 centos6-16 kernel: [&lt;ffffffffa014f505&gt;] ? cfs_hash_rw_unlock+0x15/0x20 [libcfs] Mar 27 21:12:13 centos6-16 kernel: [&lt;ffffffffa04ed672&gt;] ldlm_handle_enqueue0+0x502/0x1520 [ptlrpc] Mar 27 21:12:13 centos6-16 kernel: [&lt;ffffffffa05135b0&gt;] ? lustre_swab_ldlm_lock_desc+0x30/0x30 [ptlrpc] Mar 27 21:12:13 centos6-16 kernel: [&lt;ffffffffa05670e2&gt;] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 27 21:12:13 centos6-16 kernel: [&lt;ffffffffa056b5c5&gt;] tgt_request_handle+0x645/0xfe0 [ptlrpc] Mar 27 21:12:13 centos6-16 kernel: [&lt;ffffffffa051c881&gt;] ptlrpc_server_handle_request+0x231/0xab0 [ptlrpc] Mar 27 21:12:13 centos6-16 kernel: [&lt;ffffffffa051a3f8&gt;] ? ptlrpc_wait_event+0xb8/0x360 [ptlrpc] Mar 27 21:12:13 centos6-16 kernel: [&lt;ffffffffa05207d0&gt;] ptlrpc_main+0xae0/0x1ee0 [ptlrpc] Mar 27 21:12:13 centos6-16 kernel: [&lt;ffffffff816d5b97&gt;] ? _raw_spin_unlock_irq+0x27/0x50 Mar 27 21:12:13 centos6-16 kernel: [&lt;ffffffff816d38be&gt;] ? __schedule+0x2fe/0x810 Mar 27 21:12:13 centos6-16 kernel: [&lt;ffffffffa051fcf0&gt;] ? ptlrpc_register_service+0xf20/0xf20 [ptlrpc] Mar 27 21:12:13 centos6-16 kernel: [&lt;ffffffff8109c00a&gt;] kthread+0xea/0xf0 Mar 27 21:12:13 centos6-16 kernel: [&lt;ffffffff8109bf20&gt;] ? kthread_create_on_node+0x140/0x140 Mar 27 21:12:13 centos6-16 kernel: [&lt;ffffffff816df2bc&gt;] ret_from_fork+0x7c/0xb0 Mar 27 21:12:13 centos6-16 kernel: [&lt;ffffffff8109bf20&gt;] ? kthread_create_on_node+0x140/0x140 </pre> </div></div> LU-6447

mdt_identity_upcall calls sleeping function under rwlock

Bug Critical Resolved Fixed Niu Yawei Oleg Drokin llnl Fri, 10 Apr 2015 02:26:58 +0000 Thu, 16 Feb 2017 20:49:21 +0000 Thu, 11 Aug 2016 12:09:30 +0000 Lustre 2.9.0 1 8 <p>Oleg Drokin (oleg.drokin@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/14432" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/14432</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6447" title="mdt_identity_upcall calls sleeping function under rwlock" class="issue-link" data-issue-key="LU-6447"><del>LU-6447</del></a> mdt: mdt_identity_upcall to not block with rwlock held<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 912a0861bc6fff558483a58bc29cad76c7ae4681</p> <p>This is my first stab at the problem unless somebody has better ideas</p> <p>We've started seeing a very similar call stack with the same error in a testbed we've stood up.</p> <p>Yes, it's a problem on rhel7 with debug atomic sleep option enabled.<br/> I am carrying my patch referenced above even though it's not perfect meanwhile to reduce amount of noise in my logs.</p> <p>This seems to be implicated in long (15 second) hangs that I am sometimes seeing when poking around the lustre filesystem interactively. That is going to be a big issue for our hotline when 2.8 goes live, so we really need a fix. I would this should be on the docket for fixing before 2.9 comes out.</p> <p>Niu</p> <p>Could you please refresh Oleg's patch to address the existing concerns?</p> <p>Thanks</p> <p>Peter</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/14432/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/14432/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6447" title="mdt_identity_upcall calls sleeping function under rwlock" class="issue-link" data-issue-key="LU-6447"><del>LU-6447</del></a> mdt: mdt_identity_upcall to not block with rwlock held<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: e8273a3dd71c4e6ab5ca9de3fbfbc0f7603d6930</p> <p>Landed for 2.9</p> Related Development Rank 1|hzxai7: Rank (Obsolete) 9223372036854775807 Severity