https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6409 Lustre <p>nodemap_destroy seems to be using mutexes when called under spinlock/rwlock.</p> <p>Here's the relevant error message:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 6003.169249] Lustre: server umount lustre-OST0001 complete [ 6007.190173] BUG: sleeping function called from invalid context at kernel/mutex.c:576 [ 6007.190482] in_atomic(): 1, irqs_disabled(): 0, pid: 20641, name: rmmod [ 6007.190659] 1 lock held by rmmod/20641: [ 6007.190805] #0: (&amp;new_bkts[i]-&gt;hsb_lock.rw){......}, at: [&lt;ffffffffa015151d&gt;] cfs_hash_rw_lock+0xd/0x20 [libcfs] [ 6007.191177] CPU: 3 PID: 20641 Comm: rmmod Tainted: GF W O-------------- 3.10.0-debug #5 [ 6007.191480] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 6007.191648] ffff880083aac540 000000008428f159 ffff8800967e7d50 ffffffff816ccb68 [ 6007.191954] ffff8800967e7d68 ffffffff810a8fd9 ffff8800a8d2b0c0 ffff8800967e7de8 [ 6007.192275] ffffffff816d0dd5 0000000000000000 ffff8800967e7de0 0000000000000246 [ 6007.192582] Call Trace: [ 6007.192716] [&lt;ffffffff816ccb68&gt;] dump_stack+0x19/0x1b [ 6007.192879] [&lt;ffffffff810a8fd9&gt;] __might_sleep+0xe9/0x110 [ 6007.193063] [&lt;ffffffff816d0dd5&gt;] mutex_lock_nested+0x55/0x490 [ 6007.193281] [&lt;ffffffffa057cfa4&gt;] ? nodemap_destroy+0x74/0x1f0 [ptlrpc] [ 6007.193491] [&lt;ffffffffa05812ab&gt;] nm_member_reclassify_nodemap+0x1b/0x40 [ptlrpc] [ 6007.193799] [&lt;ffffffffa057cfbc&gt;] nodemap_destroy+0x8c/0x1f0 [ptlrpc] [ 6007.193993] [&lt;ffffffffa057d14d&gt;] nodemap_putref+0x2d/0xa0 [ptlrpc] [ 6007.194201] [&lt;ffffffffa057e1f5&gt;] nodemap_hs_put_locked+0x15/0x20 [ptlrpc] [ 6007.194392] [&lt;ffffffffa0151f25&gt;] cfs_hash_bd_del_locked+0x75/0xf0 [libcfs] [ 6007.194578] [&lt;ffffffffa015579c&gt;] cfs_hash_putref+0x1cc/0x500 [libcfs] [ 6007.194771] [&lt;ffffffffa057e22a&gt;] nodemap_cleanup_all+0x2a/0x30 [ptlrpc] [ 6007.194967] [&lt;ffffffffa057e23e&gt;] nodemap_mod_exit+0xe/0x20 [ptlrpc] [ 6007.195177] [&lt;ffffffffa0583bbd&gt;] ptlrpc_exit+0x9/0x44c [ptlrpc] [ 6007.195354] [&lt;ffffffff810e769b&gt;] SyS_delete_module+0x16b/0x2d0 [ 6007.195526] [&lt;ffffffff816df369&gt;] system_call_fastpath+0x16/0x1b </pre> </div></div> LU-6409

sleeping while atomic in nodemap_destroy

Bug Major Resolved Fixed WC Triage Oleg Drokin patch Sat, 28 Mar 2015 01:09:30 +0000 Thu, 23 Mar 2017 19:33:42 +0000 Mon, 29 Jun 2015 22:50:33 +0000 Lustre 2.8.0 0 8 <p>Kit Westneat (kit.westneat@gmail.com) uploaded a new patch: <a href="http://review.whamcloud.com/14275" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/14275</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6409" title="sleeping while atomic in nodemap_destroy" class="issue-link" data-issue-key="LU-6409"><del>LU-6409</del></a> nodemap: replace mutex with spinlock<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: c334985339b345fc90128f3d9d55d0779e2dd8ca</p> <p>Please note that with this latest patch I still get this warning:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Apr 11 00:01:22 centos6-1 kernel: BUG: sleeping function called from invalid context at /home/green/git/lustre-release/libcfs/libcfs/hash.c:1148 Apr 11 00:01:22 centos6-1 kernel: in_atomic(): 1, irqs_disabled(): 0, pid: 17878, name: rmmod Apr 11 00:01:22 centos6-1 kernel: 1 lock held by rmmod/17878: Apr 11 00:01:22 centos6-1 kernel: #0: (&amp;new_bkts[i]-&gt;hsb_lock.rw){......}, at: [&lt;ffffffffa01c030d&gt;] cfs_hash_rw_lock+0xd/0x20 [libcfs] Apr 11 00:01:22 centos6-1 kernel: CPU: 7 PID: 17878 Comm: rmmod Tainted: GF W O-------------- 3.10.0-debug #5 Apr 11 00:01:22 centos6-1 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Apr 11 00:01:22 centos6-1 kernel: ffff880077774740 0000000096882851 ffff880091d93d80 ffffffff816ccb68 Apr 11 00:01:22 centos6-1 kernel: ffff880091d93d98 ffffffff810a8fd9 0000000000000003 ffff880091d93de8 Apr 11 00:01:22 centos6-1 kernel: ffffffffa01c45f3 0000000096882851 ffff88007aeebcc0 ffff880000000008 Apr 11 00:01:22 centos6-1 kernel: Call Trace: Apr 11 00:01:22 centos6-1 kernel: [&lt;ffffffff816ccb68&gt;] dump_stack+0x19/0x1b Apr 11 00:01:22 centos6-1 kernel: [&lt;ffffffff810a8fd9&gt;] __might_sleep+0xe9/0x110 Apr 11 00:01:22 centos6-1 kernel: [&lt;ffffffffa01c45f3&gt;] cfs_hash_putref+0x233/0x500 [libcfs] Apr 11 00:01:22 centos6-1 kernel: [&lt;ffffffffa05ac15f&gt;] nm_member_delete_hash+0x2f/0x40 [ptlrpc] Apr 11 00:01:22 centos6-1 kernel: [&lt;ffffffffa05a8014&gt;] nodemap_destroy+0xa4/0x1f0 [ptlrpc] Apr 11 00:01:22 centos6-1 kernel: [&lt;ffffffffa05a818d&gt;] nodemap_putref+0x2d/0xa0 [ptlrpc] Apr 11 00:01:22 centos6-1 kernel: [&lt;ffffffffa05a9235&gt;] nodemap_hs_put_locked+0x15/0x20 [ptlrpc] Apr 11 00:01:22 centos6-1 kernel: [&lt;ffffffffa01c0d15&gt;] cfs_hash_bd_del_locked+0x75/0xf0 [libcfs] Apr 11 00:01:22 centos6-1 kernel: [&lt;ffffffffa01c458c&gt;] cfs_hash_putref+0x1cc/0x500 [libcfs] Apr 11 00:01:22 centos6-1 kernel: [&lt;ffffffffa05a926a&gt;] nodemap_cleanup_all+0x2a/0x30 [ptlrpc] Apr 11 00:01:22 centos6-1 kernel: [&lt;ffffffffa05a927e&gt;] nodemap_mod_exit+0xe/0x20 [ptlrpc] Apr 11 00:01:22 centos6-1 kernel: [&lt;ffffffffa05aebc2&gt;] ptlrpc_exit+0x9/0x447 [ptlrpc] Apr 11 00:01:22 centos6-1 kernel: [&lt;ffffffff810e769b&gt;] SyS_delete_module+0x16b/0x2d0 Apr 11 00:01:22 centos6-1 kernel: [&lt;ffffffff816df369&gt;] system_call_fastpath+0x16/0x1b </pre> </div></div> <p>This is because cfs_hash_putref might call cfs_hash_destroy if that was the last reference relesed and that might sleep.</p> <p>Aha! so this is also the reason for quite a bunch of spinlock bugs I was having where spinlock debugging code was complaining that a lock was taken on one cpu, but released on another, I guess:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Apr 11 00:30:38 centos6-7 kernel: BUG: sleeping function called from invalid context at /home/green/git/lustre-release/libcfs/libcfs/hash.c:1148 Apr 11 00:30:38 centos6-7 kernel: in_atomic(): 1, irqs_disabled(): 0, pid: 3620, name: rmmod Apr 11 00:30:38 centos6-7 kernel: 1 lock held by rmmod/3620: Apr 11 00:30:38 centos6-7 kernel: #0: (&amp;new_bkts[i]-&gt;hsb_lock.rw){......}, at: [&lt;ffffffffa014e30d&gt;] cfs_hash_rw_lock+0xd/0x20 [libcfs] Apr 11 00:30:38 centos6-7 kernel: CPU: 2 PID: 3620 Comm: rmmod Tainted: GF W O-------------- 3.10.0-debug #5 Apr 11 00:30:38 centos6-7 kernel: Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 Apr 11 00:30:38 centos6-7 kernel: ffff88003ae74780 00000000d7e1d1ee ffff88008473bd80 ffffffff816ccb68 Apr 11 00:30:38 centos6-7 kernel: ffff88008473bd98 ffffffff810a8fd9 0000000000000003 ffff88008473bde8 Apr 11 00:30:38 centos6-7 kernel: ffffffffa01525f3 00000000d7e1d1ee ffff88008d62bc00 ffff880000000008 Apr 11 00:30:38 centos6-7 kernel: Call Trace: Apr 11 00:30:38 centos6-7 kernel: [&lt;ffffffff816ccb68&gt;] dump_stack+0x19/0x1b Apr 11 00:30:38 centos6-7 kernel: [&lt;ffffffff810a8fd9&gt;] __might_sleep+0xe9/0x110 Apr 11 00:30:38 centos6-7 kernel: [&lt;ffffffffa01525f3&gt;] cfs_hash_putref+0x233/0x500 [libcfs] Apr 11 00:30:38 centos6-7 kernel: [&lt;ffffffffa05cb15f&gt;] nm_member_delete_hash+0x2f/0x40 [ptlrpc] Apr 11 00:30:38 centos6-7 kernel: [&lt;ffffffffa05c7014&gt;] nodemap_destroy+0xa4/0x1f0 [ptlrpc] Apr 11 00:30:38 centos6-7 kernel: [&lt;ffffffffa05c718d&gt;] nodemap_putref+0x2d/0xa0 [ptlrpc] Apr 11 00:30:38 centos6-7 kernel: [&lt;ffffffffa05c8235&gt;] nodemap_hs_put_locked+0x15/0x20 [ptlrpc] Apr 11 00:30:38 centos6-7 kernel: [&lt;ffffffffa014ed15&gt;] cfs_hash_bd_del_locked+0x75/0xf0 [libcfs] Apr 11 00:30:38 centos6-7 kernel: [&lt;ffffffffa015258c&gt;] cfs_hash_putref+0x1cc/0x500 [libcfs] Apr 11 00:30:38 centos6-7 kernel: [&lt;ffffffffa05c826a&gt;] nodemap_cleanup_all+0x2a/0x30 [ptlrpc] Apr 11 00:30:38 centos6-7 kernel: [&lt;ffffffffa05c827e&gt;] nodemap_mod_exit+0xe/0x20 [ptlrpc] Apr 11 00:30:38 centos6-7 kernel: [&lt;ffffffffa05cdbc2&gt;] ptlrpc_exit+0x9/0x447 [ptlrpc] Apr 11 00:30:38 centos6-7 kernel: [&lt;ffffffff810e769b&gt;] SyS_delete_module+0x16b/0x2d0 Apr 11 00:30:38 centos6-7 kernel: [&lt;ffffffff816df369&gt;] system_call_fastpath+0x16/0x1b Apr 11 00:30:38 centos6-7 kernel: BUG: scheduling while atomic: rmmod/3620/0x10000002 Apr 11 00:30:38 centos6-7 kernel: 1 lock held by rmmod/3620: Apr 11 00:30:38 centos6-7 kernel: #0: (&amp;new_bkts[i]-&gt;hsb_lock.rw){......}, at: [&lt;ffffffffa014e30d&gt;] cfs_hash_rw_lock+0xd/0x20 [libcfs] Apr 11 00:30:38 centos6-7 kernel: Modules linked in: ptlrpc(OF-) obdclass(OF) ksocklnd(OF) lnet(OF) libcfs(OF) loop(F) jbd2(F) sha512_generic(F) mbcache(F) rpcsec_gss_krb5(F) ata_generic(F) pata_acpi(F) ttm(F) drm_kms_helper(F) drm(F) i2c_piix4(F) ata_piix(F) virtio_blk(F) i2c_core(F) virtio_balloon(F) virtio_console(F) floppy(F) libata(F) serio_raw(F) pcspkr(F) [last unloaded: lquota] Apr 11 00:30:38 centos6-7 kernel: CPU: 2 PID: 3620 Comm: rmmod Tainted: GF W O-------------- 3.10.0-debug #5 Apr 11 00:30:38 centos6-7 kernel: Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 Apr 11 00:30:38 centos6-7 kernel: ffff88003ae74780 00000000d7e1d1ee ffff88008473bcf8 ffffffff816ccb68 Apr 11 00:30:38 centos6-7 kernel: ffff88008473bd10 ffffffff816c827a ffff88008473bfd8 ffff88008473bd70 Apr 11 00:30:38 centos6-7 kernel: ffffffff816d3d1c ffff88003ae74780 ffff88008473bfd8 ffff88008473bfd8 Apr 11 00:30:38 centos6-7 kernel: Call Trace: Apr 11 00:30:38 centos6-7 kernel: [&lt;ffffffff816ccb68&gt;] dump_stack+0x19/0x1b Apr 11 00:30:38 centos6-7 kernel: [&lt;ffffffff816c827a&gt;] __schedule_bug+0x59/0x68 Apr 11 00:30:38 centos6-7 kernel: [&lt;ffffffff816d3d1c&gt;] __schedule+0x75c/0x810 Apr 11 00:30:38 centos6-7 kernel: [&lt;ffffffff810acab6&gt;] __cond_resched+0x26/0x30 Apr 11 00:30:38 centos6-7 kernel: [&lt;ffffffff816d41ea&gt;] _cond_resched+0x3a/0x50 Apr 11 00:30:38 centos6-7 kernel: [&lt;ffffffffa01525f8&gt;] cfs_hash_putref+0x238/0x500 [libcfs] Apr 11 00:30:38 centos6-7 kernel: [&lt;ffffffffa05cb15f&gt;] nm_member_delete_hash+0x2f/0x40 [ptlrpc] Apr 11 00:30:38 centos6-7 kernel: [&lt;ffffffffa05c7014&gt;] nodemap_destroy+0xa4/0x1f0 [ptlrpc] Apr 11 00:30:38 centos6-7 kernel: [&lt;ffffffffa05c718d&gt;] nodemap_putref+0x2d/0xa0 [ptlrpc] Apr 11 00:30:38 centos6-7 kernel: [&lt;ffffffffa05c8235&gt;] nodemap_hs_put_locked+0x15/0x20 [ptlrpc] Apr 11 00:30:38 centos6-7 kernel: [&lt;ffffffffa014ed15&gt;] cfs_hash_bd_del_locked+0x75/0xf0 [libcfs] Apr 11 00:30:38 centos6-7 kernel: [&lt;ffffffffa015258c&gt;] cfs_hash_putref+0x1cc/0x500 [libcfs] Apr 11 00:30:38 centos6-7 kernel: [&lt;ffffffffa05c826a&gt;] nodemap_cleanup_all+0x2a/0x30 [ptlrpc] Apr 11 00:30:38 centos6-7 kernel: [&lt;ffffffffa05c827e&gt;] nodemap_mod_exit+0xe/0x20 [ptlrpc] Apr 11 00:30:38 centos6-7 kernel: [&lt;ffffffffa05cdbc2&gt;] ptlrpc_exit+0x9/0x447 [ptlrpc] Apr 11 00:30:38 centos6-7 kernel: [&lt;ffffffff810e769b&gt;] SyS_delete_module+0x16b/0x2d0 Apr 11 00:30:38 centos6-7 kernel: [&lt;ffffffff816df369&gt;] system_call_fastpath+0x16/0x1b Apr 11 00:30:38 centos6-7 kernel: BUG: rwlock wrong CPU on CPU#3, rmmod/3620, ffff88005ca55dc0 Apr 11 00:30:38 centos6-7 kernel: CPU: 3 PID: 3620 Comm: rmmod Tainted: GF W O-------------- 3.10.0-debug #5 Apr 11 00:30:38 centos6-7 kernel: Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 Apr 11 00:30:38 centos6-7 kernel: ffffffff81a971b4 00000000d7e1d1ee ffff88008473be10 ffffffff816ccb68 Apr 11 00:30:38 centos6-7 kernel: ffff88008473be30 ffffffff816ccc6f ffff88005ca55dc0 ffff88005ca57e00 Apr 11 00:30:38 centos6-7 kernel: ffff88008473be48 ffffffff813707c0 ffff88005ca55dc0 ffff88008473be60 Apr 11 00:30:38 centos6-7 kernel: Call Trace: Apr 11 00:30:38 centos6-7 kernel: [&lt;ffffffff816ccb68&gt;] dump_stack+0x19/0x1b Apr 11 00:30:38 centos6-7 kernel: [&lt;ffffffff816ccc6f&gt;] rwlock_bug+0x4e/0x53 Apr 11 00:30:38 centos6-7 kernel: [&lt;ffffffff813707c0&gt;] do_raw_write_unlock+0x60/0x80 Apr 11 00:30:38 centos6-7 kernel: [&lt;ffffffff816d6107&gt;] _raw_write_unlock+0x27/0x40 Apr 11 00:30:38 centos6-7 kernel: [&lt;ffffffffa014e2ed&gt;] cfs_hash_rw_unlock+0xd/0x20 [libcfs] Apr 11 00:30:38 centos6-7 kernel: [&lt;ffffffffa01525e0&gt;] cfs_hash_putref+0x220/0x500 [libcfs] Apr 11 00:30:38 centos6-7 kernel: [&lt;ffffffffa05c826a&gt;] nodemap_cleanup_all+0x2a/0x30 [ptlrpc] Apr 11 00:30:38 centos6-7 kernel: [&lt;ffffffffa05c827e&gt;] nodemap_mod_exit+0xe/0x20 [ptlrpc] Apr 11 00:30:38 centos6-7 kernel: [&lt;ffffffffa05cdbc2&gt;] ptlrpc_exit+0x9/0x447 [ptlrpc] Apr 11 00:30:38 centos6-7 kernel: [&lt;ffffffff810e769b&gt;] SyS_delete_module+0x16b/0x2d0 Apr 11 00:30:38 centos6-7 kernel: [&lt;ffffffff816df369&gt;] system_call_fastpath+0x16/0x1b </pre> </div></div> <p>I rebased the patch from Kit. Oh I see bugs remain.</p> <p>I realized that the patch I submitted was actually making things worse, as you aren't allowed to call cfs_hash_for_each while holding a spinlock. I instead modified nodemap_cleanup_all to convert the hash to a list and then iterate the list, calling nodemap_putref on each nodemap. Does this sound reasonable?</p> <p>Hm, so I have several patches from you:<br/> 93ccc4a <a href="https://jira.whamcloud.com/browse/LU-5092" title="nodemap: transfer idmaps between MGS, MDT, OST" class="issue-link" data-issue-key="LU-5092"><del>LU-5092</del></a> nodemap: add structure to hold nodemap config<br/> e4d2197 <a href="https://jira.whamcloud.com/browse/LU-5092" title="nodemap: transfer idmaps between MGS, MDT, OST" class="issue-link" data-issue-key="LU-5092"><del>LU-5092</del></a> nodemap: convert member hash to a list<br/> 3f0bb87 <a href="https://jira.whamcloud.com/browse/LU-6409" title="sleeping while atomic in nodemap_destroy" class="issue-link" data-issue-key="LU-6409"><del>LU-6409</del></a> nodemap: don't destroy nodemaps in cfs_hash_for_each</p> <p>And this is not enough.</p> <p>I still get this in the logs:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>leeping function called from invalid context at kernel/mutex.c:576 May 30 22:19:09 centos6-0 kernel: in_atomic(): 1, irqs_disabled(): 0, pid: 9196, name: mdt00_002 May 30 22:19:09 centos6-0 kernel: 2 locks held by mdt00_002/9196: May 30 22:19:09 centos6-0 kernel: #0: (active_config_lock){......}, at: [&lt;fffff fffa05cb147&gt;] nodemap_lock_active_ranges+0x17/0x30 [ptlrpc] May 30 22:19:09 centos6-0 kernel: #1: (&amp;config-&gt;nmc_range_tree_lock){......}, a t: [&lt;ffffffffa05cb157&gt;] nodemap_lock_active_ranges+0x27/0x30 [ptlrpc] PID: 9196 Comm: mdt00_002 Tainted: GF W O-------------- 3.10.0-debug # re name: Red Hat KVM, BIOS 0.5.1 01/01/2011 0019366680 00000000b6a08141 ffff88003ac9b9a0 ffffffff816ccb68 003ac9b9b8 ffffffff810a8fd9 ffff8800aa50 0e00 ffff88003ac9ba40 ff816d0dd5 0000000000000000 0000000000000002 0000000000000000 race: ffff816ccb68&gt;] dump_stack+0x19/0x1b ffff810a8fd9&gt;] __might_sleep+0xe9/0x110 ffff816d0dd5&gt;] mutex_lock_nested+0x55/0x490 ffffa05cb157&gt;] ? nodemap_lock_active_ranges+0x27/0x30 [ptlrpc] ffffa05cfbe8&gt;] nm_member_add+0x78/0x180 [ptlrpc] ffffa05cc80d&gt;] nodemap_add_member+0x2d/0x50 [ptlrpc] ffffa0b147f3&gt;] mdt_obd_connect+0x3b3/0x760 [mdt] ffffa01c4a87&gt;] ? libcfs_debug_msg+0x57/0x80 [libcfs] ffffa051ded3&gt;] target_handle_connect+0x12c3/0x2be0 [ptlrpc] ffff81363263&gt;] ? number.isra.2+0x323/0x360 ffffa02dbcbd&gt;] ? keys_fill+0x1d/0x130 [obdclass] ffffa02dbcd2&gt;] ? keys_fill+0x32/0x130 [obdclass] ffffa05b445f&gt;] tgt_request_handle+0x3ef/0x1070 [ptlrpc] ffffa0565c01&gt;] ptlrpc_server_handle_request+0x231/0xab0 [ptlrpc] ffffa0563768&gt;] ? ptlrpc_wait_event+0xb8/0x370 [ptlrpc] ffffa0569b70&gt;] ptlrpc_main+0xae0/0x1ee0 [ptlrpc] ffffa0569090&gt;] ? ptlrpc_register_service+0xf30/0xf30 [ptlrpc] ffff8109c00a&gt;] kthread+0xea/0xf0 ffff8109bf20&gt;] ? kthread_create_on_node+0x140/0x140 ffff816df2bc&gt;] ret_from_fork+0x7c/0xb0 ffff8109bf20&gt;] ? kthread_create_on_node+0x140/0x140 </pre> </div></div> <p>seemingly from every target.</p> <p>Ah ok, I see what's going on, I'll update the patches I have in gerrit.</p> <p>Can you describe what tests/config options you are running to get these warnings? I am not getting them on my build and I don't want to waste your time if I can catch them before I submit. </p> <p>You need to use rhel7 for this, there's a new kernel option you need to enable:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">CONFIG_DEBUG_ATOMIC_SLEEP=y </pre> </div></div> <p>Turn this on and run something like sanity, you'll get a bunch of warnings. Also apply patch from <a href="https://jira.whamcloud.com/browse/LU-6447" title="mdt_identity_upcall calls sleeping function under rwlock" class="issue-link" data-issue-key="LU-6447"><del>LU-6447</del></a> (I know it's imperfect) to quiet down unrelated warnings.</p> <p>I finally got the debug kernel working and reproduced the warning you saw. Should I report any other non-nodemap warnings I find? There is a lock inversion warning with s_dquot.dqio_mutex and i_data_sem that I couldn't find any Jira tickets about, but I'm only getting it during mkfs so far. </p> <p>Sure, feel free to report the other ones you see too.</p> <p>Ok, I reported the potential lock inversion as <a href="https://jira.whamcloud.com/browse/LU-6700" title="possible lock inversion during mkfs" class="issue-link" data-issue-key="LU-6700"><del>LU-6700</del></a>.</p> <p>I updated the nodemap patches and no longer get the warning messages.</p> <p>I tried the three patches from <a href="https://jira.whamcloud.com/browse/LU-5092" title="nodemap: transfer idmaps between MGS, MDT, OST" class="issue-link" data-issue-key="LU-5092"><del>LU-5092</del></a>plus the one patch from this ticket and hit a deadlock when running sanity 133g:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[15921.648643] LustreError: 7937:0:(lproc_fid.c:182:lprocfs_server_fid_width_seq_write()) Skipped 17 previous similar messages [16080.884140] INFO: task badarea_io:7981 blocked for more than 120 seconds. [16080.884666] Tainted: P --------------- 2.6.32-rhe6.6-debug #1 [16080.885381] "echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. [16080.886109] badarea_io D 0000000000000002 2792 7981 7003 0x00000000 [16080.886582] ffff880046b5fd78 0000000000000086 0000000000000000 ffffffff8114e4a0 [16080.887360] 0000000000000000 ffffffff8104e658 00000e7b1ce2fe1b ffff880046b5fd58 [16080.888121] ffffffffa1395a20 00000001003b97ab ffff88005f2708f8 ffff880046b5ffd8 [16080.888904] Call Trace: [16080.889234] [&lt;ffffffff8114e4a0&gt;] ? handle_pte_fault+0x90/0xa50 [16080.889663] [&lt;ffffffff8104e658&gt;] ? __change_page_attr_set_clr+0x808/0xcc0 [16080.890154] [&lt;ffffffffa1395a20&gt;] ? lprocfs_single_release+0x0/0x10 [obdclass] [16080.890876] [&lt;ffffffff8151ff50&gt;] __mutex_lock_slowpath+0x120/0x2e0 [16080.891364] [&lt;ffffffffa166d8d0&gt;] ? nodemap_del_nodemap_write+0x0/0xc0 [ptlrpc] [16080.892098] [&lt;ffffffff81520141&gt;] mutex_lock+0x31/0x50 [16080.892598] [&lt;ffffffffa16699bc&gt;] nodemap_del+0x3c/0x1c0 [ptlrpc] [16080.893118] [&lt;ffffffffa166d8d0&gt;] ? nodemap_del_nodemap_write+0x0/0xc0 [ptlrpc] [16080.896192] [&lt;ffffffffa166d975&gt;] nodemap_del_nodemap_write+0xa5/0xc0 [ptlrpc] [16080.896810] [&lt;ffffffff811fb8c5&gt;] proc_reg_write+0x85/0xc0 [16080.897094] [&lt;ffffffff8118f4a8&gt;] vfs_write+0xb8/0x1a0 [16080.897388] [&lt;ffffffff8118fde1&gt;] sys_write+0x51/0x90 [16080.897719] [&lt;ffffffff8100b0b2&gt;] system_call_fastpath+0x16/0x1b </pre> </div></div> <p>ah, I see it's likely a bug in <a href="http://review.whamcloud.com/#/c/11813/8" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/11813/8</a></p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/14275/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/14275/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6409" title="sleeping while atomic in nodemap_destroy" class="issue-link" data-issue-key="LU-6409"><del>LU-6409</del></a> nodemap: don't destroy nodemaps in cfs_hash_for_each<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 6299d78713ef2b6f69f6dd046256aa91a85147a8</p> <p>Landed for 2.8</p> Related LU-5092 LU-7811 Development Rank 1|hzx9n3: Rank (Obsolete) 9223372036854775807 Severity