https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4474 Lustre <p>When I ran quotacheck directly on ldiskfs, the process stuck. Following is the dump stack. The ext4_quota_off() function added by ext4-quota-force-block-alloc-quotaoff.patch tried to grab read lock of sb-&gt;s_umount. But it stuck there because it was holding write lock of sb-&gt;s_umount.</p> <p>COMMAND: "quotacheck" <br/> TASK: ffff88010286aae0 <span class="error">&#91;THREAD_INFO: ffff880102610000&#93;</span><br/> CPU: 0<br/> STATE: TASK_UNINTERRUPTIBLE <br/> crash&gt; bt<br/> PID: 21560 TASK: ffff88010286aae0 CPU: 0 COMMAND: "quotacheck" <br/> #0 <span class="error">&#91;ffff880102611b60&#93;</span> schedule at ffffffff8150c432<br/> #1 <span class="error">&#91;ffff880102611c28&#93;</span> rwsem_down_failed_common at ffffffff8150e9e5<br/> #2 <span class="error">&#91;ffff880102611c88&#93;</span> rwsem_down_read_failed at ffffffff8150eb76<br/> #3 <span class="error">&#91;ffff880102611cc8&#93;</span> call_rwsem_down_read_failed at ffffffff81281b94<br/> #4 <span class="error">&#91;ffff880102611d30&#93;</span> ldiskfs_quota_off at ffffffffa02fe115 <span class="error">&#91;ldiskfs&#93;</span><br/> #5 <span class="error">&#91;ffff880102611d90&#93;</span> ldiskfs_remount at ffffffffa02fd41d <span class="error">&#91;ldiskfs&#93;</span><br/> #6 <span class="error">&#91;ffff880102611e60&#93;</span> do_remount_sb at ffffffff811825bc<br/> #7 <span class="error">&#91;ffff880102611ea0&#93;</span> do_mount at ffffffff811a3915<br/> #8 <span class="error">&#91;ffff880102611f20&#93;</span> sys_mount at ffffffff811a3c10<br/> #9 <span class="error">&#91;ffff880102611f80&#93;</span> system_call_fastpath at ffffffff8100b072<br/> RIP: 00007f97e757ee0a RSP: 00007fffe66e9388 RFLAGS: 00010297<br/> RAX: 00000000000000a5 RBX: ffffffff8100b072 RCX: 0000000000000001<br/> RDX: 00007f97e9f02940 RSI: 00007f97e9f01740 RDI: 0000000000000000<br/> RBP: 00007f97e822b4a0 R8: 0000000000000000 R9: 0000000000000000<br/> R10: 00000000c0ed0021 R11: 0000000000000206 R12: 00007f97e7ea7047<br/> R13: 00007f97e7ea5c00 R14: 0000000000000001 R15: 00007f97e822b4a0<br/> ORIG_RAX: 00000000000000a5 CS: 0033 SS: 002b<br/> crash&gt; </p> <p>Following is how to reproduce it. Please note that the version of quota-tools is 4.0.1. Somehow quota-tools of version 3.17 refuses to quotacheck a ldiskfs mount point.</p> <p><span class="error">&#91;root@vm1 quota-tools&#93;</span># mount -t ldiskfs /dev/sdb3 -o usrquota /mnt/<br/> <span class="error">&#91;root@vm1 quota-tools&#93;</span># ./quotacheck /mnt/ -f<br/> .</p> LU-4474

deadlock of ldiskfs_quota_off()

Bug Major Resolved Fixed Niu Yawei Li Xi ldiskfs Sat, 11 Jan 2014 15:14:09 +0000 Thu, 29 Sep 2016 17:52:18 +0000 Thu, 29 Sep 2016 17:52:18 +0000 Lustre 2.9.0 0 7 <p>I am sure, but maybe this problem could be fixed by removing the down_read(&amp;sb-&gt;s_umount)? It seems that a write lock of sb-&gt;s_umount is alway being hold when ext4_quota_off() is called. Please correct me if I am wrong. <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>NIu</p> <p>Could you please comment?</p> <p>Thanks</p> <p>Peter</p> <p>Hi, LiXi</p> <p>What kernel version did you use? I didn't see how ldiskfs_remount calls ldiskfs_quota_off in my kernel (2.6.32-279).</p> <p>BTW: Why do you need a quotacheck on ldiskfs? Isn't it obsoleted?</p> <p>Hi Yawei,</p> <p>I am using linux-2.6.32-358.18.1.el6.x86_64. vfs_dq_off() is called in ext4_remount().</p> <p>static int ext4_remount(struct super_block *sb, int *flags, char *data)<br/> {<br/> ...<br/> if ((*flags &amp; MS_RDONLY) != (sb-&gt;s_flags &amp; MS_RDONLY) ||<br/> n_blocks_count &gt; ext4_blocks_count(es)) {<br/> if (sbi-&gt;s_mount_flags &amp; EXT4_MF_FS_ABORTED) </p> { err = -EROFS; goto restore_opts; } <p> if (*flags &amp; MS_RDONLY) {<br/> err = vfs_dq_off(sb, 1);<br/> if (err &lt; 0 &amp;&amp; err != -ENOSYS) </p> { err = -EBUSY; goto restore_opts; } <p> /*</p> <ul> <li>First of all, the unconditional stuff we have to do</li> <li>to disable replay of the journal when we next remount<br/> */<br/> sb-&gt;s_flags |= MS_RDONLY;<br/> ...</li> </ul> <p>Yeah, I understand this is not a common operation for Lustre. I thought we can use ldiskfs as a normal file system, so I ran quota utility directly on ldiskfs. If it is not proper to do so, do you have any suggestions about how to check the quota mechanism of ldiskfs? As you know, I am working on a quota stuff. Quota codes of ldiskfs have to be updated for support. Thanks!</p> <p>quota accounting is always enabled on ldiskfs (see ext4-quota-first-class.patch), so quotacheck is no longer needed anymore. (if the filesystem is upgraded from old ldiskfs which doesn't have EXT4_FEATURE_RO_COMPAT_QUOTA feature, the quota accounting will be udpated automatically when you enable the feature by 'tune2fs -O quota')</p> <p>Ah, thank you! Do you think this problem could be trigger by any operation from Lustre or not? I think it would be better to fix this problem anyway if it is not too difficult.</p> <p>I think it could probably be triggered in Lustre, but I didn't see why it was never reported before.</p> <p>Looks it can't be fixed simply be removing down_read(s_umount), because I didn't see s_umount is held in quotactl path (do_quotactl()). Could you try to search to see if it's already fixed in upstream?</p> <p>Looks it has been fixed in upstream:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">It must be already acquired by caller. Fix lock_dep complain. Signed-off-by: Dmitry Monakhov &lt;dmon...@gmail.com&gt; Reviewed-by: Jan Kara &lt;ja...@suse.cz&gt; diff --git a/fs/ext4/<span class="code-keyword">super</span>.c b/fs/ext4/<span class="code-keyword">super</span>.c index e59eb37..062d1bc 100644 --- a/fs/ext4/<span class="code-keyword">super</span>.c +++ b/fs/ext4/<span class="code-keyword">super</span>.c @@ -4107,12 +4107,10 @@ <span class="code-keyword">static</span> <span class="code-object">int</span> ext4_quota_on(struct super_block *sb, <span class="code-object">int</span> type, <span class="code-object">int</span> format_id, <span class="code-keyword">static</span> <span class="code-object">int</span> ext4_quota_off(struct super_block *sb, <span class="code-object">int</span> type) { - <span class="code-comment">/* Force all delayed allocation blocks to be allocated */</span> - <span class="code-keyword">if</span> (test_opt(sb, DELALLOC)) { - down_read(&amp;sb-&gt;s_umount); + /* Force all delayed allocation blocks to be allocated. + * Caller already holds s_umount sem */ + <span class="code-keyword">if</span> (test_opt(sb, DELALLOC)) sync_filesystem(sb); - up_read(&amp;sb-&gt;s_umount); - } <span class="code-keyword">return</span> dquot_quota_off(sb, type); } </pre> </div></div> <p>I'll backport it to Lustre, thank you, LiXi.</p> <p>Ah, Yawei, thank you so much for investigte this!</p> <p><a href="http://review.whamcloud.com/8828" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8828</a></p> <p>Hi Yawei,</p> <p>I found following messages when running mkfs.lustre on a server which has applied your patch. It seems sync_filesystem() complained that: WARN_ON(!rwsem_is_locked(&amp;sb-&gt;s_umount));</p> <p>Do you think this related to the patch?</p> <p>Thanks!</p> <p>Jan 16 13:36:53 vm1 kernel: LDISKFS-fs (sdb3): mounted filesystem with ordered data mode. quota=on. Opts: <br/> Jan 16 13:36:53 vm1 kernel: -----------<del>[ cut here ]</del>-----------<br/> Jan 16 13:36:53 vm1 kernel: WARNING: at fs/sync.c:64 sync_filesystem+0x63/0x70() (Tainted: G W --------------- )<br/> Jan 16 13:36:53 vm1 kernel: Hardware name: KVM<br/> Jan 16 13:36:53 vm1 kernel: Modules linked in: ldiskfs sha512_generic sha256_generic bridge stp llc nfs lockd fscache auth_rpcgss nfs_acl sunrpc ipv6 microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core sg ext4 jbd2 mbcache sd_mod crc_t10dif pata_acpi ata_generic ata_piix virtio_pci virtio_ring virtio dm_mirror dm_region_hash dm_log dm_mod <span class="error">&#91;last unloaded: libcfs&#93;</span><br/> Jan 16 13:36:53 vm1 kernel: Pid: 19717, comm: mkfs.lustre Tainted: G W --------------- 2.6.32 #4<br/> Jan 16 13:36:53 vm1 kernel: Call Trace:<br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffff8106e3e7&gt;&#93;</span> ? warn_slowpath_common+0x87/0xc0<br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffff8106e43a&gt;&#93;</span> ? warn_slowpath_null+0x1a/0x20<br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffff811b1893&gt;&#93;</span> ? sync_filesystem+0x63/0x70<br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffffa02e7048&gt;&#93;</span> ? ldiskfs_quota_off+0x218/0x220 <span class="error">&#91;ldiskfs&#93;</span><br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffff81183496&gt;&#93;</span> ? deactivate_super+0x46/0x80<br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffff811a17ef&gt;&#93;</span> ? mntput_no_expire+0xbf/0x110<br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffff811a225b&gt;&#93;</span> ? sys_umount+0x7b/0x3a0<br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffff8100b072&gt;&#93;</span> ? system_call_fastpath+0x16/0x1b<br/> Jan 16 13:36:53 vm1 kernel: --<del>[ end trace 7cd192156324e646 ]</del>--<br/> Jan 16 13:36:53 vm1 kernel: -----------<del>[ cut here ]</del>-----------<br/> Jan 16 13:36:53 vm1 kernel: WARNING: at fs/fs-writeback.c:1051 writeback_inodes_sb_nr+0x94/0xa0() (Tainted: G W --------------- )<br/> Jan 16 13:36:53 vm1 kernel: Hardware name: KVM<br/> Jan 16 13:36:53 vm1 kernel: Modules linked in: ldiskfs sha512_generic sha256_generic bridge stp llc nfs lockd fscache auth_rpcgss nfs_acl sunrpc ipv6 microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core sg ext4 jbd2 mbcache sd_mod crc_t10dif pata_acpi ata_generic ata_piix virtio_pci virtio_ring virtio dm_mirror dm_region_hash dm_log dm_mod <span class="error">&#91;last unloaded: libcfs&#93;</span><br/> Jan 16 13:36:53 vm1 kernel: Pid: 19717, comm: mkfs.lustre Tainted: G W --------------- 2.6.32 #4<br/> Jan 16 13:36:53 vm1 kernel: Call Trace:<br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffff8106e3e7&gt;&#93;</span> ? warn_slowpath_common+0x87/0xc0<br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffff8106e43a&gt;&#93;</span> ? warn_slowpath_null+0x1a/0x20<br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffff811ab704&gt;&#93;</span> ? writeback_inodes_sb_nr+0x94/0xa0<br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffff811ab753&gt;&#93;</span> ? writeback_inodes_sb+0x43/0x50<br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffff811b162e&gt;&#93;</span> ? __sync_filesystem+0x3e/0x90<br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffff811b186a&gt;&#93;</span> ? sync_filesystem+0x3a/0x70<br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffffa02e7048&gt;&#93;</span> ? ldiskfs_quota_off+0x218/0x220 <span class="error">&#91;ldiskfs&#93;</span><br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffff81183496&gt;&#93;</span> ? deactivate_super+0x46/0x80<br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffff811a17ef&gt;&#93;</span> ? mntput_no_expire+0xbf/0x110<br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffff811a225b&gt;&#93;</span> ? sys_umount+0x7b/0x3a0<br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffff8100b072&gt;&#93;</span> ? system_call_fastpath+0x16/0x1b<br/> Jan 16 13:36:53 vm1 kernel: --<del>[ end trace 7cd192156324e647 ]</del>--<br/> Jan 16 13:36:53 vm1 kernel: -----------<del>[ cut here ]</del>-----------<br/> Jan 16 13:36:53 vm1 kernel: WARNING: at fs/fs-writeback.c:548 wb_writeback+0x339/0x3f0() (Tainted: G W --------------- )<br/> Jan 16 13:36:53 vm1 kernel: Hardware name: KVM<br/> Jan 16 13:36:53 vm1 kernel: Modules linked in: ldiskfs sha512_generic sha256_generic bridge stp llc nfs lockd fscache auth_rpcgss nfs_acl sunrpc ipv6 microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core sg ext4 jbd2 mbcache sd_mod crc_t10dif pata_acpi ata_generic ata_piix virtio_pci virtio_ring virtio dm_mirror dm_region_hash dm_log dm_mod <span class="error">&#91;last unloaded: libcfs&#93;</span><br/> Jan 16 13:36:53 vm1 kernel: Pid: 18643, comm: flush-8:16 Tainted: G W --------------- 2.6.32 #4<br/> Jan 16 13:36:53 vm1 kernel: Call Trace:<br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffff8106e3e7&gt;&#93;</span> ? warn_slowpath_common+0x87/0xc0<br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffff8106e43a&gt;&#93;</span> ? warn_slowpath_null+0x1a/0x20<br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffff811aca49&gt;&#93;</span> ? wb_writeback+0x339/0x3f0<br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffff8150c480&gt;&#93;</span> ? thread_return+0x4e/0x76e<br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffff81081be2&gt;&#93;</span> ? del_timer_sync+0x22/0x30<br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffff811acbbb&gt;&#93;</span> ? wb_do_writeback+0xbb/0x240<br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffff811acda3&gt;&#93;</span> ? bdi_writeback_task+0x63/0x1b0<br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffff81096c67&gt;&#93;</span> ? bit_waitqueue+0x17/0xd0<br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffff8113c1d0&gt;&#93;</span> ? bdi_start_fn+0x0/0x100<br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffff8113c256&gt;&#93;</span> ? bdi_start_fn+0x86/0x100<br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffff8113c1d0&gt;&#93;</span> ? bdi_start_fn+0x0/0x100<br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffff81096a36&gt;&#93;</span> ? kthread+0x96/0xa0<br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffff8100c0ca&gt;&#93;</span> ? child_rip+0xa/0x20<br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffff810969a0&gt;&#93;</span> ? kthread+0x0/0xa0<br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffff8100c0c0&gt;&#93;</span> ? child_rip+0x0/0x20<br/> Jan 16 13:36:53 vm1 kernel: --<del>[ end trace 7cd192156324e648 ]</del>--<br/> Jan 16 13:36:53 vm1 kernel: -----------<del>[ cut here ]</del>-----------<br/> Jan 16 13:36:53 vm1 kernel: WARNING: at fs/fs-writeback.c:1131 sync_inodes_sb+0x169/0x190() (Tainted: G W --------------- )<br/> Jan 16 13:36:53 vm1 kernel: Hardware name: KVM<br/> Jan 16 13:36:53 vm1 kernel: Modules linked in: ldiskfs sha512_generic sha256_generic bridge stp llc nfs lockd fscache auth_rpcgss nfs_acl sunrpc ipv6 microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core sg ext4 jbd2 mbcache sd_mod crc_t10dif pata_acpi ata_generic ata_piix virtio_pci virtio_ring virtio dm_mirror dm_region_hash dm_log dm_mod <span class="error">&#91;last unloaded: libcfs&#93;</span><br/> Jan 16 13:36:53 vm1 kernel: Pid: 19717, comm: mkfs.lustre Tainted: G W --------------- 2.6.32 #4<br/> Jan 16 13:36:53 vm1 kernel: Call Trace:<br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffff8106e3e7&gt;&#93;</span> ? warn_slowpath_common+0x87/0xc0<br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffff8106e43a&gt;&#93;</span> ? warn_slowpath_null+0x1a/0x20<br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffff811ab8c9&gt;&#93;</span> ? sync_inodes_sb+0x169/0x190<br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffff811b1672&gt;&#93;</span> ? __sync_filesystem+0x82/0x90<br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffff811b187b&gt;&#93;</span> ? sync_filesystem+0x4b/0x70<br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffffa02e7048&gt;&#93;</span> ? ldiskfs_quota_off+0x218/0x220 <span class="error">&#91;ldiskfs&#93;</span><br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffff81183496&gt;&#93;</span> ? deactivate_super+0x46/0x80<br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffff811a17ef&gt;&#93;</span> ? mntput_no_expire+0xbf/0x110<br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffff811a225b&gt;&#93;</span> ? sys_umount+0x7b/0x3a0<br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffff8100b072&gt;&#93;</span> ? system_call_fastpath+0x16/0x1b<br/> Jan 16 13:36:53 vm1 kernel: --<del>[ end trace 7cd192156324e649 ]</del>--<br/> Jan 16 13:36:53 vm1 kernel: -----------<del>[ cut here ]</del>-----------<br/> Jan 16 13:36:53 vm1 kernel: WARNING: at fs/fs-writeback.c:548 wb_writeback+0x339/0x3f0() (Tainted: G W --------------- )<br/> Jan 16 13:36:53 vm1 kernel: Hardware name: KVM<br/> Jan 16 13:36:53 vm1 kernel: Modules linked in: ldiskfs sha512_generic sha256_generic bridge stp llc nfs lockd fscache auth_rpcgss nfs_acl sunrpc ipv6 microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core sg ext4 jbd2 mbcache sd_mod crc_t10dif pata_acpi ata_generic ata_piix virtio_pci virtio_ring virtio dm_mirror dm_region_hash dm_log dm_mod <span class="error">&#91;last unloaded: libcfs&#93;</span><br/> Jan 16 13:36:53 vm1 kernel: Pid: 18643, comm: flush-8:16 Tainted: G W --------------- 2.6.32 #4<br/> Jan 16 13:36:53 vm1 kernel: Call Trace:<br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffff8106e3e7&gt;&#93;</span> ? warn_slowpath_common+0x87/0xc0<br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffff8106e43a&gt;&#93;</span> ? warn_slowpath_null+0x1a/0x20<br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffff811aca49&gt;&#93;</span> ? wb_writeback+0x339/0x3f0<br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffff8150c480&gt;&#93;</span> ? thread_return+0x4e/0x76e<br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffff81081be2&gt;&#93;</span> ? del_timer_sync+0x22/0x30<br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffff811acbbb&gt;&#93;</span> ? wb_do_writeback+0xbb/0x240<br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffff811acda3&gt;&#93;</span> ? bdi_writeback_task+0x63/0x1b0<br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffff81096c67&gt;&#93;</span> ? bit_waitqueue+0x17/0xd0<br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffff8113c1d0&gt;&#93;</span> ? bdi_start_fn+0x0/0x100<br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffff8113c256&gt;&#93;</span> ? bdi_start_fn+0x86/0x100<br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffff8113c1d0&gt;&#93;</span> ? bdi_start_fn+0x0/0x100<br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffff81096a36&gt;&#93;</span> ? kthread+0x96/0xa0<br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffff8100c0ca&gt;&#93;</span> ? child_rip+0xa/0x20<br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffff810969a0&gt;&#93;</span> ? kthread+0x0/0xa0<br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffff8100c0c0&gt;&#93;</span> ? child_rip+0x0/0x20<br/> Jan 16 13:36:53 vm1 kernel: --<del>[ end trace 7cd192156324e64a ]</del>--<br/> Jan 16 13:36:53 vm1 kernel: -----------<del>[ cut here ]</del>-----------<br/> Jan 16 13:36:53 vm1 kernel: WARNING: at fs/fs-writeback.c:990 sync_inodes_sb+0x17f/0x190() (Tainted: G W --------------- )<br/> Jan 16 13:36:53 vm1 kernel: Hardware name: KVM<br/> Jan 16 13:36:53 vm1 kernel: Modules linked in: ldiskfs sha512_generic sha256_generic bridge stp llc nfs lockd fscache auth_rpcgss nfs_acl sunrpc ipv6 microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core sg ext4 jbd2 mbcache sd_mod crc_t10dif pata_acpi ata_generic ata_piix virtio_pci virtio_ring virtio dm_mirror dm_region_hash dm_log dm_mod <span class="error">&#91;last unloaded: libcfs&#93;</span><br/> Jan 16 13:36:53 vm1 kernel: Pid: 19717, comm: mkfs.lustre Tainted: G W --------------- 2.6.32 #4<br/> Jan 16 13:36:53 vm1 kernel: Call Trace:<br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffff8106e3e7&gt;&#93;</span> ? warn_slowpath_common+0x87/0xc0<br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffff8106e43a&gt;&#93;</span> ? warn_slowpath_null+0x1a/0x20<br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffff811ab8df&gt;&#93;</span> ? sync_inodes_sb+0x17f/0x190<br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffff811b1672&gt;&#93;</span> ? __sync_filesystem+0x82/0x90<br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffff811b187b&gt;&#93;</span> ? sync_filesystem+0x4b/0x70<br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffffa02e7048&gt;&#93;</span> ? ldiskfs_quota_off+0x218/0x220 <span class="error">&#91;ldiskfs&#93;</span><br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffff81183496&gt;&#93;</span> ? deactivate_super+0x46/0x80<br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffff811a17ef&gt;&#93;</span> ? mntput_no_expire+0xbf/0x110<br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffff811a225b&gt;&#93;</span> ? sys_umount+0x7b/0x3a0<br/> Jan 16 13:36:53 vm1 kernel: <span class="error">&#91;&lt;ffffffff8100b072&gt;&#93;</span> ? system_call_fastpath+0x16/0x1b</p> <p>Yes, indeed. Looks there were heavy changes about the s_umount lock in upstream kernel, and we can't simply backport the path.</p> <p>Hi, LiXi<br/> Given that remount operation will never be performed on Lustre device in normal usage, I think this deadlock should never happen on Lustre, I'm wondering if we can just leave it as Not Fixed. What do you think about? (Is there any operation in Lustre normal uasage could trigger this deadlock?)</p> <p>Hi Yawei,</p> <p>Sorry for my late reply. I got this problem when changing some codes of ldiskfs. And I don't know there is any way to trigger this problem from Lustre operation. So, I agree to close this ticket as won't fix.</p> <p>Thanks!</p> <p>Thank you, LiXi.</p> <p>Artem Blagodarenko (artem.blagodarenko@seagate.com) uploaded a new patch: <a href="http://review.whamcloud.com/21181" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/21181</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-4474" title="deadlock of ldiskfs_quota_off()" class="issue-link" data-issue-key="LU-4474"><del>LU-4474</del></a> osd: Add nodelalloc to ldisk mount options<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 4130fa92a09b80abf57ba7d539214948ddbf2869</p> <p>This hang happened on my system because SELinux is enabled (permissive mode) on my VM. But it can happened in different situations then os_mount cleanup happens</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-quote">"echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs"</span> disables <span class="code-keyword">this</span> message. mount.lustre D 0000000000000000 0 10566 10564 0x00000080 ffff88004e689710 0000000000000082 0000000000000000 ffff88004e689718 0000000000000000 ffff88004e6896e8 0000000000000286 ffff88004e6896e8 ffff88004d5d7af8 ffff88004e689fd8 000000000000fbc8 ffff88004d5d7af8 Call Trace: [&lt;ffffffff8152a6d5&gt;] rwsem_down_failed_common+0x95/0x1d0 [&lt;ffffffff8152a866&gt;] rwsem_down_read_failed+0x26/0x30 [&lt;ffffffff8128f174&gt;] call_rwsem_down_read_failed+0x14/0x30 [&lt;ffffffff81529d64&gt;] ? down_read+0x24/0x30 [&lt;ffffffffa060217c&gt;] ldiskfs_quota_off+0x1bc/0x1f0 [ldiskfs] [&lt;ffffffff812325aa&gt;] ? selinux_sb_copy_data+0x14a/0x1e0 [&lt;ffffffff8118b866&gt;] deactivate_locked_super+0x46/0x90 [&lt;ffffffff8118b9bd&gt;] vfs_kern_mount+0x10d/0x1b0 [&lt;ffffffffa0654cc4&gt;] osd_mount+0x544/0xf90 [osd_ldiskfs] [&lt;ffffffffa0655c1f&gt;] osd_device_alloc+0x50f/0x9d0 [osd_ldiskfs] [&lt;ffffffffa1010d7f&gt;] obd_setup+0x1bf/0x290 [obdclass] [&lt;ffffffffa0ef9820&gt;] ? cfs_hash_create+0x2e0/0x940 [libcfs] [&lt;ffffffffa1010fff&gt;] class_setup+0x1af/0x7d0 [obdclass] [&lt;ffffffffa0ff550f&gt;] ? class_name2dev+0x2f/0x40 [obdclass] [&lt;ffffffffa1018fac&gt;] class_process_config+0xc6c/0x1ad0 [obdclass] [&lt;ffffffffa0eeb888&gt;] ? libcfs_log_return+0x28/0x40 [libcfs] [&lt;ffffffffa101dfab&gt;] ? lustre_cfg_new+0x40b/0x6f0 [obdclass] [&lt;ffffffffa101e3e8&gt;] do_lcfg+0x158/0x450 [obdclass] [&lt;ffffffffa101e774&gt;] lustre_start_simple+0x94/0x200 [obdclass] [&lt;ffffffffa1059411&gt;] server_fill_super+0xf61/0x1abc [obdclass] [&lt;ffffffffa0eeb888&gt;] ? libcfs_log_return+0x28/0x40 [libcfs] [&lt;ffffffffa1024388&gt;] lustre_fill_super+0x1d8/0x550 [obdclass] [&lt;ffffffffa10241b0&gt;] ? lustre_fill_super+0x0/0x550 [obdclass] [&lt;ffffffff8118c2cf&gt;] get_sb_nodev+0x5f/0xa0 [&lt;ffffffffa101bec5&gt;] lustre_get_sb+0x25/0x30 [obdclass] [&lt;ffffffff8118b92b&gt;] vfs_kern_mount+0x7b/0x1b0 [&lt;ffffffff8118bad2&gt;] do_kern_mount+0x52/0x130 [&lt;ffffffff811acb0b&gt;] do_mount+0x2fb/0x930 [&lt;ffffffff811ad1d0&gt;] sys_mount+0x90/0xe0 [&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b </pre> </div></div> <p>Just tested on my local environment.<br/> Without the patch</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">root@devvm-sl6-2]# sestatus SELinux status: enabled SELinuxfs mount: /selinux Current mode: permissive Mode from config file: permissive Policy version: 24 Policy from config file: mls [root@devvm-sl6-2]# LOAD=yes lustre/tests/llmount.sh Loading modules from /lustre/mnt/work/MRP-3473/lustre/tests/.. detected 8 online CPUs by sysfs libcfs will create CPU partition based on online CPUs debug=vfstrace rpctrace dlmtrace neterror ha config ioctl <span class="code-keyword">super</span> lfsck subsystem_debug=all -lnet -lnd -pinger gss/krb5 is not supported [root@devvm-sl6-2]# mount -t lustre -o loop -o context=fake_context /tmp/lustre-mdt1 /mnt/mds1 SELinux: duplicate or incompatible mount options INFO: task mount.lustre:10567 blocked <span class="code-keyword">for</span> more than 120 seconds. Not tainted 2.6.32master #0 <span class="code-quote">"echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs"</span> disables <span class="code-keyword">this</span> message. mount.lustre D 0000000000000007 0 10567 10565 0x00000080 ffff880064bd7710 0000000000000086 00000000ffffffea ffff880064bd7718 0000000000000000 ffff880064bd76e8 0000000000000282 ffff880064bd76e8 ffff880037335098 ffff880064bd7fd8 000000000000fbc8 ffff880037335098 Call Trace: [&lt;ffffffff8152a6d5&gt;] rwsem_down_failed_common+0x95/0x1d0 [&lt;ffffffff8152a866&gt;] rwsem_down_read_failed+0x26/0x30 [&lt;ffffffff8128f174&gt;] call_rwsem_down_read_failed+0x14/0x30 [&lt;ffffffff81529d64&gt;] ? down_read+0x24/0x30 [&lt;ffffffffa0cd326c&gt;] ldiskfs_quota_off+0x1bc/0x1f0 [ldiskfs] [&lt;ffffffff812325aa&gt;] ? selinux_sb_copy_data+0x14a/0x1e0 [&lt;ffffffff8118b866&gt;] deactivate_locked_super+0x46/0x90 [&lt;ffffffff8118b9bd&gt;] vfs_kern_mount+0x10d/0x1b0 [&lt;ffffffffa0d25cd4&gt;] osd_mount+0x544/0xf90 [osd_ldiskfs] [&lt;ffffffffa0d26c2f&gt;] osd_device_alloc+0x50f/0x9d0 [osd_ldiskfs] [&lt;ffffffffa061b3ff&gt;] obd_setup+0x1bf/0x290 [obdclass] [&lt;ffffffffa052f840&gt;] ? cfs_hash_create+0x2e0/0x940 [libcfs] [&lt;ffffffffa061b67f&gt;] class_setup+0x1af/0x7d0 [obdclass] [&lt;ffffffffa05ffb8f&gt;] ? class_name2dev+0x2f/0x40 [obdclass] [&lt;ffffffffa062362c&gt;] class_process_config+0xc6c/0x1ad0 [obdclass] [&lt;ffffffffa0628715&gt;] ? lustre_cfg_new+0x4f5/0x6f0 [obdclass] [&lt;ffffffffa0628a68&gt;] do_lcfg+0x158/0x450 [obdclass] [&lt;ffffffffa0628df4&gt;] lustre_start_simple+0x94/0x200 [obdclass] [&lt;ffffffffa0663c51&gt;] server_fill_super+0x1021/0x1b76 [obdclass] [&lt;ffffffffa062ea58&gt;] lustre_fill_super+0x218/0x5f0 [obdclass] [&lt;ffffffffa062e840&gt;] ? lustre_fill_super+0x0/0x5f0 [obdclass] [&lt;ffffffff8118c2cf&gt;] get_sb_nodev+0x5f/0xa0 [&lt;ffffffffa0626545&gt;] lustre_get_sb+0x25/0x30 [obdclass] [&lt;ffffffff8118b92b&gt;] vfs_kern_mount+0x7b/0x1b0 [&lt;ffffffff8118bad2&gt;] do_kern_mount+0x52/0x130 [&lt;ffffffff811acb0b&gt;] do_mount+0x2fb/0x930 [&lt;ffffffff811ad1d0&gt;] sys_mount+0x90/0xe0 [&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b INFO: task mount.lustre:10567 blocked <span class="code-keyword">for</span> more than 120 seconds. Not tainted 2.6.32master #0 <span class="code-quote">"echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs"</span> disables <span class="code-keyword">this</span> message. </pre> </div></div> <p>With patch</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">SELinux status: enabled SELinuxfs mount: /selinux Current mode: permissive Mode from config file: permissive Policy version: 24 Policy from config file: mls root@devvm-sl6-2]# mount -t lustre -o loop -o context=fake_context /tmp/lustre-mdt1 /mnt/mds1 mount.lustre: mount /dev/loop0 at /mnt/mds1 failed: Invalid argument This may have multiple causes. Are the mount options correct? Check the syslog <span class="code-keyword">for</span> more info. [root@devvm-sl6-2]# dmesg | tail -n 50 LDISKFS-fs (loop0): mounted filesystem with ordered data mode. quota=on. Opts: SELinux: duplicate or incompatible mount options LustreError: 7914:0:(osd_handler.c:5900:osd_mount()) lustre-MDT0000-osd: can't mount /dev/loop0: -22 LustreError: 7914:0:(obd_config.c:517:class_setup()) setup lustre-MDT0000-osd failed (-22) LustreError: 7914:0:(obd_mount.c:207:lustre_start_simple()) lustre-MDT0000-osd setup error -22 LustreError: 7914:0:(obd_mount_server.c:1713:server_fill_super()) Unable to start osd on /dev/loop0: -22 LustreError: 7914:0:(obd_mount.c:1327:lustre_fill_super()) Unable to mount (-22) [root@devvm-sl6-2]# </pre> </div></div> <p>After patch mount exit with error as expected.</p> <p>Artem,<br/> I wonder if your change #21181 will not introduce some performance decrease for some workloads??<br/> Did you expose it to some filesystem benchmarks suites?</p> <p>Well, after having some internal discussions, it has been established that "delayed allocation" feature of ext4 is not being used in ldiskfs, so your patch (Gerrit-change #21181) finally looks as an easy way to fix this problem!</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/21181/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/21181/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-4474" title="deadlock of ldiskfs_quota_off()" class="issue-link" data-issue-key="LU-4474"><del>LU-4474</del></a> osd: Add nodelalloc to ldiskfs mount options<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 7597c672850e7e9232f7400170455fd12aa08b8a</p> <p>Landed for 2.9. Please speak up if any further work is still needed</p> Duplicate Development Rank 1|hzwco7: Rank (Obsolete) 12256 Severity