https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-13617 Lustre <p>One thread got PR lock and waited on inode_lock to notify security context. Another thread got inode_lock and waited on CW lock(conflicts with PR). After timeout the client was evicted, it didn't cancel a PR lock. The dead lock happens on a parent directory.</p> LU-13617

Client dead lock leads to eviction from MDS (selinux is enabled)

Bug Major Resolved Fixed Alexander Boyko Alexander Boyko patch Mon, 1 Jun 2020 12:56:08 +0000 Mon, 27 Jun 2022 02:28:59 +0000 Thu, 13 Aug 2020 08:58:57 +0000 Lustre 2.14.0 Lustre 2.12.7 Lustre 2.12.8 Lustre 2.14.0 0 6 <p>Alexander Boyko (alexander.boyko@hpe.com) uploaded a new patch: <a href="https://review.whamcloud.com/38792" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38792</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13617" title="Client dead lock leads to eviction from MDS (selinux is enabled)" class="issue-link" data-issue-key="LU-13617"><del>LU-13617</del></a> llite: don't hold inode_lock for security notify<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 3f0d01e64c5248cf9542625bf5d5d3034460912b</p> <p>Alexander Boyko (alexander.boyko@hpe.com) uploaded a new patch: <a href="https://review.whamcloud.com/38793" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38793</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13617" title="Client dead lock leads to eviction from MDS (selinux is enabled)" class="issue-link" data-issue-key="LU-13617"><del>LU-13617</del></a> tests: check client deadlock selinux<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 5efb04c50e42abc2e017cef49e065cc089fb7323</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/38792/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38792/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13617" title="Client dead lock leads to eviction from MDS (selinux is enabled)" class="issue-link" data-issue-key="LU-13617"><del>LU-13617</del></a> llite: don't hold inode_lock for security notify<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: f87359b51f61a4baa9bf62faebb6625d518d23b4</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/38793/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38793/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13617" title="Client dead lock leads to eviction from MDS (selinux is enabled)" class="issue-link" data-issue-key="LU-13617"><del>LU-13617</del></a> tests: check client deadlock selinux<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: f519f22c8ba3a6de00af0bef77cae3b4b18acdab</p> <p>We hit this behavior in 2.12.8. Client got evicted due to lock timeout when selinux is enabled. Wonder if this patch also should go into 2.12?</p> <p>regression was</p> <blockquote><p> Fixes: 1d44980bcb ("<a href="https://jira.whamcloud.com/browse/LU-8956" title="Improve create performance with SELinux enabled" class="issue-link" data-issue-key="LU-8956"><del>LU-8956</del></a> llite: set sec ctx on client's inode at create time")</p></blockquote> <p>git log -<del>grep <a href="https://jira.whamcloud.com/browse/LU-8956" title="Improve create performance with SELinux enabled" class="issue-link" data-issue-key="LU-8956"><del>LU-8956</del></a>  </del>-oneline origin/b2_12<br/> 1d44980 <a href="https://jira.whamcloud.com/browse/LU-8956" title="Improve create performance with SELinux enabled" class="issue-link" data-issue-key="LU-8956"><del>LU-8956</del></a> llite: set sec ctx on client's inode at create time</p> <p>So 2.12 has the same problem as description.</p> <p>We hit this on a robinhood node:</p> <ul class="alternate" type="square"> <li>with cleanup policy doing parallel "mv" on a folder</li> <li>and stat/getstripe threads on the same folders</li> </ul> <p>"Etienne AUJAMES &lt;eaujames@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/47025" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/47025</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13617" title="Client dead lock leads to eviction from MDS (selinux is enabled)" class="issue-link" data-issue-key="LU-13617"><del>LU-13617</del></a> llite: don't hold inode_lock for security notify<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: bf9ca48adbb69cda4196622ed89a55778c773e85</p> <p>"Etienne AUJAMES &lt;eaujames@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/47034" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/47034</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13617" title="Client dead lock leads to eviction from MDS (selinux is enabled)" class="issue-link" data-issue-key="LU-13617"><del>LU-13617</del></a> tests: check client deadlock selinux<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: c0906f8c6d23382fb36eb6e7c309012048d8b67f</p> <p>We hit this issue during maintenance regression tests (after updating Lustre clients from 2.12.6 LTS to 2.12.7 LTS on compute nodes).<br/> The file system was not usable with selinux enabled (permissive or enabled) on compute nodes: client evictions, mds unreachable (lot of mdt threads hang waiting ldlm lock).</p> <p>We were able to reproduce the issue with 5 nodes running mdtest.</p> <p>The CEA will patch Lustre clients with <a href="https://review.whamcloud.com/47034" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/47034</a> on compute nodes and run regression tests on it.<br/> I will let you know the results.</p> <p>For now, we cannot activate selinux on clients without this patch on Lustre LTS &gt;= 2.12.7.<br/> The b2_12's regression seems to come from the <a href="https://review.whamcloud.com/41387/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/41387/</a> ("<a href="https://jira.whamcloud.com/browse/LU-9193" title="Multiple hangs observed with many open/getattr" class="issue-link" data-issue-key="LU-9193"><del>LU-9193</del></a> security: return security context for metadata ops<br/> ")</p> Related Development Rank 1|i011on: Rank (Obsolete) 9223372036854775807 Severity