https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3975 Lustre <p>Parallel calls to mount targets when ldiskfs is not already loaded can lead to hitting a race in the kernel when it attempts to load the module, which can result in the second mount failing. This race is not unique to ldiskfs and can affect any module that does not protect itself with some sort of locking mechanism.</p> <p>This bug was fixed in kernel-3.7.0, and is documented here: <a href="https://bugzilla.redhat.com/show_bug.cgi?id=771285" class="external-link" target="_blank" rel="nofollow noopener">https://bugzilla.redhat.com/show_bug.cgi?id=771285</a></p> <p>Which links to this fix here: <a href="http://thread.gmane.org/gmane.linux.kernel/1358707/focus=1358709" class="external-link" target="_blank" rel="nofollow noopener">http://thread.gmane.org/gmane.linux.kernel/1358707/focus=1358709</a></p> <p>We have confirmed that this fix has not been backported to the 2.6.32 kernel yet. We have opened a bug with RedHat regarding the issue here: <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1009704" class="external-link" target="_blank" rel="nofollow noopener">https://bugzilla.redhat.com/show_bug.cgi?id=1009704</a></p> <p>This can cause parallel calls to mkfs.lustre to fail as well, as the mounts in ldiskfs_write_ldd can hit this race if ldiskfs is not already loaded.</p> <p>I think there are two outstanding questions here:</p> <p>(1) Do we want to try to do the backport ourselves and not wait on RedHat?<br/> (2) Is it safe to explicitly "modprobe ldiskfs" prior to calling mkfs.lustre to protect ourselves against the race? Or could loading the module explicitly cause some other issues with Lustre?</p> LU-3975

Race loading ldiskfs with parallel mounts

Bug Minor Resolved Duplicate Hongchao Zhang Kelsey Prantis Thu, 19 Sep 2013 21:21:27 +0000 Mon, 23 Dec 2013 21:35:59 +0000 Wed, 27 Nov 2013 19:13:38 +0000 0 11 <p>The ldiskfs module is standalone, so there shouldn't be any problems loading it explicitly. It would be good to load it in mount.lustre and mkfs.lustre so that this problem is fixed for all Lustre users and we don't have to debug it again. There is also no problem to have IML load ldiskfs and patch mkfs.lustre and mount.lustre to do the same.</p> <p>It would also be possible to patch the kernel, since we already patch the RHEL kernel for other reasons, though we are trying to eliminate the kernel patches. This would be better from the POV that the patch would naturally disappear when the fix is backported (unlike the workarounds), but has the problem that it only works for the specific kernels that are patched (i.e. we'd need to patch SLES11 in addition to RHEL6) and would likely be more effort to catch the various Lustre kernels in use.</p> <p>Summary: any/all of the above fixes/workarounds are acceptable.</p> <p>From my understanding of the problem, modprobing the module doesn't really eliminate the race but rather just mitigates it by allowing a sufficient amount of time to go by so that everyone that will want the module will have it loaded by the time they get to a point of mounting something. So in order for <tt>mkfs.lustre</tt> to benefit from this work-around, it should issue the <tt>modprobe</tt> before starting the <tt>mke2fs</tt> to give as much time as possible before it's <tt>mount(2)</tt> will want it to be there.</p> <p>This looks like a duplicate of <a href="https://jira.whamcloud.com/browse/LU-1279" title="failure trying to mount two targets at the same time after boot" class="issue-link" data-issue-key="LU-1279"><del>LU-1279</del></a>. We are able to avoid this by pre-loading modules in the correct order although that approach isn't without it's problems, in particular the module list that we need to load is very long and Lustre version specific, we've also found <a href="https://jira.whamcloud.com/browse/LU-3948" title="Unable to load lustre module when the filesystem is running." class="issue-link" data-issue-key="LU-3948"><del>LU-3948</del></a>.</p> <p>Hongchao</p> <p>Could you please confirm whether this is a duplicate of <a href="https://jira.whamcloud.com/browse/LU-1279" title="failure trying to mount two targets at the same time after boot" class="issue-link" data-issue-key="LU-1279"><del>LU-1279</del></a>?</p> <p>Thanks</p> <p>Peter</p> <p>Hongchao,</p> <p>Trying to port this kernel patch back to CentOS 6.4. It doesn't land cleanly it all, in fact it seems to depend on a fairly significant rewrite of module.c.</p> <p>For example, the second component of this patch shows module_mutex being locked (in load_module), but in 2.6.32, that mutex isn't referenced in load_module at all, as best I can tell.</p> <p>There are other differences, such as there's no label 'free_arch_cleanup' in load_module, and the code that returns EEXIST appears to have been relocated in load_module as well.</p> <p>I'm trying to get the kernel git repo checked out so I can try tracking down more of the patch history here, but I'm getting 404s trying to check out the kernel.org Linux repo... We'll see.</p> <p>Kelsey, has there been any update on the RedHat bug? It'd be great if they'd do the porting.</p> <p>Here's the problematic part of the patch I was referring to:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">+again: mutex_lock(&amp;module_mutex); - <span class="code-keyword">if</span> (find_module(mod-&gt;name)) { + <span class="code-keyword">if</span> ((old = find_module(mod-&gt;name)) != NULL) { + <span class="code-keyword">if</span> (old-&gt;state == MODULE_STATE_COMING) { + <span class="code-comment">/* Wait in <span class="code-keyword">case</span> it fails to load. */</span> + mutex_unlock(&amp;module_mutex); + err = wait_event_interruptible(module_wq, + finished_loading(mod-&gt;name)); + <span class="code-keyword">if</span> (err) + <span class="code-keyword">goto</span> free_arch_cleanup; + <span class="code-keyword">goto</span> again; + } err = -EEXIST; <span class="code-keyword">goto</span> unlock; } </pre> </div></div> <p>Closing this as a duplicate of <a href="https://jira.whamcloud.com/browse/LU-1279" title="failure trying to mount two targets at the same time after boot" class="issue-link" data-issue-key="LU-1279"><del>LU-1279</del></a>.</p> Duplicate LU-1279 Related LU-1279 Development Rank 1|hzw38n: Rank (Obsolete) 10609 Severity