https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3410 Lustre <p>At our first attempt to mount the file system we experienced the following assertion on about 40 OSS. Th eassertion we hit is as follows:</p> <p>May 28 10:36:43 widow-oss11c4 kernel: [ 2743.137026] LDISKFS-fs (dm-12): mounted filesystem with ordered data mode. quota=on. Opts:<br/> May 28 10:37:07 widow-oss11c4 kernel: [ 2767.330069] LustreError: 24348:0:(ofd_obd.c:1535:ofd_iocontrol()) routed1-OST0113: aborting recovery<br/> May 28 10:37:28 widow-oss11c4 kernel: [ 2788.099708] LustreError: 24283:0:(ofd_obd.c:1535:ofd_iocontrol()) routed1-OST00b3: aborting recovery<br/> May 28 10:37:28 widow-oss11c4 kernel: [ 2788.145570] LustreError: 24562:0:(mgc_request.c:1693:mgc_copy_llog()) Failed to copy remote log routed1-OST00b3 (-2) <br/> May 28 10:37:28 widow-oss11c4 kernel: [ 2788.166762] LustreError: 13a-8: Failed to get MGS log routed1-OST00b3 and no local copy.<br/> May 28 10:37:28 widow-oss11c4 kernel: [ 2788.183045] LustreError: 24562:0:(lvfs_linux.c:175:pop_ctxt()) ASSERTION( cfs_fs_pwd(current-&gt;fs) == new_ctx-&gt;pwd ) failed: ffff8803bcea02c0 != ffff8803bcefcf00<br/> May 28 10:37:28 widow-oss11c4 kernel: [ 2788.211874] LustreError: 24562:0:(lvfs_linux.c:175:pop_ctxt()) LBUG<br/> May 28 10:37:28 widow-oss11c4 kernel: [ 2788.224479] Pid: 24562, comm: ll_cfg_requeue<br/> May 28 10:37:28 widow-oss11c4 kernel: [ 2788.233060] <br/> May 28 10:37:28 widow-oss11c4 kernel: [ 2788.233061] Call Trace:<br/> May 28 10:37:28 widow-oss11c4 kernel: [ 2788.240987] <span class="error">&#91;&lt;ffffffffa045e895&gt;&#93;</span> libcfs_debug_dumpstack+0x55/0x80 <span class="error">&#91;libcfs&#93;</span><br/> May 28 10:37:28 widow-oss11c4 kernel: [ 2788.254989] <span class="error">&#91;&lt;ffffffffa045ee97&gt;&#93;</span> lbug_with_loc+0x47/0xb0 <span class="error">&#91;libcfs&#93;</span><br/> May 28 10:37:28 widow-oss11c4 kernel: [ 2788.267434] <span class="error">&#91;&lt;ffffffffa03cecb5&gt;&#93;</span> pop_ctxt+0x295/0x2e0 <span class="error">&#91;lvfs&#93;</span><br/> May 28 10:37:28 widow-oss11c4 kernel: [ 2788.278976] <span class="error">&#91;&lt;ffffffffa06d6225&gt;&#93;</span> mgc_process_cfg_log+0x6a5/0x15c0 <span class="error">&#91;mgc&#93;</span><br/> May 28 10:37:28 widow-oss11c4 kernel: [ 2788.292448] <span class="error">&#91;&lt;ffffffffa06d9133&gt;&#93;</span> mgc_process_log+0x463/0x1390 <span class="error">&#91;mgc&#93;</span><br/> May 28 10:37:28 widow-oss11c4 kernel: [ 2788.305221] <span class="error">&#91;&lt;ffffffff8150ed8a&gt;&#93;</span> ? schedule_timeout+0x19a/0x2e0<br/> May 28 10:37:28 widow-oss11c4 kernel: [ 2788.317303] <span class="error">&#91;&lt;ffffffffa06d3a60&gt;&#93;</span> ? mgc_blocking_ast+0x0/0x7e0 <span class="error">&#91;mgc&#93;</span><br/> May 28 10:37:28 widow-oss11c4 kernel: [ 2788.330171] <span class="error">&#91;&lt;ffffffffa071cb90&gt;&#93;</span> ? ldlm_completion_ast+0x0/0x960 <span class="error">&#91;ptlrpc&#93;</span><br/> May 28 10:37:28 widow-oss11c4 kernel: [ 2788.343991] <span class="error">&#91;&lt;ffffffff8109705c&gt;&#93;</span> ? remove_wait_queue+0x3c/0x50<br/> May 28 10:37:28 widow-oss11c4 kernel: [ 2788.355882] <span class="error">&#91;&lt;ffffffffa06daa13&gt;&#93;</span> mgc_requeue_thread+0x1a3/0x750 <span class="error">&#91;mgc&#93;</span><br/> May 28 10:37:28 widow-oss11c4 kernel: [ 2788.368999] <span class="error">&#91;&lt;ffffffff81063310&gt;&#93;</span> ? default_wake_function+0x0/0x20<br/> May 28 10:37:28 widow-oss11c4 kernel: [ 2788.381448] <span class="error">&#91;&lt;ffffffffa06da870&gt;&#93;</span> ? mgc_requeue_thread+0x0/0x750 <span class="error">&#91;mgc&#93;</span><br/> May 28 10:37:28 widow-oss11c4 kernel: [ 2788.394565] <span class="error">&#91;&lt;ffffffff8100c0ca&gt;&#93;</span> child_rip+0xa/0x20<br/> May 28 10:37:28 widow-oss11c4 kernel: [ 2788.404563] <span class="error">&#91;&lt;ffffffffa06da870&gt;&#93;</span> ? mgc_requeue_thread+0x0/0x750 <span class="error">&#91;mgc&#93;</span><br/> May 28 10:37:28 widow-oss11c4 kernel: [ 2788.423777] <span class="error">&#91;&lt;ffffffffa06da870&gt;&#93;</span> ? mgc_requeue_thread+0x0/0x750 <span class="error">&#91;mgc&#93;</span><br/> May 28 10:37:28 widow-oss11c4 kernel: [ 2788.442884] <span class="error">&#91;&lt;ffffffff8100c0c0&gt;&#93;</span> ? child_rip+0x0/0x20<br/> May 28 10:37:28 widow-oss11c4 kernel: [ 2788.459196]</p> RHEL6.4 running 2.4.0-RC2 LU-3410

mgc_copy_llog()) Failed to copy remote log routed1-OST00b3 (-2) ecountered during bring up of a 2.4 file system

Bug Major Closed Duplicate Mikhail Pershin James A Simmons Tue, 28 May 2013 17:01:38 +0000 Fri, 11 Apr 2014 20:02:30 +0000 Fri, 11 Apr 2014 20:02:13 +0000 Lustre 2.4.0 0 9 <p>James, did the same problem happen on a second restart, or did it work on the next try?</p> <p>James, what is the origin of this filesystem? Is this a newly formatted filesystem with 2.4.0 or is it upgraded from some previous version of Lustre?</p> <p>Mike rewrote the MGC llog backup code to use the OSD API recently in <a href="http://review.whamcloud.com/5049" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5049</a> (patch is ready to land, but missed "feature freeze" window), so any work in this area should take that into account. I'm not sure it will solve this problem, but it doesn't make sense to work on the old code when it is being replaced.</p> <p>This only happened once, second restart worked.<br/> The filesystem was freshly formatted 2.4 system.</p> <p>the mgs_fs_setup() may rewrite the obd_lvfs_ctxt with new values and there is no any protection that could prevent that between push/pop pair. That can be the reasonfor this bug. Note, that mgs_fs_setup() is called from obd_mount.c code to set new bottom fs and can be called multiple times with several OSTs on one node. Meanwhile this is old code and I am not sure why it happens now.</p> <p>Mike,</p> <p>looks you are wrong (or it's second bug with same assert).<br/> mgc_requeue_thread - don't have an cfs_daemonize_ctxt() at top of function so share a current-&gt;fs between several threads, as ldlm_elt, zombid, ldlm_poold (with vitaly patch with stale lock cancel) and some other.<br/> in that case different thread may change a own current-&gt;fs but it's affected to whole threads with same fs.<br/> Xyratex hit that bug in ldlm_elt thread (don't have an cfs_daemonize_ctxt() call also.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">.. 00002000:00040000:9.0:1375891841.290003:0:1448:0:(lvfs_linux.c:175:pop_ctxt()) ASSERTION( cfs_fs_pwd(current-&gt;fs) == new_ctx-&gt;pwd ) failed: ffff88076b4d68c0 != ffff88069df8fd80 .. crash&gt; bt PID: 1448 TASK: ffff880817656040 CPU: 9 COMMAND: <span class="code-quote">"ldlm_elt"</span> #0 [ffff880774e7dba8] machine_kexec at ffffffff810310db #1 [ffff880774e7dc08] crash_kexec at ffffffff810b6332 #2 [ffff880774e7dcd8] panic at ffffffff814d684f #3 [ffff880774e7dd58] lbug_with_loc at ffffffffa0380ecb [libcfs] #4 [ffff880774e7dd78] pop_ctxt at ffffffffa0427b77 [lvfs] #5 [ffff880774e7ddc8] filter_client_del at ffffffffa0ed5637 [obdfilter] #6 [ffff880774e7de78] filter_disconnect at ffffffffa0ed7820 [obdfilter] #7 [ffff880774e7dea8] class_fail_export at ffffffffa0648365 [obdclass] #8 [ffff880774e7dec8] expired_lock_main at ffffffffa07b81b1 [ptlrpc] #9 [ffff880774e7df48] kernel_thread at ffffffff8100c1ca </pre> </div></div> <p>but in task fs is correct value</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">crash&gt; p *((struct task_struct *)0xffff880817656040)-&gt;fs $119 = { users = 481, lock = { raw_lock = { lock = 16777216 } }, umask = 0, in_exec = 0, root = { mnt = 0xffff880817b090c0, dentry = 0xffff880818e4fa40 }, pwd = { mnt = 0xffff8808174b6180, dentry = 0xffff88069df8fd80 } } </pre> </div></div> <p>as you see - fs have a correct value and assert should be don't hit.<br/> <em>BUT</em> users is large number which should be impossible for a thread with disk IO.<br/> so it's race between modification for same fs struct.</p> <p>additional bugs in that area<br/> 1) current-&gt;fs.pwd accessed without lock, but readlock should be held. need look carefully about bug impact.<br/> 2) cfs_daemonize_ctxt have lost a return code from unshare_fs_struct() call so we have failed with ENOMEM and none will know about it<br/> 3) none error code checks for cfs_daemonize_ctxt() calls, so function may failed with ENOMEM but none know about it and uses a shared FS after fail.</p> <p>Mike, I also see "13a-8: Failed to get MGS log routed1-OST00b3 and no local copy." in many test logs. This is a misleading error message and should probably be removed, or at least quieted in the common case when a new filesystem is first mounted.</p> <p>I filed a new ticket <a href="https://jira.whamcloud.com/browse/LU-4783" title="start params llog automatically during mgs setup" class="issue-link" data-issue-key="LU-4783"><del>LU-4783</del></a> to fix the misleading error message "LustreError: 13a-8: Failed to get MGS log params and no local copy.".</p> <p>So, can we close this one ?</p> <p>closing in favor of <a href="https://jira.whamcloud.com/browse/LU-4783" title="start params llog automatically during mgs setup" class="issue-link" data-issue-key="LU-4783"><del>LU-4783</del></a></p> Related LU-4783 Development Rank 1|hzvryn: Rank (Obsolete) 8427 Severity