https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-8907 Lustre <p>One Lustre client crashed right after a recovery with the following messages:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2016-10-29 16:00:01 Lustre: DEBUG MARKER: Sat Oct 29 16:00:01 2016 2016-10-29 16:00:01 2016-10-29 16:01:07 Lustre: 17766:0:(client.c:1942:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1477749640/real 1477749640] req@ffff8811e1b9a400 x1547883906151520/t0(0) o101-&gt;scratch3-MDT0000-mdc-ffff88205ec91400@JO.BOO.AL.IL@o2ib2:12/10 lens 632/1136 e 0 to 1 dl 1477749667 ref 2 fl Rpc:XP/0/ffffffff rc 0/-1 2016-10-29 16:01:07 Lustre: 17766:0:(client.c:1942:ptlrpc_expire_one_request()) Skipped 12 previous similar messages 2016-10-29 16:01:07 Lustre: scratch3-MDT0000-mdc-ffff88205ec91400: Connection to scratch3-MDT0000 (at JO.BOO.AL.IL@o2ib2) was lost; in progress operations using this service will wait for recovery to complet e 2016-10-29 16:01:07 Lustre: Skipped 12 previous similar messages 2016-10-29 16:01:07 Lustre: scratch3-MDT0000-mdc-ffff88205ec91400: Connection restored to scratch3-MDT0000 (at JO.BOO.AL.IL@o2ib2) 2016-10-29 16:01:07 Lustre: Skipped 12 previous similar messages 2016-10-29 16:01:07 LustreError: 17766:0:(namei.c:816:ll_create_node()) ASSERTION( list_empty(&amp;inode-&gt;i_dentry) ) failed: 2016-10-29 16:01:07 LustreError: 17766:0:(namei.c:816:ll_create_node()) LBUG </pre> </div></div> <p>The stack is as follows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>PID: 17766 TASK: ffff8819ab5b0ab0 CPU: 6 COMMAND: "rsync" #0 [ffff88189c697b20] machine_kexec at ffffffff8103ff4b #1 [ffff88189c697b80] crash_kexec at ffffffff810cfce2 #2 [ffff88189c697c50] panic at ffffffff81546ce9 #3 [ffff88189c697cd0] lbug_with_loc at ffffffffa067beeb [libcfs] #4 [ffff88189c697cf0] ll_create_nd at ffffffffa0dbf854 [lustre] #5 [ffff88189c697d70] vfs_create at ffffffff811a7946 #6 [ffff88189c697db0] do_filp_open at ffffffff811ab75e #7 [ffff88189c697f20] do_sys_open at ffffffff81194e87 #8 [ffff88189c697f70] sys_open at ffffffff81194f90 #9 [ffff88189c697f80] system_call_fastpath at ffffffff8100b0d2 RIP: 0000003c88adb480 RSP: 00007fffd85b6bc8 RFLAGS: 00010246 RAX: 0000000000000002 RBX: ffffffff8100b0d2 RCX: 0000000000000001 RDX: 0000000000000180 RSI: 00000000000000c2 RDI: 00007fffd85b7d70 RBP: 00007fffd85b7d9b R8: 7a672e7761722e30 R9: 00000000ffffffff R10: 0000000000000001 R11: 0000000000000246 R12: ffffffff81194f90 R13: ffff88189c697f78 R14: 00007fffd85b7d9c R15: 0000000000000000 ORIG_RAX: 0000000000000002 CS: 0033 SS: 002b </pre> </div></div> <p> </p> <p>Even though the intent is to create the inode with a dentry in the filesystem namespace (/somepath/.tmpfile), the inode structure field i_dentry is populated with one dentry pointing at /.lustre/fid/<span class="error">&#91;0x298cd542a:0x3b3d:0x0&#93;</span>.</p> <p>Strangely enough, even though the crash occured in ll_create_nd() -&gt; ll_create_it() -&gt; ll_create_node, before the call to d_instantiate() in ll_create_nd() -&gt; ll_create_it(), the fid actually really points back to the requested location:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># lfs fid2path /mountpoint [0x298cd542a:0x3b3d:0x0] /somepath/.tmpfile </pre> </div></div> <p>I can't upload the crashdump due to site restriction, but will be happy to give more informations as you request them.</p> RHEL 6 Bull kernel 2.6.32-642.4.2.el6.Bull.100.x86_64<br/> Lustre build based on 2.5.3.90 LU-8907

LBUG: (namei.c:816:ll_create_node()) ASSERTION( list_empty(&inode->i_dentry) ) failed

Bug Minor Resolved Fixed Bruno Faccini Sebastien Piechurski p4b Mon, 5 Dec 2016 16:29:22 +0000 Wed, 7 Jun 2017 12:09:57 +0000 Wed, 7 Jun 2017 12:09:57 +0000 Lustre 2.5.3 Lustre 2.10.0 0 8 <p>Hello Sebastien,<br/> Could it be possible for me to work on crash-dump with you on-site or when you access it from your office?</p> <p>Hi Bruno,</p> <p> </p> <p>For sure, I'd be happy to work with you on this !</p> <p>Ok, then I will try to contact you offline to see how/when/where we can schedule a RDV.</p> <p>I forgot to mention that the intent dentry which should not had time to be instantiated (since the assert crashed before the call to d_instantiate()) is really /somepath/.tmpfile (the file actually exists on the fs currently).</p> <p> </p> <p>Hi Bruno, and Happy New year to you !</p> <p>Were you able to discuss the data we extracted from the dump with your colleagues on this issue ?</p> <p>Thank you Seb and I wish the same for you and your family!<br/> About the problem, it looks definitely like an unknown single-shoot for us.<br/> But I am still working on both Lustre and Kernel code in order to find a hole which can lead to it.<br/> BTW, can you tell me how does your kernel config (i.e., inode.i_dentry/dentry.d_alias use h_list or double-linked list) cause ll_d_hlist_empty() macro to be converted ?</p> <p>Oops, forget my previous question, the conversion can be found in the LBUG text ...</p> <p>Hello Seb,<br/> As we already have discussed off-line, can you also give us some idea about any additional patches that have been applied on top of Lustre 2.5.3.90 and Kernel 2.6.32-642.4.2 ??<br/> Thanks in advance.</p> <p>Hello Seb, did you see my previous update/query?<br/> On my side, I am working on a possible reproducer, based on some other known occurrences of this same LBUG. Will let you know asap on how it goes.</p> <p>Hmm first attempt to reproduce looks interesting (promising?).<br/> Running a modified version (to enable it on a single-node config) of replay-single/test_70b in a loop, I just got the following LBUG at the 25th pass :</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Lustre: DEBUG MARKER: -----============= acceptance-small: replay-single ============----- Wed Jan 11 16:40:48 UTC 2017 Lustre: DEBUG MARKER: only running test 70b Lustre: DEBUG MARKER: excepting tests: 61d 33a 33b 89 62 Lustre: DEBUG MARKER: skipping tests SLOW=no: 1 2 3 4 6 12 16 44a 44b 65 66 67 68 Lustre: DEBUG MARKER: Using TIMEOUT=20 LustreError: 27268:0:(llite_lib.c:1225:ll_clear_inode()) ASSERTION( lli-&gt;u.d.d_sai == ((void *)0) ) failed: LustreError: 27268:0:(llite_lib.c:1225:ll_clear_inode()) LBUG Pid: 27268, comm: rm Call Trace: [&lt;ffffffffa03fe895&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] [&lt;ffffffffa03fee97&gt;] lbug_with_loc+0x47/0xb0 [libcfs] [&lt;ffffffffa0f64e72&gt;] ll_clear_inode+0x8e2/0x920 [lustre] [&lt;ffffffffa0f63300&gt;] ? ll_delete_inode+0x0/0x1e0 [lustre] [&lt;ffffffff811a69fc&gt;] clear_inode+0xac/0x140 [&lt;ffffffffa0f6338d&gt;] ll_delete_inode+0x8d/0x1e0 [lustre] [&lt;ffffffffa0f34000&gt;] ? return_if_equal+0x0/0x30 [lustre] [&lt;ffffffff811a70fe&gt;] generic_delete_inode+0xde/0x1d0 [&lt;ffffffff811a7255&gt;] generic_drop_inode+0x65/0x80 [&lt;ffffffff811a60a2&gt;] iput+0x62/0x70 [&lt;ffffffffa0f3590a&gt;] ll_d_iput+0xfa/0x820 [lustre] [&lt;ffffffff811a2c7c&gt;] dentry_iput+0x7c/0x100 [&lt;ffffffff811a2df1&gt;] d_kill+0x31/0x60 [&lt;ffffffff811a49cc&gt;] dput+0x7c/0x150 [&lt;ffffffff8119b1d9&gt;] do_rmdir+0x199/0x1f0 [&lt;ffffffff81196df6&gt;] ? final_putname+0x26/0x50 [&lt;ffffffff810e2257&gt;] ? audit_syscall_entry+0x1d7/0x200 [&lt;ffffffff8119b25d&gt;] sys_unlinkat+0x2d/0x40 [&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b </pre> </div></div> <p>which is only reported in <a href="https://jira.whamcloud.com/browse/LU-2048" title="Crash in llite_lib.c:1161:ll_clear_inode()) ASSERTION( lli-&gt;u.d.d_sai == ((void *)0) ) replay-vbr 7e" class="issue-link" data-issue-key="LU-2048"><del>LU-2048</del></a> (quite old !!!) and without any analyze/explanation !!….</p> <p>But unfortunately, kdump over the LAN/nfs has failed :</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>…………….. mapping eth0 to eth0 ADDRCONF(NETDEV_UP): eth0: link is not ready udhcpc (v1.15.1) started Sending discover... e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready Sending discover... Sending discover... Unable to get a DHCP address retsd 5:0:0:0: [sdf] Synchronizing SCSI cache ry... No lease, failing eth0 failed to come up Saving to remote location eagle-2.eagle.hpdd.intel.com:/export/scratch mount: RPC: Remote system error - Network is unreachable sd 4:0:0:0: [sde] Synchronizing SCSI cache sd 3:0:0:0: [sdd] Synchronizing SCSI cache sd 2:0:0:0: [sdc] Synchronizing SCSI cache sd 1:0:0:0: [sdb] Synchronizing SCSI cache sd 0:0:0:0: [sda] Synchronizing SCSI cache e1000e 0000:02:00.0: PCI INT A disabled e1000e 0000:01:00.0: PCI INT A disabled Restarting system. machine restart </pre> </div></div> <p>Hi Bruno, </p> <p> </p> <p>Here is the list of patches we apply to our servers kernel:</p> <p>0003-Applied-Bull_lmigr-x86_64.el6.0.7.patch-for-RHEL-2.6.patch<br/> Patch which delays migration of pages between numa nodes</p> <p>0006-Applied-Bull_311740-GENERIC_ALLOCATOR-4.patch<br/> Include generic memory allocator for x86 architecture.</p> <p>0008-Applied-Bull_Dynamic_Power_Limitation_2.6.32-573.patch<br/> CState manipulation interface through /sys</p> <p>0011-Applied-Bull_mce75xx_2.6.32-504.diff.patch<br/> Intel Xeon 7500 series corrected memory error driver</p> <p>0012-Applied-Bull_improve-x86_64.csum.patch.patch<br/> Performance improvement on TCP/IP checksumming</p> <p>0017-Applied-raid5-mmp-unplug-dev-rhel6.Bull.3.patch<br/> Forces sync reads on MD devices' disks for MMP compatibility</p> <p>0018-lustre-blkdev_tunables-2.6-rhel6.patch.patch<br/> Some block device parameters value modifications</p> <p>0021-lustre-bh_lru_size_config.patch.patch<br/> Buffer head per-cpu LRU size increased from 8 to 16</p> <p>0022-quota-replace-dqptr-sem.patch<br/> Replaces dqptr semaphore by spin locks (lustre server kernel patch)</p> <p>0029-Applied-proc_pid_limits.patch.patch<br/> Ease use of writes to /proc/&lt;pid&gt;/limits file</p> <p>0030-Applied-patch-Bull_oomprotect.diff.patch<br/> Patch to protect processes from root to be killed by oom</p> <p>0055-quota-avoid-dqget-calls.patch<br/> (lustre server kernel patch)</p> <p>0058-fix-CONFIG_XEN.patch<br/> Some modifications to have kvm work when XEN is not enabled</p> <p>0059-ocfs2-getname.patch<br/> fix getname/putname changes in ocfs2</p> <p>0090-kernel_H-2.6.32-573.1.1-config-x86_64.patch<br/> 0091-kernel_H-2.6.32-573.1.1-config-x86_64.debug.patch<br/> Some modifications to the default RHEL kernel configuration.</p> <p> </p> <p>For the lustre code, this is based on 2.5.3.90 (commit 35bb8577c83f787451cd9842a07ac670cb6f4d3f), with some patches that I will send you by mail.</p> <p>Thanks Seb, I don't see any Kernel/Lustre patch that could be related to the problem.</p> <p>Concerning the other LBUG, related to <a href="https://jira.whamcloud.com/browse/LU-2048" title="Crash in llite_lib.c:1161:ll_clear_inode()) ASSERTION( lli-&gt;u.d.d_sai == ((void *)0) ) replay-vbr 7e" class="issue-link" data-issue-key="LU-2048"><del>LU-2048</del></a>, that I have triggered during my first attempts to reproduce, it is a pure problem related to statahead code where some fixes appear to be missing in 2.5.3.90.</p> <p>On the other hand, I have been able to reproduce your LBUG by introducing the following tempo just before the LBUG :</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@eagle-28 lustre-release]# git status # On branch b2_5 # Changed but not updated: # (use "git add &lt;file&gt;..." to update what will be committed) # (use "git checkout -- &lt;file&gt;..." to discard changes in working directory) # # modified: lustre/include/obd_support.h # modified: lustre/llite/namei.c # # Untracked files: # (use "git add &lt;file&gt;..." to include in what will be committed) # # lustre/include/obd_support.h.bfi # lustre/llite/namei.c.bfi # lustre/scripts/make_META.pl no changes added to commit (use "git add" and/or "git commit -a") [root@eagle-28 lustre-release]# [root@eagle-28 lustre-release]# git diff diff --git a/lustre/include/obd_support.h b/lustre/include/obd_support.h index 37bee22..a21f8a0 100644 --- a/lustre/include/obd_support.h +++ b/lustre/include/obd_support.h @@ -430,6 +430,7 @@ int obd_alloc_fail(const void *ptr, const char *name, const char *type, #define OBD_FAIL_MDC_GETATTR_ENQUEUE 0x803 #define OBD_FAIL_MDC_RPCS_SEM 0x804 #define OBD_FAIL_MDC_LIGHTWEIGHT 0x805 +#define OBD_FAIL_MDC_CREATE_PAUSE 0x805 #define OBD_FAIL_MGS 0x900 #define OBD_FAIL_MGS_ALL_REQUEST_NET 0x901 diff --git a/lustre/llite/namei.c b/lustre/llite/namei.c index 39cd6d5..1cf3518 100644 --- a/lustre/llite/namei.c +++ b/lustre/llite/namei.c @@ -813,6 +813,8 @@ static struct inode *ll_create_node(struct inode *dir, const char *name, if (rc) GOTO(out, inode = ERR_PTR(rc)); + OBD_FAIL_TIMEOUT(OBD_FAIL_MDC_CREATE_PAUSE, 5); + LASSERT(ll_d_hlist_empty(&amp;inode-&gt;i_dentry)); /* We asked for a lock on the directory, but were granted a [root@eagle-28 lustre-release]# </pre> </div></div> <p>and by running the following snippet :</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@eagle-28 lustre-release]# date; lctl set_param fail_loc=0x80000805 ; echo foofour &gt; /mnt/lustre/foo8 &amp; sleep 2 ; date ; fid=$(lfs changelog lustre-MDT0000 | grep ' foo8' | sed 's/.* t=//' | sed 's/ p=.*//') ; date ; cat /mnt/lustre/.lustre/fid/$fid ; date Tue Jan 17 16:31:09 UTC 2017 fail_loc=0x80000805 [1] 27152 Tue Jan 17 16:31:11 UTC 2017 Tue Jan 17 16:31:11 UTC 2017 Tue Jan 17 16:31:11 UTC 2017 [root@eagle-28 lustre-release]# Message from syslogd@eagle-28 at Jan 17 16:31:14 ... kernel:LustreError: 27152:0:(namei.c:818:ll_create_node()) ASSERTION( list_empty(&amp;inode-&gt;i_dentry) ) failed: Message from syslogd@eagle-28 at Jan 17 16:31:14 ... kernel:LustreError: 27152:0:(namei.c:818:ll_create_node()) LBUG </pre> </div></div> <p>So this seems to mean that, as I have suspected from the start of our joint analysis, there is a potential race during file creation (including directory/dentry populate) and concurrent access using .lustre/fid method that is allowed to a ChangeLog reader, some kind of a<br/> "crime of insider trading". Or may be the LBUG is finally inaccurate and could be simply removed?</p> <p>More to come from crash-dump/debug-log analysis, and will also check if problem is still alive (like the same LBUG!) in master.</p> <p>Faccini Bruno (bruno.faccini@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/24976" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/24976</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8907" title="LBUG: (namei.c:816:ll_create_node()) ASSERTION( list_empty(&amp;inode-&gt;i_dentry) ) failed" class="issue-link" data-issue-key="LU-8907"><del>LU-8907</del></a> llite: handle client racy case during create<br/> Project: fs/lustre-release<br/> Branch: b2_5<br/> Current Patch Set: 1<br/> Commit: 376e27c4ac97d31f9729f351317891bff3f54147</p> <p>I have just pushed <a href="https://review.whamcloud.com/24976" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/24976</a> to b2_5 branch. It introduces sanity/test_161d sub-test in order to check that the reproducer I have identified also fails in the auto-tests environment in a first time. And then, with next patch-set that removing LBUG could fix and not introduce any regression.</p> <p>Next, will also check, using the same way, if problem also exists in master.</p> <p>Hi Bruno, did you see the problem reproduced in the auto tests environment ?</p> <p>Faccini Bruno (bruno.faccini@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/25296" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/25296</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8907" title="LBUG: (namei.c:816:ll_create_node()) ASSERTION( list_empty(&amp;inode-&gt;i_dentry) ) failed" class="issue-link" data-issue-key="LU-8907"><del>LU-8907</del></a> llite: handle client racy case during create<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 3d35f58c3b6c736e5db3d22dea2451bf634cdb95</p> <p>Yes, new sanity/test_161d sub-test is also a reproducer in auto-tests environment and it also demonstrate that problem is still alive in master!<br/> This has been proofed by patch-set #2 testing of <a href="https://review.whamcloud.com/25296" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/25296</a>, and now patch-set #3 introduces a first attempt to fix simply removing the inaccurate </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LASSERT(ll_d_hlist_empty(&amp;inode-&gt;i_dentry));</pre> </div></div> <p> in ll_create_node() and exposes it to full tests suites.</p> <p>Hello Bruno,</p> <p> </p> <p>What is the next step for this patch ?</p> <p>Are we waiting for reviewers ?</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/25296/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/25296/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8907" title="LBUG: (namei.c:816:ll_create_node()) ASSERTION( list_empty(&amp;inode-&gt;i_dentry) ) failed" class="issue-link" data-issue-key="LU-8907"><del>LU-8907</del></a> llite: handle client racy case during create<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: ee25babfe72378f9496a9732742984f26eb7d4a5</p> <p>Hello Seb,<br/> As you can see <a href="https://review.whamcloud.com/25296/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/25296/</a> has just landed on master. <br/> And as per its content (at least its pure fix part in core code, i.e. simply remove LBUG), it should be pretty easy for you to back-port it on top of your Lustre branch based on 2.5.3.90.</p> <p>Thanks Bruno.</p> <p>The patch was backported to our branch.</p> <p>This ticket can be closed.</p> <p>Thanks</p> <p>Landed for 2.10</p> Related Development Rank 1|hzyxgn: Rank (Obsolete) 9223372036854775807 Severity