https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7777 Lustre <p>We've started some client testing of TOSS 3 alpha against servers running toss-release-2.4-2.ch5.4. </p> <p>This is more of a heads up, perhaps there are already newer builds of the client in action somewhere? If not and you cannot reproduce I'll start an Intel issue.</p> <p>We can repeatably cause a kernel panic by doing active I/O to one file system (cp -ar &lt;something large&gt;) while simultaneously running ls -l in the other mounted file system, or occasionally in the same file system from a different session or node. This is regardless of the backing fs of the file system.</p> <p>Here's a representative trace:<br/> 12:27:45 <span class="error">&#91;10713.441784&#93;</span> BUG: unable to handle kernel paging request at 000000011df5000a<br/> 12:27:45 <span class="error">&#91;10713.442712&#93;</span> IP: <span class="error">&#91;&lt;ffffffffa0d20ed5&gt;&#93;</span> ll_get_dir_page+0x3e5/0x1650 <span class="error">&#91;lustre&#93;</span><br/> 12:27:45 <span class="error">&#91;10713.442712&#93;</span> PGD 40dacd067 PUD 0<br/> 12:27:45 <span class="error">&#91;10713.442712&#93;</span> Oops: 0000 <a href="#1" target="_blank" rel="noopener">1</a> SMP<br/> 12:27:45 <span class="error">&#91;10713.442712&#93;</span> Modules linked in: lmv(OE) fld(OE) mgc(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) fid(OE) ksocklnd(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) sha512_generic crypto_null libcfs(OE) xprtrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod crc_t10dif crct10dif_generic crct10dif_common ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_sa iTCO_wdt iTCO_vendor_support ipmi_devintf coretemp kvm radeon dcdbas ib_mthca ttm i5000_edac drm_kms_helper ib_mad lpc_ich edac_core mfd_core ib_core drm ipmi_si serio_raw pcspkr ib_addr ipmi_msghandler i5k_amb shpchp acpi_cpufreq binfmt_misc ip_tables nfsv4 dns_resolver nfsv3 nfs fscache lockd grace nfs_acl sunrpc broadcom tg3 bnx2 igb dca i2c_algo_bit i2c_core e1000e ptp pps_core e1000<br/> 12:27:45 <span class="error">&#91;10713.442712&#93;</span> CPU: 2 PID: 15157 Comm: bash Tainted: G OE ------------ 3.10.0-327.0.0.1chaos.ch6.x86_64 #1<br/> 12:27:45 <span class="error">&#91;10713.442712&#93;</span> Hardware name: Dell Inc. PowerEdge 1950/0NK937, BIOS 1.1.0 06/21/2006<br/> 12:27:45 <span class="error">&#91;10713.442712&#93;</span> task: ffff8803fd255080 ti: ffff8801310e8000 task.ti: ffff8801310e8000<br/> 12:27:45 <span class="error">&#91;10713.442712&#93;</span> RIP: 0010:<span class="error">&#91;&lt;ffffffffa0d20ed5&gt;&#93;</span> <span class="error">&#91;&lt;ffffffffa0d20ed5&gt;&#93;</span> ll_get_dir_page+0x3e5/0x1650 <span class="error">&#91;lustre&#93;</span><br/> 12:27:45 <span class="error">&#91;10713.442712&#93;</span> RSP: 0018:ffff8801310ebcb0 EFLAGS: 00010002<br/> 12:27:45 <span class="error">&#91;10713.442712&#93;</span> RAX: 0000000000000001 RBX: ffff880409560108 RCX: 0000000000000000<br/> 12:27:45 <span class="error">&#91;10713.442712&#93;</span> RDX: 000000011df5000a RSI: ffff8801310ebc60 RDI: 0000000000000000<br/> 12:27:45 <span class="error">&#91;10713.442712&#93;</span> RBP: ffff8801310ebdc8 R08: 0000000000000000 R09: ffffffffffffffff<br/> 12:27:45 <span class="error">&#91;10713.442712&#93;</span> R10: 0000000000000000 R11: 0000000000000220 R12: 0000000000000000<br/> 12:27:45 <span class="error">&#91;10713.442712&#93;</span> R13: fffffffffffffffe R14: 000000011df5000a R15: ffff880409560258<br/> 12:27:45 <span class="error">&#91;10713.442712&#93;</span> FS: 00002aaaaab086c0(0000) GS:ffff88042fc80000(0000) knlGS:0000000000000000<br/> 12:27:45 <span class="error">&#91;10713.442712&#93;</span> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033<br/> 12:27:45 <span class="error">&#91;10713.442712&#93;</span> CR2: 000000011df5000a CR3: 0000000361820000 CR4: 00000000000007e0<br/> 12:27:45 <span class="error">&#91;10713.442712&#93;</span> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000<br/> 12:27:45 <span class="error">&#91;10713.442712&#93;</span> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400<br/> 12:27:45 <span class="error">&#91;10713.442712&#93;</span> Stack:<br/> 12:27:45 <span class="error">&#91;10713.442712&#93;</span> ffff8801310ebce8 ffff8801310ebdf0 ffff8801310ebcd0 ffff880409560270<br/> 12:27:46 <span class="error">&#91;10713.442712&#93;</span> ffff880409560258 ffff880409560390 ffff8801310ebce8 a707e6e818891384<br/> 12:27:46 <span class="error">&#91;10713.442712&#93;</span> 0000000000000000 ffffffffa0d4191a ffff8801310ebd28 ffffffffa0d6b520<br/> 12:27:46 <span class="error">&#91;10713.442712&#93;</span> Call Trace:<br/> 12:27:46 <span class="error">&#91;10713.442712&#93;</span> <span class="error">&#91;&lt;ffffffffa0d4191a&gt;&#93;</span> ? __ll_inode_revalidate_it+0x1ea/0xc00 <span class="error">&#91;lustre&#93;</span><br/> 12:27:46 <span class="error">&#91;10713.442712&#93;</span> <span class="error">&#91;&lt;ffffffffa0d6b520&gt;&#93;</span> ? ll_symlink+0x1d0/0x1d0 <span class="error">&#91;lustre&#93;</span><br/> 12:27:46 <span class="error">&#91;10713.442712&#93;</span> <span class="error">&#91;&lt;ffffffff8120836e&gt;&#93;</span> ? mntput_no_expire+0x3e/0x120<br/> 12:27:46 <span class="error">&#91;10713.442712&#93;</span> <span class="error">&#91;&lt;ffffffff81208474&gt;&#93;</span> ? mntput+0x24/0x40<br/> 12:27:46 <span class="error">&#91;10713.442712&#93;</span> <span class="error">&#91;&lt;ffffffffa0d221e5&gt;&#93;</span> ll_dir_read+0xa5/0x380 <span class="error">&#91;lustre&#93;</span><br/> 12:27:46 <span class="error">&#91;10713.442712&#93;</span> <span class="error">&#91;&lt;ffffffff811fb990&gt;&#93;</span> ? fillonedir+0xf0/0xf0<br/> 12:27:46 <span class="error">&#91;10713.442712&#93;</span> <span class="error">&#91;&lt;ffffffff81647f9a&gt;&#93;</span> ? __slab_free+0x10e/0x277<br/> 12:27:46 <span class="error">&#91;10713.442712&#93;</span> <span class="error">&#91;&lt;ffffffffa0d2258b&gt;&#93;</span> ll_readdir+0xcb/0x360 <span class="error">&#91;lustre&#93;</span><br/> 12:27:46 <span class="error">&#91;10713.442712&#93;</span> <span class="error">&#91;&lt;ffffffff811fb990&gt;&#93;</span> ? fillonedir+0xf0/0xf0<br/> 12:27:46 <span class="error">&#91;10713.442712&#93;</span> <span class="error">&#91;&lt;ffffffff811fb990&gt;&#93;</span> ? fillonedir+0xf0/0xf0<br/> 12:27:46 <span class="error">&#91;10713.442712&#93;</span> <span class="error">&#91;&lt;ffffffff811fb870&gt;&#93;</span> vfs_readdir+0xb0/0xe0<br/> 12:27:46 <span class="error">&#91;10713.442712&#93;</span> <span class="error">&#91;&lt;ffffffff811fbce5&gt;&#93;</span> SyS_getdents+0x95/0x130<br/> 12:27:46 <span class="error">&#91;10713.442712&#93;</span> <span class="error">&#91;&lt;ffffffff8165c6c9&gt;&#93;</span> system_call_fastpath+0x16/0x1b<br/> 12:27:46 <span class="error">&#91;10713.442712&#93;</span> Code: 89 85 00 ff ff ff e8 6b 29 93 e0 49 8d 7f 08 b9 01 00 00 00 4c 89 ea 4c 89 f6 e8 87 53 5e e0 85 c0 0f 8e bf 05 00 00 4c 8b 75 88 &lt;49&gt; 8b 06 f6 c4 80 0f 85 27 11 00 00 f0 41 ff 46 1c 66 66 66 66<br/> 12:27:46 <span class="error">&#91;10713.442712&#93;</span> RIP <span class="error">&#91;&lt;ffffffffa0d20ed5&gt;&#93;</span> ll_get_dir_page+0x3e5/0x1650 <span class="error">&#91;lustre&#93;</span><br/> 12:27:46 <span class="error">&#91;10713.442712&#93;</span> RSP &lt;ffff8801310ebcb0&gt;<br/> 12:27:46 <span class="error">&#91;10713.442712&#93;</span> CR2: 000000011df5000a<br/> 12:27:46 <span class="error">&#91;10713.442712&#93;</span> --<del>[ end trace f520f054f8c4c1cb ]</del>--<br/> 12:27:46 <span class="error">&#91;10713.442712&#93;</span> Kernel panic - not syncing: Fatal exception</p> toss-release-3.0-36alpha.ch6 lustre 2.5.5-3chaos-CHANGED-3.10.0-327.0.0.1chaos.ch6.x86_64 LU-7777

toss 3 client kernel panic in ll_get_dir_page()

Bug Minor Resolved Duplicate Zhenyu Xu Ruth Klundt llnl Mon, 15 Feb 2016 21:50:07 +0000 Tue, 23 Feb 2016 17:59:14 +0000 Tue, 23 Feb 2016 17:59:14 +0000 Lustre 2.5.5 0 6 <p>ah so I created this in the wrong jira, but I was likely about to report this at Intel as well. Sorry for the confusion.</p> <p>Bobijam</p> <p>Could you please advise on this issue?</p> <p>Thanks</p> <p>Peter</p> <p>Update and clarification, the panic is repeatable also when one process is doing rm -rf on some part of the lustre tree, and another one is doing recursive ls -l. Also in the description I misspoke, the processes are not on different nodes, but the same node. The processes can be on the same file system or 2 different ones. Thanks for taking a look. </p> <p>can we know which code line does ll_get_dir_page+0x3e5 point at?</p> <p>I haven't got the source for this particular build, I've asked for it and will let you know. I believe it is built from the latest supported Intel 2.5.5 tag, but there may be other patches applied. </p> <p>Ruth </p> <p>We have access to the source, but Bobijam is hoping that you can run a gdb command to map the address referenced in the crash to the line of code affected. Are you familar with this process?</p> <p>Peter</p> <p>I was missing the debuginfo rpm, I have that now. it's a bit busy here today. I'm familiar with disassembly various ways, was going to do objdump but whatever is needed. probably tomorrow</p> <p>objdump is a bit overkill.</p> <p>Just use gdb on the lustre.ko (with debug symbols still in, out of the debuginfo rpm) and issue a "l *(ll_get_dir_page+0x3e5)" command.</p> <p>here you go, let me know if more is needed.</p> <p>(gdb) l *(ll_get_dir_page+0x3e5)<br/> 0x6ed5 is in ll_get_dir_page (/usr/src/kernels/3.10.0-327.0.0.1chaos.ch6.x86_64/arch/x86/include/asm/bitops.h:319).<br/> 314 }<br/> 315<br/> 316 static __always_inline int constant_test_bit(unsigned int nr, const volatile unsigned long *addr)<br/> 317 </p> { 318 return ((1UL &lt;&lt; (nr % BITS_PER_LONG)) &amp; 319 (addr[nr / BITS_PER_LONG])) != 0; 320 } <p>321<br/> 322 static inline int variable_test_bit(int nr, volatile const unsigned long *addr)<br/> 323 {</p> <p>hm, thats a bit less useful than we hoped, I guess.</p> <p>Can you substract 4 from the the 0x3e5 number until the resulting l command output in gdb will show us something in the ll_get_dir_page, please?</p> <p>sure, this looks a bit better:<br/> (gdb) l *(ll_get_dir_page+0x3e1)<br/> 0x6ed1 is in ll_get_dir_page (/usr/src/debug/lustre-2.5.5/lustre/llite/dir.c:282).<br/> 277 found = radix_tree_gang_lookup(&amp;mapping-&gt;page_tree,<br/> 278 (void **)&amp;page, offset, 1);<br/> 279 if (found &gt; 0) {<br/> 280 struct lu_dirpage *dp;<br/> 281<br/> 282 page_cache_get(page);<br/> 283 spin_unlock_irq(&amp;mapping-&gt;tree_lock);<br/> 284 /*<br/> 285 * In contrast to find_lock_page() we are sure that directory<br/> 286 * page cannot be truncated (while DLM lock is held) and,</p> <p>Ok, that clear it.</p> <p>Your bug is a duplicate of <a href="https://jira.whamcloud.com/browse/LU-5162" title="sanity test_65ic oops on rhel7" class="issue-link" data-issue-key="LU-5162"><del>LU-5162</del></a>, there's a patch too, so can you please close this as the dup once confirmed?</p> <p>thanks, will do</p> <p>fyi, the patch in <a href="https://jira.whamcloud.com/browse/LU-5162" title="sanity test_65ic oops on rhel7" class="issue-link" data-issue-key="LU-5162"><del>LU-5162</del></a> is for mdc_request.c, however the b2_5 tree does not have a call to radix_tree_gang_lookup in that file, although it does appear in llite/dir.c. So I'm doing the similar change there for testing.</p> <p>Intel, you will be providing a patch for b2_5_fe, I assume?</p> <p>Yup. One is in flight already and will be flagged when it is ready for you to pick up.</p> <p>I'm not able to trigger the oops with the following change built in:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>diff --git a/lustre/llite/dir.c b/lustre/llite/dir.c index 4f1b853..cf69cc4 100644 --- a/lustre/llite/dir.c +++ b/lustre/llite/dir.c @@ -276,7 +276,7 @@ static struct page *ll_dir_page_locate(struct inode *dir, __u64 *hash, spin_lock_irq(&amp;mapping-&gt;tree_lock); found = radix_tree_gang_lookup(&amp;mapping-&gt;page_tree, (void **)&amp;page, offset, 1); - if (found &gt; 0) { + if (found &gt; 0 &amp;&amp; !radix_tree_exceptional_entry(page)) { struct lu_dirpage *dp; page_cache_get(page); </pre> </div></div> <p>2.5.x FE fix for <a href="https://jira.whamcloud.com/browse/LU-5162" title="sanity test_65ic oops on rhel7" class="issue-link" data-issue-key="LU-5162"><del>LU-5162</del></a> ready to pick up</p> Related LU-5162 Development Rank 1|hzy1br: Rank (Obsolete) 9223372036854775807 Severity