https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7866 Lustre <p>Error occurred during soak testing of build '20160309' (b2_8 RC5) (see: <a href="https://wiki.hpdd.intel.com/display/Releases/Soak+Testing+on+Lola#SoakTestingonLola-20160309" class="external-link" target="_blank" rel="nofollow noopener">https://wiki.hpdd.intel.com/display/Releases/Soak+Testing+on+Lola#SoakTestingonLola-20160309</a> also). DNE is enabled. MDTs had been formatted using ldiskfs, OSTs using zfs. MDS nodes are configured in active - active HA failover configuration. (For teset set-up configuration see <a href="https://wiki.hpdd.intel.com/display/Releases/Soak+Testing+on+Lola#SoakTestingonLola-Configuration" class="external-link" target="_blank" rel="nofollow noopener">https://wiki.hpdd.intel.com/display/Releases/Soak+Testing+on+Lola#SoakTestingonLola-Configuration</a>)</p> <p>Sequence of events:</p> <ul> <li>mds_restart : 2016-03-11 03:41:05,597 - 2016-03-11 03:54:49,109 lola-8</li> <li>2016-03-11 03:56 Lustre client <tt>lola-32</tt> crashed with the following error: <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>&lt;1&gt;BUG: unable to handle kernel NULL pointer dereference at (null) &lt;1&gt;IP: [&lt;ffffffffa0a0241f&gt;] ll_open_cleanup+0xaf/0x600 [lustre] &lt;4&gt;PGD 38a867067 PUD 775372067 PMD 0 &lt;4&gt;Oops: 0000 [#1] SMP &lt;4&gt;last sysfs file: /sys/devices/pci0000:00/0000:00:02.0/0000:06:00.0/infiniband_mad/umad0/port &lt;4&gt;CPU 1 &lt;4&gt;Modules linked in: osc(U) mgc(U) lustre(U) lov(U) mdc(U) fid(U) lmv(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) sha512_generic crc32c_intel libcfs(U) nfsi] &lt;4&gt; &lt;4&gt;Pid: 201682, comm: simul Not tainted 2.6.32-504.30.3.el6.x86_64 #1 Intel Corporation S2600GZ/S2600GZ &lt;4&gt;RIP: 0010:[&lt;ffffffffa0a0241f&gt;] [&lt;ffffffffa0a0241f&gt;] ll_open_cleanup+0xaf/0x600 [lustre] &lt;4&gt;RSP: 0018:ffff8801d3f298e8 EFLAGS: 00010286 &lt;4&gt;RAX: 0000000000000000 RBX: ffff8807f7271a00 RCX: ffff88102dd13ca0 &lt;4&gt;RDX: 0000000000000002 RSI: 0000000000000000 RDI: ffff88102f395000 &lt;4&gt;RBP: ffff8801d3f29928 R08: ffff88034f47e9c0 R09: 0000000000000000 &lt;4&gt;R10: 0000000000000010 R11: 0000000000000000 R12: 0000000000000000 &lt;4&gt;R13: ffff88082dd02c00 R14: ffff88081a5fc800 R15: ffff8801d3f29988 &lt;4&gt;FS: 00007f72f89d3700(0000) GS:ffff880045e20000(0000) knlGS:0000000000000000 &lt;4&gt;CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b &lt;4&gt;CR2: 0000000000000000 CR3: 00000003b89f2000 CR4: 00000000001407e0 &lt;4&gt;DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 &lt;4&gt;DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 &lt;4&gt;Process simul (pid: 201682, threadinfo ffff8801d3f28000, task ffff8804c6d73520) &lt;4&gt;Stack: &lt;4&gt; ffff8801d3f298f8 0000000000000000 ffff8801d3f29928 0000000000000001 &lt;4&gt;&lt;d&gt; ffff88082dd02c00 ffff88081a5fc800 ffff8809648898c0 ffff8801d3f29988 &lt;4&gt;&lt;d&gt; ffff8801d3f299f8 ffffffffa0a09c4a fffffffffffffffb 00ff880a531788c0 &lt;4&gt;Call Trace: &lt;4&gt; [&lt;ffffffffa0a09c4a&gt;] ll_prep_inode+0x20a/0xc40 [lustre] &lt;4&gt; [&lt;ffffffffa07804b2&gt;] ? __req_capsule_get+0x162/0x6e0 [ptlrpc] &lt;4&gt; [&lt;ffffffffa0a214f0&gt;] ? ll_md_blocking_ast+0x0/0x7d0 [lustre] &lt;4&gt; [&lt;ffffffffa0a21fe1&gt;] ll_lookup_it_finish+0x321/0x12e0 [lustre] &lt;4&gt; [&lt;ffffffffa0723dd0&gt;] ? ldlm_lock_decref_internal+0x2e0/0xa80 [ptlrpc] &lt;4&gt; [&lt;ffffffffa0569925&gt;] ? class_handle2object+0x95/0x190 [obdclass] &lt;4&gt; [&lt;ffffffff81174ab3&gt;] ? kmem_cache_alloc_trace+0x1b3/0x1c0 &lt;4&gt; [&lt;ffffffffa0a1e439&gt;] ? ll_i2suppgid+0x19/0x30 [lustre] &lt;4&gt; [&lt;ffffffffa0a1e47e&gt;] ? ll_i2gids+0x2e/0xd0 [lustre] &lt;4&gt; [&lt;ffffffffa0a214f0&gt;] ? ll_md_blocking_ast+0x0/0x7d0 [lustre] &lt;4&gt; [&lt;ffffffffa0a23226&gt;] ll_lookup_it+0x286/0xda0 [lustre] &lt;4&gt; [&lt;ffffffffa0a23dc9&gt;] ll_lookup_nd+0x89/0x4f0 [lustre] &lt;4&gt; [&lt;ffffffff8119e055&gt;] do_lookup+0x1a5/0x230 &lt;4&gt; [&lt;ffffffff8119ece4&gt;] __link_path_walk+0x7a4/0x1000 &lt;4&gt; [&lt;ffffffff8119f7fa&gt;] path_walk+0x6a/0xe0 &lt;4&gt; [&lt;ffffffff8119fa0b&gt;] filename_lookup+0x6b/0xc0 &lt;4&gt; [&lt;ffffffff8122daa6&gt;] ? security_file_alloc+0x16/0x20 &lt;4&gt; [&lt;ffffffff811a0ee4&gt;] do_filp_open+0x104/0xd20 &lt;4&gt; [&lt;ffffffff81063c63&gt;] ? perf_event_task_sched_out+0x33/0x70 &lt;4&gt; [&lt;ffffffff8129943a&gt;] ? strncpy_from_user+0x4a/0x90 &lt;4&gt; [&lt;ffffffff811ae392&gt;] ? alloc_fd+0x92/0x160 &lt;4&gt; [&lt;ffffffff8118b157&gt;] do_sys_open+0x67/0x130 &lt;4&gt; [&lt;ffffffff8118b260&gt;] sys_open+0x20/0x30 &lt;4&gt; [&lt;ffffffff8100b0d2&gt;] system_call_fastpath+0x16/0x1b &lt;4&gt;Code: ba 38 01 00 00 31 f6 e8 30 62 b6 ff f6 05 fd 6b a9 ff 10 74 0d 80 3d f0 6b a9 ff 00 0f 88 ba 01 00 00 48 85 db 0f 84 16 02 00 00 &lt;49&gt; 8b 04 24 48 89 03 49 &lt;1&gt;RIP [&lt;ffffffffa0a0241f&gt;] ll_open_cleanup+0xaf/0x600 [lustre] &lt;4&gt; RSP &lt;ffff8801d3f298e8&gt; &lt;4&gt;CR2: 0000000000000000 </pre> </div></div></li> </ul> <p>Attached client (<tt>lola-32)</tt> message, console and vmcore-dmesg.txt file.</p> lola<br/> build: <a href="https://build.hpdd.intel.com/job/lustre-b2_8/12/">https://build.hpdd.intel.com/job/lustre-b2_8/12/</a> LU-7866

BUG: unable to handle kernel NULL pointer dereference at (null)

Bug Blocker Open Unresolved Hongchao Zhang Frank Heckes patch soak Fri, 11 Mar 2016 19:13:55 +0000 Sun, 30 Jan 2022 10:08:36 +0000 Lustre 2.8.0 0 6 <p>Crash file has been saved to <tt>lhn.lola.hpdd.intel.com:/scratch/crashdumps/lu-7866/lola-32/127.0.0.1-2016-03-11-03\:55\:54</tt></p> <p>The actual problem here is two lines above the crash:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>&lt;3&gt;LustreError: 201682:0:(layout.c:2025:__req_capsule_get()) @@@ Wrong buffer for field `mdt_body' (2 of 1) in format `LDLM_INTENT_OPEN': 0 vs. 216 (server) &lt;3&gt; req@ffff880e244a90c0 x1528368254991312/t489629526959(489629526959) o101-&gt;soaked-MDT0001-mdc-ffff88082dd02c00@192.168.1.108@o2ib10:12/10 lens 840/192 e 1 to 0 dl 1457697328 ref 2 fl Complete:R/4/0 rc -107/-107 </pre> </div></div> <p>This then causes the crash because:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>void ll_open_cleanup(struct super_block *sb, struct ptlrpc_request *open_req) { struct mdt_body *body; struct md_op_data *op_data; struct ptlrpc_request *close_req = NULL; struct obd_export *exp = ll_s2sbi(sb)-&gt;ll_md_exp; ENTRY; body = req_capsule_server_get(&amp;open_req-&gt;rq_pill, &amp;RMF_MDT_BODY); &lt;=== Returns NULL due to message above OBD_ALLOC_PTR(op_data); if (op_data == NULL) { CWARN("%s: cannot allocate op_data to release open handle for " DFID"\n", ll_get_fsname(sb, NULL, 0), PFID(&amp;body-&gt;mbo_fid1)); RETURN_EXIT; } op_data-&gt;op_fid1 = body-&gt;mbo_fid1; &lt;==== Whoops! </pre> </div></div> <p>Hongchao</p> <p>Can you please look into this? Oleg has suggested that you should</p> <p>1) add error handling<br/> 2. figure out why the request reply does not have what we think it should have inside</p> <p>Thanks</p> <p>Peter</p> <p>Frank - are these builds using the RC5 RPM or do you self-build them from tip of whatever branch? Because it says "build '20160302'", but if you self build it does not help us and you need to put debuginfo vmlinux and debuginfo modules alongside the crashdumps</p> <p>My apologies there's a typo in the description field above (I corrected it). The build under test was b2_8 rc5 and had been downloaded from Jenkens job <em>lustre-2_8</em> . The debuginfo RPMs can be found at <tt>lhn.lola.hpdd.intel.com:/scratch/rpms/20160309/notinstalled/server/x86_64</tt>. </p> <p>the related request has been actually failed with -ENOTCONN(-107), then there is not reply fields (Reply's bufcount is 1, only contains<br/> PTLRPC_BODY) in the request, still looking why the failed request can go to the "ll_prep_inode/ll_open_cleanup".</p> <p>status update: <br/> 1, the request is just change between ll_prep_inode and ll_open_cleanup, for the MDT_BODY is also retrieved in ll_prep_inode<br/> 2, in ll_prep_inode, ll_iget failed with -EIO, then the related FID could be a directory, and the GET_ATTR request to other MDTs failed</p> <p>It's not clear where the request is modified, and it could be changed by the replay, but there is no "P" flags in the request,<br/> and I also try to reproduce the problem by replaying the request before ll_open_cleanup use it, but failed to reproduce.</p> <p>Hi Frank,<br/> Could you please extract the debug logs from the crash dump? and I can't connect to the Lola.<br/> and is the logs at MDT0000 &amp; MDT0001 available? it could help to trace the problem.<br/> Thanks!!</p> <p>the issue can be reproduced if the request is replayed but failed between ll_prep_inod and ll_open_cleanup.<br/> I have managed to simulate this case in my local VMs and it occurred, but I'm not sure whether it is the same case with this one</p> <p>Yes, to extract the debug log from the crash file. For log information on MDS side (MDT000</p> {0,1} <p> I'm going to check, too and will attach the files here.</p> <p>Odd, I decompressed the Lustre client 'normal' kernel and the debuginfo kernel, both returned the <br/> following messages (besides different names):</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@lola-16 crash_lustre]# crash /scratch/crashdumps/lu-7866/lola-32/127.0.0.1-2016-03-11-03\:55\:54/vmcore /tmp/vmlinux-2.6.32-504.30.3.el6.x86_64 crash 6.1.0-6.el6_6 Copyright (C) 2002-2012 Red Hat, Inc. Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation Copyright (C) 1999-2006 Hewlett-Packard Co Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited Copyright (C) 2006, 2007 VA Linux Systems Japan K.K. Copyright (C) 2005, 2011 NEC Corporation Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc. Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc. This program is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Enter "help copying" to see the conditions. This program has absolutely no warranty. Enter "help warranty" for details. crash: /tmp/vmlinux-2.6.32-504.30.3.el6.x86_64: no .gnu_debuglink section crash: /tmp/vmlinux-2.6.32-504.30.3.el6.x86_64: no debugging data available </pre> </div></div> <p>Path to Oleg's <tt>lustre.so</tt> is configured via <tt>~/.crashrc</tt>. Any ideas?</p> <p>the issue should be caused by the race between the replay and the normal process of the open request.</p> <p>Thread 1:<br/> send the LDLM_ENQUEUE request to MDT1 to open the directory, and send requests to other MDTs to get the attributes<br/> after get the reply from the MDT1, but it was stuck by one of the requests to other MDTs.</p> <p>Thread 2:<br/> closing the file, and the "rq_replay" of the open request is cleared in "mdc_close".</p> <p>Then, the MDT1 got failover and the recovery is initiated, then the LDLM_ENQUEUE request in Thread 1 is replayed but failed<br/> with -ENOTCONN during replaying the request to MDT1 and its "rq_repmsg" will contain only the "PTLRPC_BODY" field.<br/> then the issue is triggered in "ll_open_cleanup".</p> <p>Hongchao Zhang (hongchao.zhang@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/19256" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/19256</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7866" title="BUG: unable to handle kernel NULL pointer dereference at (null)" class="issue-link" data-issue-key="LU-7866">LU-7866</a> ptlrpc: save reply msg for replay<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: d48a27d6e1a2b63dd9d2f2195dfed2e2d0595b78</p> Development Rank 1|hzy46f: Rank (Obsolete) 9223372036854775807 Severity