https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7782 Lustre <p>This issue was created by maloo for Andreas Dilger &lt;andreas.dilger@intel.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.hpdd.intel.com/test_sets/f0270640-d4f4-11e5-9e3f-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/f0270640-d4f4-11e5-9e3f-5254006e85c2</a>.</p> <p>This is testing patch <a href="http://review.whamcloud.com/18442" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/18442</a> which is changing sanity-scrub.sh <tt>scrub_prep()</tt> to use <tt>test_mkdir -i</tt> instead of <tt>mkdir</tt> and <tt>lfs mkdir</tt> explicitly for testing, so that it works with the upstream kernel (which doesn't have DNE support).</p> <p>The sub-test test_2 failed with the following error on MDS2:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>08:17:18:LustreError: 29698:0:(client.c:1133:ptlrpc_import_delay_req()) Skipped 40 previous similar messages 08:17:18:BUG: unable to handle kernel NULL pointer dereference at 0000000000000010 08:17:18:IP: [&lt;ffffffffa057bb57&gt;] lu_context_key_get+0x1 08:17:18:CPU 0 08:17:18:Pid: 29699, comm: osp_up2-1 Not tainted 2.6.32-573.12.1.el6_lustre.g93f956d.x86_64 #1 Red Hat KVM 08:17:18:Call Trace: 08:17:18: [&lt;ffffffffa09ee83f&gt;] fld_local_lookup+0x4f/0x290 [fld] 08:17:18: [&lt;ffffffffa09eec83&gt;] fld_server_lookup+0x53/0x330 [fld] 08:17:18: [&lt;ffffffffa0e6e38f&gt;] lod_fld_lookup+0x34f/0x520 [lod] 08:17:18: [&lt;ffffffffa0e84243&gt;] lod_object_init+0x103/0x3c0 [lod] 08:17:18: [&lt;ffffffffa057f198&gt;] lu_object_alloc+0xd8/0x320 [obdclass] 08:17:18: [&lt;ffffffffa0580581&gt;] lu_object_find_try+0x151/0x260 [obdclass] 08:17:18: [&lt;ffffffffa0580741&gt;] lu_object_find_at+0xb1/0xe0 [obdclass] 08:17:18: [&lt;ffffffffa05807af&gt;] lu_object_find_slice+0x1f/0x80 [obdclass] 08:17:18: [&lt;ffffffffa0f79a4e&gt;] osp_trans_stop_cb+0x1be/0x2d0 [osp] 08:17:18: [&lt;ffffffffa0f7b2be&gt;] osp_update_interpret+0x21e/0x4a0 [osp] 08:17:18: [&lt;ffffffffa07900b5&gt;] ptlrpc_check_set+0x615/0x1da0 [ptlrpc] 08:17:18: [&lt;ffffffffa0791b9a&gt;] ptlrpc_set_wait+0x35a/0x960 [ptlrpc] 08:17:18: [&lt;ffffffffa0792221&gt;] ptlrpc_queue_wait+0x81/0x220 [ptlrpc] 08:17:18: [&lt;ffffffffa0f7b9c6&gt;] osp_send_update_req+0x256/0x850 [osp] 08:17:18: [&lt;ffffffffa0f7c63f&gt;] osp_send_update_thread+0x20f/0x7ac [osp] 08:17:18: [&lt;ffffffff810a0fce&gt;] kthread+0x9e/0xc0 </pre> </div></div> <p>Please provide additional information about the failure here.</p> <p>It shouldn't be possible to cause the node to crash, no matter how the test directories are being created.</p> <p>Info required for matching: sanity-scrub 2</p> LU-7782

sanity-scrub test_2: NULL pointer dereference at 0x10 in lu_context_key_get() on mds2

Bug Critical Resolved Fixed nasf Maloo Wed, 17 Feb 2016 01:10:36 +0000 Sat, 10 Sep 2016 04:08:17 +0000 Sat, 10 Sep 2016 04:08:17 +0000 Lustre 2.9.0 0 11 <p>This failed in sanity-scrub test_1a also: <a href="https://testing.hpdd.intel.com/test_sets/07f70048-d547-11e5-9cc2-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/07f70048-d547-11e5-9cc2-5254006e85c2</a></p> <p>Info required for matching: sanity-scrub 1a</p> <p>Wang Di, Fan Yong, could you please take a look at this? The test shouldn't cause the MDS to crash just because I slightly changed the way the directories are being created (apparently "test_mkdir" is creating a 2-stripe directory rather than a regular directory created by "mkdir"). Running sanity-scrub has failed 3x on the <a href="http://review.whamcloud.com/18442" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/18442</a> patch so it should be reproducible. </p> <p>I will investigate it.</p> <p>I cannot reproduce the issue in my local environment, but the test logs were clear, it shows that the crash happened when umount the MDTx after creating some striped directory. At that time, there was neither file-level backup/restore nor OI scrub running. File-level backup will happen after the umount, and the OI scrub will happen after the file-level backup/restore.</p> <p>It seems that when umount the MDTx, the OUT RPC for creating slave MDT-object of striped directory on remote MDT was NOT completed yet, its callback triggered object_init during MDT stack cleanup. If my guess correctly, this issue can be reproduced by repeatedly call scrub_prep() only.</p> <p>Di, would you please to check DNE async update logic? Thanks!</p> <p>Interesting, env from the request interrupt is not initialized at all, which should be the reason for this panic. See</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> * At least one request is in flight, so no * interrupts are allowed. Wait until all * complete, or an in-flight req times out. */ lwi = LWI_TIMEOUT(cfs_time_seconds(timeout? timeout : 1), ptlrpc_expired_set, set); rc = l_wait_event(set-&gt;set_waitq, ptlrpc_check_set(NULL, set), &amp;lwi); </pre> </div></div> <p>Reassigning to Di</p> <p>wangdi (di.wang@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/18493" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/18493</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7782" title="sanity-scrub test_2: NULL pointer dereference at 0x10 in lu_context_key_get() on mds2" class="issue-link" data-issue-key="LU-7782"><del>LU-7782</del></a> osp: re-initialize environment if necessary<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 1ad165b666050280a28befa6d37e9e0a7e09b9e7</p> <p>Looks like another instance on sanity-scrub test_10a at</p> <p>2016-02-21 18:51:44 - <a href="https://testing.hpdd.intel.com/test_sets/b5b59d62-d8de-11e5-b4e5-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/b5b59d62-d8de-11e5-b4e5-5254006e85c2</a></p> <p>+1 other instance on sanity-scrub test_10a at <a href="https://testing.hpdd.intel.com/test_sets/4e014c2e-db87-11e5-b8c9-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/4e014c2e-db87-11e5-b8c9-5254006e85c2</a></p> <p>Looks like this patch didn't fix sanity-scrub with my small patch to change to test_mkdir:<br/> <a href="https://testing.hpdd.intel.com/test_sets/dc48373e-d9d1-11e5-8b17-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/dc48373e-d9d1-11e5-8b17-5254006e85c2</a></p> <p>It looks different issue, and I did not see OOPs on any MDS node. Fan Yong, could you please check why LFSCK is stuck here? Thanks.</p> <p>The issues found by James and Bruno are the same as the original issue found by Andreas. But the latest failure instance in <a href="https://testing.hpdd.intel.com/test_sets/4e014c2e-db87-11e5-b8c9-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/4e014c2e-db87-11e5-b8c9-5254006e85c2</a> is different. That is because the current OSD cannot know the correct OI mapping for the slave MDT-object of the striped directory before the OI scrub completely rebuilt the OI files under MDT file level backup/restore case. I will make patch to handle it.</p> <p>Fan Yong (fan.yong@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/18801" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/18801</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7782" title="sanity-scrub test_2: NULL pointer dereference at 0x10 in lu_context_key_get() on mds2" class="issue-link" data-issue-key="LU-7782"><del>LU-7782</del></a> scrub: handle slave obj of striped directory<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 13c9ae64c78e1036f0464e78b0983f86e2383f1b</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/18493/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/18493/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7782" title="sanity-scrub test_2: NULL pointer dereference at 0x10 in lu_context_key_get() on mds2" class="issue-link" data-issue-key="LU-7782"><del>LU-7782</del></a> osp: save env for update callback<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 97510a3fcb1bc073fe9d45267cb541f3e1406d8d</p> <p>The 18801 patch also needs to land before this can be closed.</p> <p>seen in b2_8. I think the fix only went into master after the branch was made.</p> <p><a href="https://testing.hpdd.intel.com/test_sets/b2502df8-f950-11e5-812a-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/b2502df8-f950-11e5-812a-5254006e85c2</a></p> <p>Bob Glossman (bob.glossman@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/19313" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/19313</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7782" title="sanity-scrub test_2: NULL pointer dereference at 0x10 in lu_context_key_get() on mds2" class="issue-link" data-issue-key="LU-7782"><del>LU-7782</del></a> osp: save env for update callback<br/> Project: fs/lustre-release<br/> Branch: b2_8<br/> Current Patch Set: 1<br/> Commit: 079971d9943fa6f218c3b4188f0f6574e97b341d</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/18801/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/18801/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7782" title="sanity-scrub test_2: NULL pointer dereference at 0x10 in lu_context_key_get() on mds2" class="issue-link" data-issue-key="LU-7782"><del>LU-7782</del></a> scrub: handle slave obj of striped directory<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 80fe81c5b14835bbd5d751e878edbd00fe90f797</p> <p>Patches have landed to master for 2.9.0</p> <p>Oleg Drokin (oleg.drokin@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/21313" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/21313</a><br/> Subject: Revert "<a href="https://jira.whamcloud.com/browse/LU-7782" title="sanity-scrub test_2: NULL pointer dereference at 0x10 in lu_context_key_get() on mds2" class="issue-link" data-issue-key="LU-7782"><del>LU-7782</del></a> scrub: handle slave obj of striped directory"<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 60f7f0815007dffd709de93698bba3bd2380535c</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/21313/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/21313/</a><br/> Subject: Revert "<a href="https://jira.whamcloud.com/browse/LU-7782" title="sanity-scrub test_2: NULL pointer dereference at 0x10 in lu_context_key_get() on mds2" class="issue-link" data-issue-key="LU-7782"><del>LU-7782</del></a> scrub: handle slave obj of striped directory"<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 0f37c051158a399f7b00536eeec27f5dbdd54168</p> <p>The patch here was reverted because it appears to be causing multiple issues tracked under <a href="https://jira.whamcloud.com/browse/LU-8399" title="MDT hung at lu_object_find_at during umount" class="issue-link" data-issue-key="LU-8399"><del>LU-8399</del></a>, <a href="https://jira.whamcloud.com/browse/LU-8416" title="sanity-scrub test_4c: Auto trigger full scrub unexpectedly" class="issue-link" data-issue-key="LU-8416"><del>LU-8416</del></a> and others and just the fix in <a href="https://jira.whamcloud.com/browse/LU-8399" title="MDT hung at lu_object_find_at during umount" class="issue-link" data-issue-key="LU-8399"><del>LU-8399</del></a> was not enough to fix it.</p> <p>Fan Yong (fan.yong@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/21506" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/21506</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7782" title="sanity-scrub test_2: NULL pointer dereference at 0x10 in lu_context_key_get() on mds2" class="issue-link" data-issue-key="LU-7782"><del>LU-7782</del></a> scrub: handle slave obj of striped directory<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: bcde560652d19f66c0ddf650e895d620e87e3537</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/21506/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/21506/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7782" title="sanity-scrub test_2: NULL pointer dereference at 0x10 in lu_context_key_get() on mds2" class="issue-link" data-issue-key="LU-7782"><del>LU-7782</del></a> scrub: handle slave obj of striped directory<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 842bda9c5b41eef9e43dc3e00f05767147611677</p> <p>Landed for 2.9</p> Related LU-7746 LU-7935 LU-8399 LU-8416 Development Rank 1|hzy1hj: Rank (Obsolete) 9223372036854775807 Severity