https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11519 Lustre <p>This issue was created by maloo for Andreas Dilger &lt;adilger@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run review-dne-zfs-part-2:<br/> <a href="https://testing.whamcloud.com/test_sets/864ae762-d020-11e8-82f2-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/864ae762-d020-11e8-82f2-52540065bddc</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>BUG: unable to handle kernel paging request at 00000000240cd204 IP: mdt_cdt_waiting_cb.isra.25+0x357/0xc10 [mdt] CPU: 0 PID: 25301 Comm: hsm_cdtr Kdump: loaded Tainted: P RIP: 0010:[&lt;ffffffffc1230617&gt;] [&lt;ffffffffc1230617&gt;] mdt_cdt_waiting_cb.isra.25+0x357/0xc10 [mdt] Call Trace: mdt_coordinator_cb+0x162/0x290 [mdt] llog_process_thread+0x852/0x1550 [obdclass] llog_process_or_fork+0xbc/0x450 [obdclass] llog_cat_process_cb+0x239/0x250 [obdclass] llog_process_thread+0x852/0x1550 [obdclass] llog_process_or_fork+0xbc/0x450 [obdclass] llog_cat_process_or_fork+0x199/0x2a0 [obdclass] llog_cat_process+0x2e/0x30 [obdclass] cdt_llog_process+0xc6/0x3a0 [mdt] mdt_coordinator+0x541/0x19f0 [mdt] kthread+0xd1/0xe0 ret_from_fork_nospec_begin+0x21/0x21 </pre> </div></div> LU-11519

sanity-hsm: test_90 MDS crash in mdt_cdt_waiting_cb()

Bug Major Resolved Fixed John Hammond Maloo Mon, 15 Oct 2018 14:35:12 +0000 Wed, 19 Dec 2018 21:21:16 +0000 Tue, 27 Nov 2018 05:12:42 +0000 Lustre 2.12.0 Lustre 2.12.0 0 10 <p>+1 in review-dne-part-2 test:<br/> <a href="https://testing.whamcloud.com/test_sets/32eb66fe-d3a5-11e8-ad90-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/32eb66fe-d3a5-11e8-ad90-52540065bddc</a></p> <p>We are seeing sanity-hsm test 13 crash at about a 5% failure rate with this call trace. All are for DNE with ZFS configurations.</p> <p>Some example are at<br/> <a href="https://testing.whamcloud.com/test_sets/b61b466e-d34d-11e8-9238-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/b61b466e-d34d-11e8-9238-52540065bddc</a><br/> <a href="https://testing.whamcloud.com/test_sets/d47f7fc4-d313-11e8-82f2-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/d47f7fc4-d313-11e8-82f2-52540065bddc</a></p> <p>+1 on master: <a href="https://testing.whamcloud.com/test_sets/23d994e6-d644-11e8-ad90-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/23d994e6-d644-11e8-ad90-52540065bddc</a></p> <p>The crash dump cannot be loaded to analysis. </p> <p>+1 on master <a href="https://testing.whamcloud.com/test_sets/30e877d2-d6d1-11e8-9238-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/30e877d2-d6d1-11e8-9238-52540065bddc</a></p> <p>There are actually many failures like this. </p> <p>John, could you please take a look at this, it looks like a coordinator bug and has been crashing the MDS fairly commonly lately.</p> <p>I grabbed the most recent failure from maloo <a href="https://testing.whamcloud.com/test_sets/27781c7e-dde4-11e8-975a-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/27781c7e-dde4-11e8-975a-52540065bddc</a> which uses build 59629:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 5043.117404] BUG: unable to handle kernel NULL pointer dereference at 0000000000000004 [ 5043.119825] IP: [&lt;ffffffffc1327747&gt;] mdt_cdt_waiting_cb.isra.25+0x357/0xc10 [mdt] [ 5043.122276] PGD 0 $ mkdir /tmp/LU-11519 $ cd /tmp/LU-11519 $ wget 'https://build.whamcloud.com/job/lustre-reviews/59629/arch=x86_64,build_type=server,distro=el7,ib_stack=inkernel/artifact/artifacts/RPMS/x86_64/lustre-debuginfo-2.11.56_56_g098be7d-1.el7.x86_64.rpm' ... $ rpm2cpio lustre-debuginfo-2.11.56_56_g098be7d-1.el7.x86_64.rpm | cpio -id $ nm ./usr/lib/debug/lib/modules/3.10.0-862.14.4.el7_lustre.x86_64/extra/lustre/fs/mdt.ko.debug | awk '$3 == "mdt_cdt_waiting_cb.isra.25" { print $1; }' 000000000005f3f0 $ printf '%#lx\n' $(( 0x357 + 0x5f3f0 )) 0x5f747 $ addr2line -e ./usr/lib/debug/lib/modules/3.10.0-862.14.4.el7_lustre.x86_64/extra/lustre/fs/mdt.ko.debug --functions --inlines 0x5f747 mdt_cdt_waiting_cb /usr/src/debug/lustre-2.11.56_56_g098be7d/lustre/mdt/mdt_coordinator.c:228 $ cat -n ./usr/src/debug/lustre-2.11.56_56_g098be7d/lustre/mdt/mdt_coordinator.c | sed -n '220,230p' 220 } while (request-&gt;hal_used_sz + hai_size &gt; 221 LDLM_MAXREQSIZE); 222 } else if (hsd-&gt;hsd_housekeeping) { 223 struct hsm_scan_request *tmp; 224 225 /* Discard the (whole) last hal */ 226 hsd-&gt;hsd_request_count--; 227 tmp = &amp;hsd-&gt;hsd_request[hsd-&gt;hsd_request_count]; 228 hsd-&gt;hsd_action_count -= tmp-&gt;hal-&gt;hal_count; 229 OBD_FREE(tmp-&gt;hal, tmp-&gt;hal_sz); 230 } else { </pre> </div></div> <p>So it looks like tmp-&gt;hal is NULL.</p> <blockquote> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>$ nm ... $ printf .. $ addr2line -e ... $ cat -n ... </pre> </div></div></blockquote> <p>I typically use the following, which is considerably easier:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>$ gdb .../mdt.ko.debug -ex "list *(mdt_cdt_waiting_cb+0x357)" -ex quit (gdb) list *(mdt_cdt_waiting_cb+0x357) 0x5f777 is in mdt_cdt_waiting_cb (/usr/src/lustre-head/lustre/mdt/mdt_coordinator.c:228). 223 struct hsm_scan_request *tmp; 224 225 /* Discard the (whole) last hal */ 226 hsd-&gt;hsd_request_count--; 227 tmp = &amp;hsd-&gt;hsd_request[hsd-&gt;hsd_request_count]; 228 hsd-&gt;hsd_action_count -= tmp-&gt;hal-&gt;hal_count; 229 OBD_FREE(tmp-&gt;hal, tmp-&gt;hal_sz); 230 } else { 231 /* Bailing out, this code path is too hot */ 232 RETURN(LLOG_PROC_BREAK); </pre> </div></div> <p>Note the "<tt>list *()</tt>" argument is <em>typically</em> the same format as how addresses are printed from stack dumps/oops messages, but in this case the "<tt>.isra.25</tt>" part needs to be dropped.</p> <p>sanity-hsm test 13 also hit the same failure in review-dne-zfs-part-2 test session on master branch:<br/> <a href="https://testing.whamcloud.com/test_sets/ee81ca36-defc-11e8-a251-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/ee81ca36-defc-11e8-a251-52540065bddc</a></p> <p>sanity-hsm test 13 hit this failure again:<br/> <a href="https://testing.whamcloud.com/test_sets/b6a66416-dfe5-11e8-89f8-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/b6a66416-dfe5-11e8-89f8-52540065bddc</a></p> <p>&gt; I typically use the following, which is considerably easier:</p> <p>(I like my way because it makes inlining much easier to understand. Which is not an issue here. Maybe gcc has a better way to do this in those cases.)</p> <p>John L. Hammond (jhammond@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/33580" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33580</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11519" title="sanity-hsm: test_90 MDS crash in mdt_cdt_waiting_cb()" class="issue-link" data-issue-key="LU-11519"><del>LU-11519</del></a> hsm: handle hsd_request_count == 0 properly<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 5021ced380ad820a5f92ac1b41363f5919d79e16</p> <p>Quentin Bouget (quentin.bouget@cea.fr) uploaded a new patch: <a href="https://review.whamcloud.com/33590" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33590</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11519" title="sanity-hsm: test_90 MDS crash in mdt_cdt_waiting_cb()" class="issue-link" data-issue-key="LU-11519"><del>LU-11519</del></a> hsm: improve the testing of hsm.max_requests<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: f56c3963b1314e641044464b50e8a549762c751e</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/33580/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33580/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11519" title="sanity-hsm: test_90 MDS crash in mdt_cdt_waiting_cb()" class="issue-link" data-issue-key="LU-11519"><del>LU-11519</del></a> hsm: handle hsd_request_count == 0 properly<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: af05afdfb5781806ab3bc059c86c289b01713ade</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/33590/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33590/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11519" title="sanity-hsm: test_90 MDS crash in mdt_cdt_waiting_cb()" class="issue-link" data-issue-key="LU-11519"><del>LU-11519</del></a> hsm: improve the testing of hsm.max_requests<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: c1df28b34acdb56acff444400fc9a05d8adf493a</p> <p>Landed for 2.12</p> Duplicate LU-11630 Development Rank 1|i0047j: Rank (Obsolete) 9223372036854775807 Severity