Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 [LU-5982] sanity-lfsck test_29c: (8) unexpected status, test_30: timeout https://jira.whamcloud.com/browse/LU-5982 Lustre <p>This issue was created by maloo for Li Wei &lt;liwei@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.hpdd.intel.com/test_sets/5278cc92-7b3d-11e4-8c6d-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/5278cc92-7b3d-11e4-8c6d-5254006e85c2</a>.</p> <p>The sub-test test_29c failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>(8) unexpected status </pre> </div></div> <p>More detailed test output:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== sanity-lfsck test 29c: Not verify nlink attr if hark links exceed linkEA limitation == 08:43:50 (1417625030) ##### There are too much hard links to the object, and exceeds the echo object's linkEA limitation, as to NOT all the known name entries will be recorded in the linkEA. Under such case, LFSCK should skip the nlink verification for this object. ##### rm: cannot remove `/mnt/lustre/.lustre/lost+found/MDT0001/[0x340000bd1:0xb:0x0]-P-0': Directory not empty Inject failure stub on MDT0 to simulate the case that foo's hard links exceed the object's linkEA limitation. CMD: onyx-32vm3 /usr/sbin/lctl set_param fail_loc=0x1627 fail_loc=0x1627 /mnt/lustre/d29c.sanity-lfsck/d0/foo /mnt/lustre/d29c.sanity-lfsck/d0/h1 Trigger namespace LFSCK to repair the nlink count CMD: onyx-32vm3 /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -r -A Started LFSCK on the device lustre-MDT0000: scrub namespace CMD: onyx-32vm3 /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | awk '/^status/ { print \$2 }' pdsh@onyx-32vm6: onyx-32vm3: mcmd: xpoll (setting up stderr): Interrupted system call CMD: onyx-32vm3 /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | awk '/^status/ { print \$2 }' onyx-32vm3: error: get_param: mdd/lustre-MDT0000/lfsck_namespace: Found no match Waiting 32 secs for update CMD: onyx-32vm3 /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | awk '/^status/ { print \$2 }' onyx-32vm3: error: get_param: mdd/lustre-MDT0000/lfsck_namespace: Found no match CMD: onyx-32vm3 /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | awk '/^status/ { print \$2 }' onyx-32vm3: error: get_param: mdd/lustre-MDT0000/lfsck_namespace: Found no match CMD: onyx-32vm3 /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | awk '/^status/ { print \$2 }' onyx-32vm3: error: get_param: mdd/lustre-MDT0000/lfsck_namespace: Found no match CMD: onyx-32vm3 /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | awk '/^status/ { print \$2 }' onyx-32vm3: error: get_param: mdd/lustre-MDT0000/lfsck_namespace: Found no match CMD: onyx-32vm3 /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | awk '/^status/ { print \$2 }' onyx-32vm3: error: get_param: mdd/lustre-MDT0000/lfsck_namespace: Found no match CMD: onyx-32vm3 /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | awk '/^status/ { print \$2 }' onyx-32vm3: error: get_param: mdd/lustre-MDT0000/lfsck_namespace: Found no match CMD: onyx-32vm3 /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | awk '/^status/ { print \$2 }' onyx-32vm3: error: get_param: mdd/lustre-MDT0000/lfsck_namespace: Found no match CMD: onyx-32vm3 /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | awk '/^status/ { print \$2 }' onyx-32vm3: error: get_param: mdd/lustre-MDT0000/lfsck_namespace: Found no match CMD: onyx-32vm3 /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | awk '/^status/ { print \$2 }' onyx-32vm3: error: get_param: mdd/lustre-MDT0000/lfsck_namespace: Found no match CMD: onyx-32vm3 /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | awk '/^status/ { print \$2 }' onyx-32vm3: error: get_param: mdd/lustre-MDT0000/lfsck_namespace: Found no match Waiting 22 secs for update CMD: onyx-32vm3 /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | awk '/^status/ { print \$2 }' onyx-32vm3: error: get_param: mdd/lustre-MDT0000/lfsck_namespace: Found no match CMD: onyx-32vm3 /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | awk '/^status/ { print \$2 }' onyx-32vm3: error: get_param: mdd/lustre-MDT0000/lfsck_namespace: Found no match CMD: onyx-32vm3 /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | awk '/^status/ { print \$2 }' onyx-32vm3: error: get_param: mdd/lustre-MDT0000/lfsck_namespace: Found no match CMD: onyx-32vm3 /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | awk '/^status/ { print \$2 }' onyx-32vm3: error: get_param: mdd/lustre-MDT0000/lfsck_namespace: Found no match CMD: onyx-32vm3 /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | awk '/^status/ { print \$2 }' onyx-32vm3: error: get_param: mdd/lustre-MDT0000/lfsck_namespace: Found no match CMD: onyx-32vm3 /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | awk '/^status/ { print \$2 }' onyx-32vm3: error: get_param: mdd/lustre-MDT0000/lfsck_namespace: Found no match CMD: onyx-32vm3 /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | awk '/^status/ { print \$2 }' onyx-32vm3: error: get_param: mdd/lustre-MDT0000/lfsck_namespace: Found no match CMD: onyx-32vm3 /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | awk '/^status/ { print \$2 }' onyx-32vm3: error: get_param: mdd/lustre-MDT0000/lfsck_namespace: Found no match CMD: onyx-32vm3 /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | awk '/^status/ { print \$2 }' onyx-32vm3: error: get_param: mdd/lustre-MDT0000/lfsck_namespace: Found no match CMD: onyx-32vm3 /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | awk '/^status/ { print \$2 }' onyx-32vm3: error: get_param: mdd/lustre-MDT0000/lfsck_namespace: Found no match Waiting 12 secs for update CMD: onyx-32vm3 /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | awk '/^status/ { print \$2 }' onyx-32vm3: error: get_param: mdd/lustre-MDT0000/lfsck_namespace: Found no match CMD: onyx-32vm3 /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | awk '/^status/ { print \$2 }' onyx-32vm3: error: get_param: mdd/lustre-MDT0000/lfsck_namespace: Found no match CMD: onyx-32vm3 /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | awk '/^status/ { print \$2 }' onyx-32vm3: error: get_param: mdd/lustre-MDT0000/lfsck_namespace: Found no match CMD: onyx-32vm3 /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | awk '/^status/ { print \$2 }' onyx-32vm3: error: get_param: mdd/lustre-MDT0000/lfsck_namespace: Found no match CMD: onyx-32vm3 /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | awk '/^status/ { print \$2 }' onyx-32vm3: error: get_param: mdd/lustre-MDT0000/lfsck_namespace: Found no match CMD: onyx-32vm3 /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | awk '/^status/ { print \$2 }' onyx-32vm3: error: get_param: mdd/lustre-MDT0000/lfsck_namespace: Found no match CMD: onyx-32vm3 /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | awk '/^status/ { print \$2 }' onyx-32vm3: error: get_param: mdd/lustre-MDT0000/lfsck_namespace: Found no match CMD: onyx-32vm3 /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | awk '/^status/ { print \$2 }' onyx-32vm3: error: get_param: mdd/lustre-MDT0000/lfsck_namespace: Found no match CMD: onyx-32vm3 /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | awk '/^status/ { print \$2 }' onyx-32vm3: error: get_param: mdd/lustre-MDT0000/lfsck_namespace: Found no match CMD: onyx-32vm3 /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | awk '/^status/ { print \$2 }' onyx-32vm3: error: get_param: mdd/lustre-MDT0000/lfsck_namespace: Found no match Waiting 2 secs for update CMD: onyx-32vm3 /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | awk '/^status/ { print \$2 }' onyx-32vm3: error: get_param: mdd/lustre-MDT0000/lfsck_namespace: Found no match CMD: onyx-32vm3 /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | awk '/^status/ { print \$2 }' onyx-32vm3: error: get_param: mdd/lustre-MDT0000/lfsck_namespace: Found no match Update not seen after 32s: wanted 'completed' got '' CMD: onyx-32vm3 /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace onyx-32vm3: error: get_param: mdd/lustre-MDT0000/lfsck_namespace: Found no match sanity-lfsck test_29c: @@@@@@ FAIL: (8) unexpected status </pre> </div></div> <p>Info required for matching: sanity-lfsck 29c</p> LU-5982 sanity-lfsck test_29c: (8) unexpected status, test_30: timeout Bug Blocker Resolved Fixed nasf Maloo Thu, 4 Dec 2014 02:55:38 +0000 Sat, 6 Dec 2014 23:04:17 +0000 Sat, 6 Dec 2014 23:04:17 +0000 Lustre 2.7.0 Lustre 2.7.0 0 7 <p>I was testing a recent build of master, #2760, on the OpenSFS cluster and ran into the same test failure. On the MDS console in my testing, I see </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Message from syslogd@mds01 at Dec 3 16:38:19 ... kernel:LustreError: 29360:0:(osd_handler.c:1571:osd_object_write_unlock()) ASSERTION( obj-&gt;oo_owner == env ) failed: Message from syslogd@mds01 at Dec 3 16:38:19 ... kernel:LustreError: 29360:0:(osd_handler.c:1571:osd_object_write_unlock()) LBUG </pre> </div></div> <p>Then the MDS crashed. </p> <p>Looking at your test logs, it looks like the primary MDS crashed:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>09:03:40:Lustre: DEBUG MARKER: == sanity-lfsck test 29c: Not verify nlink attr if hark links exceed linkEA limitation == 08:43:50 (1417625030) 09:03:40:Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1627 09:03:40:Lustre: *** cfs_fail_loc=1627, val=0*** 09:03:40:Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -r -A 09:03:40:LustreError: 28234:0:(osd_handler.c:1571:osd_object_write_unlock()) ASSERTION( obj-&gt;oo_owner == env ) failed: 09:03:40:LustreError: 28234:0:(osd_handler.c:1571:osd_object_write_unlock()) LBUG 09:03:40:Pid: 28234, comm: lfsck_namespace 09:03:40: 09:03:40:Call Trace: 09:03:40: [&lt;ffffffffa048e895&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] 09:03:40: [&lt;ffffffffa048ee97&gt;] lbug_with_loc+0x47/0xb0 [libcfs] 09:03:40: [&lt;ffffffffa0d1cb66&gt;] osd_object_write_unlock+0xd6/0x110 [osd_ldiskfs] 09:03:40: [&lt;ffffffffa0e3fcd2&gt;] dt_write_unlock+0x22/0xc0 [lfsck] 09:03:40: [&lt;ffffffffa0e51dfc&gt;] lfsck_namespace_insert_orphan+0xc9c/0x1100 [lfsck] 09:03:40: [&lt;ffffffff8128d756&gt;] ? vsnprintf+0x336/0x5e0 09:03:40: [&lt;ffffffffa0e59d26&gt;] lfsck_namespace_double_scan_dir+0x9d6/0xe40 [lfsck] 09:03:40: [&lt;ffffffffa04a3ee9&gt;] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] 09:03:40: [&lt;ffffffffa0e5a514&gt;] lfsck_namespace_double_scan_one+0x384/0x1310 [lfsck] 09:03:40: [&lt;ffffffffa05fe163&gt;] ? lu_object_find_at+0xb3/0x100 [obdclass] 09:03:40: [&lt;ffffffffa0e5c444&gt;] lfsck_namespace_assistant_handler_p2+0xfa4/0x1140 [lfsck] 09:03:40: [&lt;ffffffffa0e3e1bc&gt;] lfsck_assistant_engine+0x148c/0x1dc0 [lfsck] 09:03:40: [&lt;ffffffff81061d00&gt;] ? default_wake_function+0x0/0x20 09:03:40: [&lt;ffffffffa0e3cd30&gt;] ? lfsck_assistant_engine+0x0/0x1dc0 [lfsck] 09:03:41: [&lt;ffffffff8109abf6&gt;] kthread+0x96/0xa0 09:03:41: [&lt;ffffffff8100c20a&gt;] child_rip+0xa/0x20 09:03:41: [&lt;ffffffff8109ab60&gt;] ? kthread+0x0/0xa0 09:03:41: [&lt;ffffffff8100c200&gt;] ? child_rip+0x0/0x20 09:03:41: 09:03:41:Kernel panic - not syncing: LBUG 09:03:41:Pid: 28234, comm: lfsck_namespace Not tainted 2.6.32-431.29.2.el6_lustre.g2856d31.x86_64 #1 09:03:41:Call Trace: 09:03:41: [&lt;ffffffff81528fdc&gt;] ? panic+0xa7/0x16f 09:03:41:Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace | 09:03:41: awk '/^status/ { print $2 }' 09:03:41: [&lt;ffffffffa048eeeb&gt;] ? lbug_with_loc+0x9b/0xb0 [libcfs] 09:03:41: [&lt;ffffffffa0d1cb66&gt;] ? osd_object_write_unlock+0xd6/0x110 [osd_ldiskfs] 09:03:41: [&lt;ffffffffa0e3fcd2&gt;] ? dt_write_unlock+0x22/0xc0 [lfsck] 09:03:41: [&lt;ffffffffa0e51dfc&gt;] ? lfsck_namespace_insert_orphan+0xc9c/0x1100 [lfsck] 09:03:41: [&lt;ffffffff8128d756&gt;] ? vsnprintf+0x336/0x5e0 09:03:41: [&lt;ffffffffa0e59d26&gt;] ? lfsck_namespace_double_scan_dir+0x9d6/0xe40 [lfsck] 09:03:41: [&lt;ffffffffa04a3ee9&gt;] ? cfs_hash_bd_add_locked+0x29/0x90 [libcfs] 09:03:41: [&lt;ffffffffa0e5a514&gt;] ? lfsck_namespace_double_scan_one+0x384/0x1310 [lfsck] 09:03:41: [&lt;ffffffffa05fe163&gt;] ? lu_object_find_at+0xb3/0x100 [obdclass] 09:03:41: [&lt;ffffffffa0e5c444&gt;] ? lfsck_namespace_assistant_handler_p2+0xfa4/0x1140 [lfsck] 09:03:41: [&lt;ffffffffa0e3e1bc&gt;] ? lfsck_assistant_engine+0x148c/0x1dc0 [lfsck] 09:03:41: [&lt;ffffffff81061d00&gt;] ? default_wake_function+0x0/0x20 09:03:41: [&lt;ffffffffa0e3cd30&gt;] ? lfsck_assistant_engine+0x0/0x1dc0 [lfsck] 09:03:41: [&lt;ffffffff8109abf6&gt;] ? kthread+0x96/0xa0 09:03:41: [&lt;ffffffff8100c20a&gt;] ? child_rip+0xa/0x20 09:03:41: [&lt;ffffffff8109ab60&gt;] ? kthread+0x0/0xa0 09:03:41: [&lt;ffffffff8100c200&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p>Hit this recently in <a href="https://testing.hpdd.intel.com/test_sets/13c4439a-7b11-11e4-9132-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/13c4439a-7b11-11e4-9132-5254006e85c2</a> as well. Looks like this is a new regression. </p> <p>More instances on master branch:<br/> <a href="https://testing.hpdd.intel.com/test_sets/8015e966-7b72-11e4-813e-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/8015e966-7b72-11e4-813e-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/c643440e-7bd8-11e4-813e-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/c643440e-7bd8-11e4-813e-5254006e85c2</a></p> <p><a href="https://testing.hpdd.intel.com/test_sets/4ceb1126-7b57-11e4-8c6d-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/4ceb1126-7b57-11e4-8c6d-5254006e85c2</a></p> <p>Fan Yong (fan.yong@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/12943" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12943</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5982" title="sanity-lfsck test_29c: (8) unexpected status, test_30: timeout" class="issue-link" data-issue-key="LU-5982"><del>LU-5982</del></a> lfsck: not unlock the object repeatedly<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 5d9c9240c598eb0bed054679440acb555f3bff1f</p> <p>This problem was added via <a href="http://review.whamcloud.com/11536" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/11536</a> "<a href="https://jira.whamcloud.com/browse/LU-5518" title="LFSCK 3: recover client visible objects from the backend /lost+found directory" class="issue-link" data-issue-key="LU-5518"><del>LU-5518</del></a> lfsck: recover orphans from backend lost+found"</p> <p>Andreas Dilger (andreas.dilger@intel.com) merged in patch <a href="http://review.whamcloud.com/12943/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12943/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5982" title="sanity-lfsck test_29c: (8) unexpected status, test_30: timeout" class="issue-link" data-issue-key="LU-5982"><del>LU-5982</del></a> lfsck: not unlock the object repeatedly<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 8ee9c59a3688e5026ebb914190b63970135ce2e5</p> <p>Patch landed to master. </p> Related LU-5518 Development Rank 1|hzx1vj: Rank (Obsolete) 16695 Severity