Whamcloud Community JIRA

Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us

9.4.14

940014

05-12-2023

[LU-6147] sanity-lfsck test_4: '(7) unexpected status' https://jira.whamcloud.com/browse/LU-6147 Lustre <p>This issue was created by maloo for John Hammond <john.hammond@intel.com></p> <p>This issue relates to the following test suite run: <a href="https://testing.hpdd.intel.com/test_sets/7626441a-a14b-11e4-807a-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/7626441a-a14b-11e4-807a-5254006e85c2</a>.</p> <p>The sub-test test_4 failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>(7) unexpected status </pre> </div></div> <p>Please provide additional information about the failure here.</p> <p>Info required for matching: sanity-lfsck 4</p> LU-6147

sanity-lfsck test_4: '(7) unexpected status'

Bug Blocker Resolved Fixed nasf Maloo Thu, 22 Jan 2015 15:57:18 +0000 Wed, 15 Jul 2015 12:43:19 +0000 Tue, 3 Feb 2015 17:41:18 +0000 Lustre 2.7.0 0 9 <p>When I runn this locally I see an lfsck thread asleep in lu_object_find_at():</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 515.234216] Lustre: DEBUG MARKER: == sanity-lfsck test 4: FID-in-dirent can be rebuilt after MDT file-level backup/restore == 10:23:00 (1421943780) [ 521.780085] Lustre: Failing over lustre-MDT0000 [ 524.984187] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.122.201@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 524.987971] LustreError: Skipped 13 previous similar messages [ 527.956175] Lustre: 5093:0:(client.c:1942:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1421943787/real 1421943787] req@ffff8 80064738188 x1491015708973216/t0(0) o251->MGC192.168.122.200@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1421943793 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 527.961754] Lustre: 5093:0:(client.c:1942:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 528.510534] Lustre: server umount lustre-MDT0000 complete [ 529.308146] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. quota=on. Opts: [ 531.180857] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. quota=on. Opts: [ 531.902794] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. quota=on. Opts: [ 537.656353] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. quota=on. Opts: [ 539.324451] Lustre: *** cfs_fail_loc=1601, val=0*** [ 547.009622] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 547.011917] Lustre: lustre-MDT0000: Denying connection for new client lustre-MDT0000-lwp-MDT0001_UUID (at 192.168.122.201@tcp), waiting for all 3 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 1:06 [ 547.128862] Lustre: lustre-MDT0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 578.733248] Lustre: DEBUG MARKER: sanity-lfsck test_4: @@@@@@ FAIL: (7) unexpected status [ 724.197393] INFO: task lfsck:5461 blocked for more than 120 seconds. [ 724.198800] Not tainted 2.6.32-431.29.2.el6.lustre.x86_64 #1 [ 724.200136] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 724.201748] lfsck D 0000000000000000 0 5461 2 0x00000080 [ 724.203226] ffff880067f37c00 0000000000000046 0000000000000000 ffff880065abe188 [ 724.204927] ffff880067f37b70 ffffffffa03183fe ffff880067f37c00 ffffffffa04a5cd9 [ 724.206596] ffff880062298cc0 ffff880067f37fd8 000000000000fcc8 ffff880062298cc0 [ 724.208431] Call Trace: [ 724.208981] [<ffffffffa03183fe>] ? cfs_hash_spin_unlock+0xe/0x10 [libcfs] [ 724.210466] [<ffffffffa04a5cd9>] ? lu_object_find_try+0x99/0x2b0 [obdclass] [ 724.211890] [<ffffffffa04a5f2d>] lu_object_find_at+0x3d/0xe0 [obdclass] [ 724.212765] [<ffffffff8155625b>] ? _read_unlock+0x2b/0x40 [ 724.213476] [<ffffffff81061d90>] ? default_wake_function+0x0/0x20 [ 724.214280] [<ffffffffa04a600f>] lu_object_find_slice+0x1f/0x80 [obdclass] [ 724.215198] [<ffffffffa09e0838>] lfsck_master_oit_engine+0x608/0x1f40 [lfsck] [ 724.216154] [<ffffffff810b74bd>] ? trace_hardirqs_on+0xd/0x10 [ 724.216999] [<ffffffffa09e2da1>] lfsck_master_engine+0xc31/0x14b0 [lfsck] [ 724.217933] [<ffffffff81553065>] ? thread_return+0x4e/0x7e9 [ 724.218670] [<ffffffff81061d90>] ? default_wake_function+0x0/0x20 [ 724.219474] [<ffffffffa09e2170>] ? lfsck_master_engine+0x0/0x14b0 [lfsck] [ 724.220382] [<ffffffff8109e856>] kthread+0x96/0xa0 [ 724.220999] [<ffffffff8100c30a>] child_rip+0xa/0x20 [ 724.221665] [<ffffffff815562e0>] ? _spin_unlock_irq+0x30/0x40 [ 724.222430] [<ffffffff8100bb10>] ? restore_args+0x0/0x30 [ 724.223110] [<ffffffff8109e7c0>] ? kthread+0x0/0xa0 [ 724.223796] [<ffffffff8100c300>] ? child_rip+0x0/0x20 </pre> </div></div> <p>It is another failure instance of <a href="https://jira.whamcloud.com/browse/LU-6109" title="LFSCK gets "inconsistent" flag and won't stop" class="issue-link" data-issue-key="LU-6109"><del>LU-6109</del></a> because of LFSCK deadlock with itself when try to find some special object, such as .lustre/lost+found/MDTxxxx</p> <p>John, would you please to verify whether the patch <a href="http://review.whamcloud.com/#/c/13493/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/13493/</a> works for you or not? Thanks!</p> <p>Yes. It works for me.</p> <p>Thanks John.</p> <p>The <a href="https://jira.whamcloud.com/browse/LU-6109" title="LFSCK gets "inconsistent" flag and won't stop" class="issue-link" data-issue-key="LU-6109"><del>LU-6109</del></a> failed for more complex reason, we need more investigate it.<br/> But the reason for this failure (<a href="https://jira.whamcloud.com/browse/LU-6147" title="sanity-lfsck test_4: '(7) unexpected status' " class="issue-link" data-issue-key="LU-6147"><del>LU-6147</del></a>) is known. So handle it separately.</p> <p>Here is the patch: <a href="http://review.whamcloud.com/#/c/13493/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/13493/</a></p> <p>Many sanity-lfsck test_4 failed on Maloo recently:<br/> <a href="https://testing.hpdd.intel.com/sub_tests/query?utf8=" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/query?utf8=</a>✓&test_set%5Btest_set_script_id%5D=4f25830c-64fe-11e2-bfb2-52540035b04c&sub_test%5Bsub_test_script_id%5D=50859f34-64fe-11e2-bfb2-52540035b04c&sub_test%5Bstatus%5D=FAIL&sub_test%5Bquery_bugs%5D=&test_session%5Btest_host%5D=&test_session%5Btest_group%5D=&test_session%5Buser_id%5D=&test_session%5Bquery_date%5D=&test_session%5Bquery_recent_period%5D=&test_node%5Bos_type_id%5D=&test_node%5Bdistribution_type_id%5D=&test_node%5Barchitecture_type_id%5D=&test_node%5Bfile_system_type_id%5D=&test_node%5Blustre_branch_id%5D=&test_node_network%5Bnetwork_type_id%5D=&commit=Update+results</p> <p>So we need to land this patch ASAP to avoid blocking others.</p> <p>This is blocking patch review testing on master branch:<br/> <a href="https://testing.hpdd.intel.com/test_sets/8e856a20-aa3a-11e4-9ffb-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/8e856a20-aa3a-11e4-9ffb-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/1c657568-aaa1-11e4-8eda-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/1c657568-aaa1-11e4-8eda-5254006e85c2</a></p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/13493/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13493/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6147" title="sanity-lfsck test_4: '(7) unexpected status' " class="issue-link" data-issue-key="LU-6147"><del>LU-6147</del></a> lfsck: NOT purge object by OI scrub<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: d11360f4cc5d38cd748a97ca05e10121353ae616</p> <p>Landed for 2.7</p> <p>Fan Yong (fan.yong@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/13840" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13840</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6147" title="sanity-lfsck test_4: '(7) unexpected status' " class="issue-link" data-issue-key="LU-6147"><del>LU-6147</del></a> lfsck: NOT purge object by OI scrub<br/> Project: fs/lustre-release<br/> Branch: b2_5<br/> Current Patch Set: 1<br/> Commit: ef9a3b471bfda6587f1da282a073f5b9a5226cfb</p> <p>+1 at <a href="https://testing.hpdd.intel.com/test_sets/6e9169d6-a179-11e4-9db6-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/6e9169d6-a179-11e4-9db6-5254006e85c2</a></p> Related Development Rank 1|hzx4lr: Rank (Obsolete) 17167 Severity