https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-12068 Lustre <p>This issue was created by maloo for wangshilong &lt;wshilong@ddn.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.whamcloud.com/test_sets/36dcb5b6-45d2-11e9-9646-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/36dcb5b6-45d2-11e9-9646-52540065bddc</a></p> <p>test_6b failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>(7.2) 0x0 is not larger than 0x0 </pre> </div></div> <p>&lt;&lt;Please provide additional information about the failure here&gt;&gt;</p> <p>VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV<br/> sanity-lfsck test_6b - (7.2) 0x0 is not larger than 0x0</p> LU-12068

sanity-lfsck test_6b: (7.2) 0x0 is not larger than 0x0

Bug Major Resolved Fixed Hongchao Zhang Maloo zfs Thu, 14 Mar 2019 02:54:19 +0000 Fri, 10 May 2019 21:09:38 +0000 Mon, 8 Apr 2019 14:21:46 +0000 Lustre 2.13.0 Lustre 2.12.1 Lustre 2.13.0 Lustre 2.12.2 0 8 <p>It looks like this has been happening a lot recently, but the first recent occurrence is on 2019-02-20:<br/> <a href="https://testing.whamcloud.com/test_sets/07e0f5ee-3513-11e9-b4f9-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/07e0f5ee-3513-11e9-b4f9-52540065bddc</a></p> <p>There were a number of patches landed on 2019-02-18 that may have triggered this issue, but none of the patches landed on the 18th look like they would be the cause. The most recent patch that changed LFSCK is <a href="https://jira.whamcloud.com/browse/LU-11111" title="crash doing LFSCK: orph_index_insert()) ASSERTION( !(obj-&gt;mod_flags &amp; ORPHAN_OBJ)" class="issue-link" data-issue-key="LU-11111"><del>LU-11111</del></a> but it landed on 2019-02-11.</p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/34417" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34417</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12068" title="sanity-lfsck test_6b: (7.2) 0x0 is not larger than 0x0" class="issue-link" data-issue-key="LU-12068"><del>LU-12068</del></a> tests: add debug for sanity-lfsck test_6b<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 3a81468d0f0fff77de01e67a479c9ea82878d4c8</p> <p>Andreas Dilger (adilger@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/34417/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34417/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12068" title="sanity-lfsck test_6b: (7.2) 0x0 is not larger than 0x0" class="issue-link" data-issue-key="LU-12068"><del>LU-12068</del></a> tests: add debug for sanity-lfsck test_6b<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 698209700faf51c227f9ba16626de5ed70fa97c8</p> <p>We have at least one sanity-lfsck test 6b failure with the debug information at <a href="https://testing.whamcloud.com/test_sets/5c532a1a-4fe0-11e9-9720-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/5c532a1a-4fe0-11e9-9720-52540065bddc</a>. The debug info for this test is:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Additional debug for 6b CMD: trevis-12vm8 /usr/sbin/lctl get_param -n mdd.lustre-MDT0000.lfsck_namespace name: lfsck_namespace magic: 0xa06249ff version: 2 status: scanning-phase1 flags: param: last_completed_time: 1553561433 time_since_last_completed: 14 seconds latest_start_time: 1553561446 time_since_latest_start: 1 seconds last_checkpoint_time: 1553561444 time_since_last_checkpoint: 3 seconds latest_start_position: 1943, [0x200000405:0x1:0x0], 0x0 last_checkpoint_position: 1680, [0x200000405:0x1:0x0], 0x0 first_failure_position: N/A, N/A, N/A checked_phase1: 4 checked_phase2: 0 updated_phase1: 0 updated_phase2: 0 failed_phase1: 0 failed_phase2: 0 directories: 2 dirent_repaired: 0 linkea_repaired: 0 nlinks_repaired: 0 multiple_linked_checked: 0 multiple_linked_repaired: 0 unknown_inconsistency: 0 unmatched_pairs_repaired: 0 dangling_repaired: 0 multiple_referenced_repaired: 0 bad_file_type_repaired: 0 lost_dirent_repaired: 0 local_lost_found_scanned: 0 local_lost_found_moved: 0 local_lost_found_skipped: 0 local_lost_found_failed: 0 striped_dirs_scanned: 0 striped_dirs_repaired: 0 striped_dirs_failed: 0 striped_dirs_disabled: 0 striped_dirs_skipped: 0 striped_shards_scanned: 0 striped_shards_repaired: 0 striped_shards_failed: 0 striped_shards_skipped: 0 name_hash_repaired: 0 linkea_overflow_cleared: 0 agent_entries_repaired: 0 success_count: 11 run_time_phase1: 7 seconds run_time_phase2: 0 seconds average_speed_phase1: 0 items/sec average_speed_phase2: N/A average_speed_total: 0 items/sec real_time_speed_phase1: 0 items/sec real_time_speed_phase2: N/A current_position: 1942, [0x200000405:0x1:0x0], 0x0 sanity-lfsck test_6b: @@@@@@ FAIL: (7.2) 0x0 is not larger than 0x0 </pre> </div></div> <p>I dug in to this for a while yesterday (I found several examples with debug, James' there is very much representative), and I mostly concluded that I don't understand lfsck tests very well.  It would be good if someone who knows the lfsck architecture could dig in - I was spending most of my time trying to figure out what normal was for this.</p> <p>HongChao, could you please look into this issue. It is causing a large number of test failures. </p> <p>Hongchao Zhang (hongchao@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/34525" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34525</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12068" title="sanity-lfsck test_6b: (7.2) 0x0 is not larger than 0x0" class="issue-link" data-issue-key="LU-12068"><del>LU-12068</del></a> test: compare position for ZFS dot entry<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 617cd0fa4b53453cd88295a7c2265817ebc38f48</p> <p>The issue is managed to be reproduced locally, and the cause of it is that the LFSCK process is stopped just when it is scanning the "." or ".." entry<br/> of some directory, for ZFS, the position of both the two entries is zero.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>static __u64 osd_dir_it_store(const struct lu_env *env, const struct dt_it *di) { struct osd_zap_it *it = (struct osd_zap_it *)di; __u64 pos; ENTRY; if (it-&gt;ozi_pos &lt;= OZI_POS_DOTDOT) pos = 0; else pos = osd_zap_cursor_serialize(it-&gt;ozi_zc); RETURN(pos); } </pre> </div></div> <p>the patch is tracked at <a href="https://review.whamcloud.com/34525" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34525</a></p> <p>The patch <a href="https://review.whamcloud.com/34098" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34098</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-11330" title="replay-single test_70d: Directory not empty rmdir fails" class="issue-link" data-issue-key="LU-11330"><del>LU-11330</del></a> osd-zfs: hash for ./.. must be 0</tt>" only landed on 2019-02-27, while there were a few tests failing on 2019-02-20 to 2019-02-25, so it is close to the first date this problem was seen, but not exactly the same. However, <b>most</b> of the tests started failing after 2019-02-27 so it is possible there are a couple of different issues here, and <a href="https://jira.whamcloud.com/browse/LU-11330" title="replay-single test_70d: Directory not empty rmdir fails" class="issue-link" data-issue-key="LU-11330"><del>LU-11330</del></a> made the problem much worse. There aren't any cases where this test failed during the testing of <a href="https://jira.whamcloud.com/browse/LU-11330" title="replay-single test_70d: Directory not empty rmdir fails" class="issue-link" data-issue-key="LU-11330"><del>LU-11330</del></a>, but it is definitely <b>not</b> being hit on ldiskfs so this is the likely cause of most of these failures.</p> <p>Hi Andreas,</p> <p>Yes, there is one failed case on 2019-02-20 and another one on 2019-02-25, both failed cases are on branch "master-next", <br/> the version is "2.12.51.85", is it possible this version contains some other patches?</p> <p><a href="https://testing.whamcloud.com/test_sessions/ae657cad-8359-4ff5-a42a-9b0e496a025b" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sessions/ae657cad-8359-4ff5-a42a-9b0e496a025b</a><br/> <a href="https://testing.whamcloud.com/test_sessions/b49849ff-483d-4982-84c5-92721cae1afe" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sessions/b49849ff-483d-4982-84c5-92721cae1afe</a></p> <p>+1 on b2_12 <a href="https://testing.whamcloud.com/test_sets/eb96e502-5523-11e9-9646-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/eb96e502-5523-11e9-9646-52540065bddc</a></p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/34525/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34525/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12068" title="sanity-lfsck test_6b: (7.2) 0x0 is not larger than 0x0" class="issue-link" data-issue-key="LU-12068"><del>LU-12068</del></a> test: compare position for ZFS dot entry<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 42adbae36f206a6ed4170e7619cd993c8fa80b1d</p> <p>Landed in 2.13</p> <p>Minh Diep (mdiep@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/34695" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34695</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12068" title="sanity-lfsck test_6b: (7.2) 0x0 is not larger than 0x0" class="issue-link" data-issue-key="LU-12068"><del>LU-12068</del></a> tests: add debug for sanity-lfsck test_6b<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: 02ebeedf6dcd83d11e13e31c06753cfaef5dcbbf</p> <p>Minh Diep (mdiep@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/34696" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34696</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12068" title="sanity-lfsck test_6b: (7.2) 0x0 is not larger than 0x0" class="issue-link" data-issue-key="LU-12068"><del>LU-12068</del></a> test: compare position for ZFS dot entry<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: 08a0fa85c6f9987dbd6c3049b46c660a1846a1c3</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/34695/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34695/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12068" title="sanity-lfsck test_6b: (7.2) 0x0 is not larger than 0x0" class="issue-link" data-issue-key="LU-12068"><del>LU-12068</del></a> tests: add debug for sanity-lfsck test_6b<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: f5e0c311ae60709070514b951f7d25d537d3dc91</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/34696/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34696/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12068" title="sanity-lfsck test_6b: (7.2) 0x0 is not larger than 0x0" class="issue-link" data-issue-key="LU-12068"><del>LU-12068</del></a> test: compare position for ZFS dot entry<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: 1e6cd6b21fc37420341fdb0dcec366bb3feb350e</p> Duplicate LU-8112 Related LU-11330 Development Rank 1|i00dc7: Rank (Obsolete) 9223372036854775807 Severity