[LU-11239] sanity-lfsck test 36a fails with 'Fail to split mirror'

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.13.0, Lustre 2.12.4
• Affects Version/s: Lustre 2.12.0, Lustre 2.13.0, Lustre 2.12.1, Lustre 2.12.2, Lustre 2.12.3
• Severity: 3
• Assignee: Zhenyu Xu
• Reporter: James Nunez (Inactive)

Description

sanity-lfsck test_36a started failing on August 6, 2018 with Lustre version 2.11.53.52 build #3774. Note, sanity-lfsck test 36a landed to master with build #3774.

In the test_log for these failures, for example at https://testing.whamcloud.com/test_sets/0e30317a-9ad2-11e8-a9f7-52540065bddc, we see the following:

…
/mnt/lustre/d36a.sanity-lfsck/f0
  lcm_layout_gen:    10
  lcm_mirror_count:  3
  lcm_entry_count:   6
    lcme_id:             65537
    lcme_mirror_id:      1
    lcme_flags:          init
    lcme_extent.e_start: 0
    lcme_extent.e_end:   1048576
      lmm_stripe_count:  2
      lmm_stripe_size:   1048576
      lmm_pattern:       raid0
      lmm_layout_gen:    0
      lmm_stripe_offset: 0
      lmm_objects:
      - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x9b6:0x0] }
      - 1: { l_ost_idx: 1, l_fid: [0x100010000:0x768:0x0] }

    lcme_id:             65538
    lcme_mirror_id:      1
    lcme_flags:          init
    lcme_extent.e_start: 1048576
    lcme_extent.e_end:   EOF
      lmm_stripe_count:  1
      lmm_stripe_size:   1048576
      lmm_pattern:       raid0
      lmm_layout_gen:    0
      lmm_stripe_offset: 2
      lmm_objects:
      - 0: { l_ost_idx: 2, l_fid: [0x100020000:0x78e:0x0] }

    lcme_id:             131075
    lcme_mirror_id:      2
    lcme_flags:          init
    lcme_extent.e_start: 0
    lcme_extent.e_end:   2097152
      lmm_stripe_count:  2
      lmm_stripe_size:   1048576
      lmm_pattern:       raid0
      lmm_layout_gen:    0
      lmm_stripe_offset: 1
      lmm_objects:
      - 0: { l_ost_idx: 1, l_fid: [0x100010000:0x769:0x0] }
      - 1: { l_ost_idx: 2, l_fid: [0x100020000:0x788:0x0] }

    lcme_id:             131076
    lcme_mirror_id:      2
    lcme_flags:          init
    lcme_extent.e_start: 2097152
    lcme_extent.e_end:   EOF
      lmm_stripe_count:  1
      lmm_stripe_size:   1048576
      lmm_pattern:       raid0
      lmm_layout_gen:    0
      lmm_stripe_offset: 0
      lmm_objects:
      - 0: { l_ost_idx: 0, l_fid: [0x100000000:0x9bc:0x0] }

    lcme_id:             196613
    lcme_mirror_id:      3
    lcme_flags:          init,stale
    lcme_extent.e_start: 0
    lcme_extent.e_end:   3145728
      lmm_stripe_count:  2
      lmm_stripe_size:   1048576
      lmm_pattern:       raid0
      lmm_layout_gen:    0
      lmm_stripe_offset: 2
      lmm_objects:
      - 0: { l_ost_idx: 2, l_fid: [0x100020000:0x789:0x0] }
      - 1: { l_ost_idx: 0, l_fid: [0x100000000:0x9b7:0x0] }

    lcme_id:             196614
    lcme_mirror_id:      3
    lcme_flags:          init,stale
    lcme_extent.e_start: 3145728
    lcme_extent.e_end:   EOF
      lmm_stripe_count:  1
      lmm_stripe_size:   1048576
      lmm_pattern:       raid0
      lmm_layout_gen:    0
      lmm_stripe_offset: 1
      lmm_objects:
      - 0: { l_ost_idx: 1, l_fid: [0x100010000:0x76e:0x0] }
…
Inject failure, to simulate the case of missing one mirror in LOV
CMD: trevis-7vm4 /usr/sbin/lctl set_param fail_loc=0x1616
fail_loc=0x1616
error: lfs mirror split: setting 'stale' is not supported
 sanity-lfsck test_36a: @@@@@@ FAIL: (12) Fail to split 1st mirror from /mnt/lustre/d36a.sanity-lfsck/f0 

In comparing sanity-lfsck test 36a that pass testing and those that don’t, the ones that fail have components with “init,stale” flags and the tests that pass don’t have “stale flags

More logs for this failure are at
https://testing.whamcloud.com/test_sets/1131f45c-99d4-11e8-a9f7-52540065bddc
https://testing.whamcloud.com/test_sets/6481868c-9ab4-11e8-a9f7-52540065bddc

Issue Links

• is related to: LU-4664 sync write should consume grant on client
• is related to: LU-12757 sanity-lfsck test 36a fails with '(N) Fail to resync /mnt/lustre/d36a.sanity-lfsck/f2'
• is related to: LU-11111 crash doing LFSCK: orph_index_insert()) ASSERTION( !(obj->mod_flags & ORPHAN_OBJ)