ai generated triage report

Symptom

The MDS LBUGs during conf-sanity test_122a "Check OST sequence update" on a
DNE ldiskfs config:

LustreError: 29639:0:(osp_dev.c:843:osp_statfs()) ASSERTION( sfs->os_fprecreated <= OST_MAX_PRECREATE * 2 ) failed: last_created [0x280000401:0xff02:0x0], next_fid [0x280000401:0x1:0x0], reserved 0
LustreError: 29639:0:(osp_dev.c:843:osp_statfs()) LBUG
Kernel panic - not syncing: LBUG #3 lbug_with_loc                [libcfs]
 #4 osp_statfs                   [osp]    lustre/osp/osp_dev.c:840
 #5 lod_statfs_and_check         [lod]
 #6 lod_qos_statfs_update        [lod]    lustre/lod/lod_qos.c:175
 #7 lod_qos_prep_create          [lod]    lustre/lod/lod_qos.c:2950
 #9 lod_declare_striped_create   [lod]
#13 mdd_declare_create_object_internal [mdd]
#16 mdd_create                   [mdd]
#17 mdt_reint_open               [mdt]

Seen on the Lustre Gerrit Janitor across unrelated patches, e.g.
https://testing.whamcloud.com/gerrit-janitor/66699/testresults/conf-sanity4-ldiskfs-DNE-rocky8.10_x86_64-rocky8.10_x86_64

Relatively rare in autotest, but 100% reproducible once the right OSP is used – see "Why it is intermittent" below.

Root cause

ofd_create_hdl() answers a normal (non-DELORPHAN) precreate request whose
last OID is below the OST's LAST_ID with the OST's own last_oid + 1

("trust the OST") and precreates nothing (lustre/ofd/ofd_dev.c:1687):

             if (diff < 0) {
                LCONSOLE(D_INFO,
                     "%s: MDS LAST_ID "DFID" (%llu) is %lld behind OST LAST_ID "DFID" (%llu), trust the OST\n", ...);
                /* Let MDS know that we are so far ahead. */
                rc = ostid_set_id(&rep_oa->o_oi,
                          ofd_seq_last_oid(oseq) + 1);
            }

osp_precreate_send() stores that reply into opd_pre_last_created_fid but never touches opd_pre_used_fid (lustre/osp/osp_precreate.c:699), so the OSP then advertises every object in between as precreated, although the OST created none of them for this request. From the failing session's debug log:

lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 65248 behind OST LAST_ID [0x280000401:0xff01:0x0] (65281), trust the OST
lustre-OST0000-osc-MDT0001: current precreated pool: [0x280000401:0x1:0x0]-[0x280000401:0xff02:0x0]

os_fprecreatedis then 0xff02 - 0x1 = 65281, above OST_MAX_PRECREATE * 2 (40000), and the next {{lod_statfs_and_check()}} during object creation hits the assertion.

When it was introduced

• 1711e26ae861 ("LU-15009 ofd: continue precreate if LAST_ID is less on MDT", landed 2.14.57/2.15.0) introduced that OST reply, but capped the jump at OST_MAX_PRECREATE – so the precreate window was already wrong, but os_fprecreated stayed below the assertion.
• 53940ff3eee8 ("LU-17987 ofd: always trust OST last_id", landed 2.15.90/2.16.0) removed that cap, so an arbitrarily large jump now reaches the assertion and the MDS LBUGs.

Why it is intermittent

test_122a sets OBD_FAIL_OFD_SET_OID (0x1e0) on ost1, which makes the OST
stamp a newly created sequence's LAST_ID at seq_width - 255. Any OSP that rolls over to a fresh sequence while that fail_loc is still set gets the "trust the OST" reply and a poisoned precreate window.

• MDT0000's OSP starts on the IDIF sequence and therefore always rolls over as soon as it connects, i.e. always while the fail_loc is set. But stock test_122a only creates files under a directory on MDT0001, so that poisoned OSP is normally never used and the test passes.
• The crash needs MDT0001's OSP to be poisoned as well. That requires an extra race: mount_facet() applies seq.cli-<OST>-super.width after the OST mount returns, so an OSP that creates its first sequence's LAST_ID object before that gets it stamped with the default
  LUSTRE_DATA_SEQ_MAX_WIDTH - 255 (0x1ffff00). The OSP later learns the real width (65536) from a reply, declares the sequence exhausted and rolls over - while the fail_loc is still set.

In the failing session: {{Got last_fid [0x280000400:0x1ffff00:0x0]}} for MDT0001 versus the healthy [...:0xff01].

Reproducer

Making it deterministic only takes using the OSP that is always poisoned - add to test_122a, after the existing createmany :

    $LFS mkdir -i0 -c1 $DIR/${tdir}0 || error "mkdir on MDT0000 failed"
    $LFS setstripe -i0 -c1 $DIR/${tdir}0
    createmany -o $DIR/${tdir}0/file_ 100 ||
        error "Fail to create on MDT0000"

With that, MDSCOUNT=2 OSTCOUNT=2 ldiskfs LBUGs the MDS on every run.