After a maintenance window we observe the following on a test FS (feature testing, tools integrations) at the CEA:
# lfs df -i /mnt/testfs UUID Inodes IUsed IFree IUse% Mounted on testfs-MDT0000_UUID 4194304 299 4194005 1% /ccc/testfs[MDT:0] testfs-MDT0001_UUID 4194304 275 4194029 1% /ccc/testfs[MDT:1] testfs-MDT0002_UUID 4194304 272 4194032 1% /ccc/testfs[MDT:2] testfs-MDT0003_UUID 4194304 277 4194027 1% /ccc/testfs[MDT:3] testfs-OST0000_UUID 768000 366999 401001 48% /ccc/testfs[OST:0] testfs-OST0001_UUID 768000 696622 71378 91% /ccc/testfs[OST:1] ...
No inodes are used on the MDT side, but the OSTs are still heavily loaded with allocated objects, leaving OST0001 almost full.
After some digging, the orphans seems to be preallocated objetcs that can't be cleaned:
Oct 18 03:22:49 oss1 kernel: Lustre: testfs-OST0001: too large difference between MDS LAST_ID [0xxx0000401:0xxx776:0x0] (391030) and OST LAST_ID [0xxx0000401:0xxx9f7:0x0] (412151), trust the OST Oct 18 03:23:14 oss1 kernel: Lustre: testfs-OST0001: too large difference between MDS LAST_ID [0xxx0000401:0xxx776:0x0] (391030) and OST LAST_ID [0xxx0000401:0xxxa18:0x0] (412184), trust the OST Oct 18 03:23:20 oss1 kernel: Lustre: testfs-OST0001: too large difference between MDS LAST_ID [0xxx0000401:0xxx776:0x0] (391030) and OST LAST_ID [0xxx0000401:0xxxa39:0x0] (412217), trust the OST Oct 18 03:23:51 oss1 kernel: Lustre: testfs-OST0001: too large difference between MDS LAST_ID [0xxx0000401:0xxx776:0x0] (391030) and OST LAST_ID [0xxx0000401:0xxxa5a:0x0] (412250), trust the OST Oct 18 03:29:19 oss1 kernel: Lustre: testfs-OST0001: too large difference between MDS LAST_ID [0xxx0000401:0xxx776:0x0] (391030) and OST LAST_ID [0xxx0000401:0xxxa5a:0x0] (412250), trust the OST Oct 18 03:29:56 oss1 kernel: Lustre: testfs-OST0001: too large difference between MDS LAST_ID [0xxx0000401:0xxx776:0x0] (391030) and OST LAST_ID [0xxx0000401:0xxxa7b:0x0] (412283), trust the OST Oct 18 03:30:27 oss1 kernel: Lustre: testfs-OST0001: too large difference between MDS LAST_ID [0xxx0000401:0xxx776:0x0] (391030) and OST LAST_ID [0xxx0000401:0xxxa7b:0x0] (412283), trust the OST Oct 18 03:30:51 oss1 kernel: Lustre: testfs-OST0001: too large difference between MDS LAST_ID [0xxx0000401:0xxx776:0x0] (391030) and OST LAST_ID [0xxx0000401:0xxxa9c:0x0] (412316), trust the OST Oct 18 03:32:28 oss1 kernel: Lustre: testfs-OST0001: too large difference between MDS LAST_ID [0xxx0000401:0xxx776:0x0] (391030) and OST LAST_ID [0xxx0000401:0xxxabd:0x0] (412349), trust the OST Oct 18 03:33:35 oss1 kernel: Lustre: testfs-OST0001: too large difference between MDS LAST_ID [0xxx0000401:0xxx776:0x0] (391030) and OST LAST_ID [0xxx0000401:0xxxade:0x0] (412382), trust the OST Oct 18 03:35:00 oss1 kernel: Lustre: testfs-OST0001: too large difference between MDS LAST_ID [0xxx0000401:0xxx776:0x0] (391030) and OST LAST_ID [0xxx0000401:0xxxaff:0x0] (412415), trust the OST Oct 18 03:36:01 oss1 kernel: Lustre: testfs-OST0001: too large difference between MDS LAST_ID [0xxx0000401:0xxx776:0x0] (391030) and OST LAST_ID [0xxx0000401:0xxxb20:0x0] (412448), trust the OST Oct 18 03:38:09 oss1 kernel: Lustre: testfs-OST0001: too large difference between MDS LAST_ID [0xxx0000401:0xxx776:0x0] (391030) and OST LAST_ID [0xxx0000401:0xxxb41:0x0] (412481), trust the OST Oct 18 03:40:48 oss1 kernel: Lustre: testfs-OST0001: too large difference between MDS LAST_ID [0xxx0000401:0xxx776:0x0] (391030) and OST LAST_ID [0xxx0000401:0xxxb62:0x0] (412514), trust the OST Oct 18 03:41:06 oss1 kernel: Lustre: testfs-OST0001: too large difference between MDS LAST_ID [0xxx0000401:0xxx776:0x0] (391030) and OST LAST_ID [0xxx0000401:0xxxb62:0x0] (412514), trust the OST Oct 18 03:41:24 oss1 kernel: Lustre: testfs-OST0001: too large difference between MDS LAST_ID [0xxx0000401:0xxx776:0x0] (391030) and OST LAST_ID [0xxx0000401:0xxxb62:0x0] (412514), trust the OST
The network was unstable, maybe the re-connections between MDT-OST explain this.
We try to clean the orphan objects with LFSCK but those are ignored:
# lctl lfsck_start -r -o -t layout -A # lctl get_param obdfilter.testfs-OST000[01].lfsck_layout obdfilter.testfs-OST0000.lfsck_layout= name: lfsck_layout magic: 0xb1732fed version: 2 status: completed flags: param: all_targets,orphan last_completed_time: 1792569531 time_since_last_completed: 14002 seconds latest_start_time: 1792569522 time_since_latest_start: 14011 seconds last_checkpoint_time: 1792569531 time_since_last_checkpoint: 14002 seconds latest_start_position: 13 last_checkpoint_position: 767996 first_failure_position: 0 success_count: 7 repaired_dangling: 0 repaired_unmatched_pair: 0 repaired_multiple_referenced: 0 repaired_orphan: 0 repaired_inconsistent_owner: 0 repaired_others: 0 skipped: 0 failed_phase1: 0 failed_phase2: 0 checked_phase1: 366633 checked_phase2: 0 run_time_phase1: 3 seconds run_time_phase2: 5 seconds average_speed_phase1: 122211 items/sec average_speed_phase2: 0 objs/sec real_time_speed_phase1: N/A real_time_speed_phase2: N/A current_position: N/A obdfilter.testfs-OST0001.lfsck_layout= name: lfsck_layout magic: 0xb1732fed version: 2 status: completed flags: param: all_targets,orphan last_completed_time: 1792569531 time_since_last_completed: 14002 seconds latest_start_time: 1792569522 time_since_latest_start: 14011 seconds last_checkpoint_time: 1792569531 time_since_last_checkpoint: 14002 seconds latest_start_position: 13 last_checkpoint_position: 696622 first_failure_position: 0 success_count: 7 repaired_dangling: 0 repaired_unmatched_pair: 0 repaired_multiple_referenced: 0 repaired_orphan: 0 repaired_inconsistent_owner: 0 repaired_others: 0 skipped: 0 failed_phase1: 0 failed_phase2: 0 checked_phase1: 696289 checked_phase2: 0 run_time_phase1: 7 seconds run_time_phase2: 2 seconds average_speed_phase1: 99469 items/sec average_speed_phase2: 0 objs/sec real_time_speed_phase1: N/A real_time_speed_phase2: N/A current_position: N/A
On the cllient:
# lfs df -i /mnt/testfs UUID Inodes IUsed IFree IUse% Mounted on testfs-MDT0000_UUID 4194304 299 4194005 1% /ccc/testfs[MDT:0] testfs-MDT0001_UUID 4194304 275 4194029 1% /ccc/testfs[MDT:1] testfs-MDT0002_UUID 4194304 272 4194032 1% /ccc/testfs[MDT:2] testfs-MDT0003_UUID 4194304 277 4194027 1% /ccc/testfs[MDT:3] testfs-OST0000_UUID 768000 366999 401001 48% /ccc/testfs[OST:0] testfs-OST0001_UUID 768000 696622 71378 91% /ccc/testfs[OST:1] # find /ccc/testfs/.lustre/lost+found/MDT* /mnt/testfs/.lustre/lost+found/MDT0000 /mnt/testfs/.lustre/lost+found/MDT0001 /mnt/testfs/.lustre/lost+found/MDT0002 /mnt/testfs/.lustre/lost+found/MDT0003
The preallocated objects seems to be intentionally ignored by the following:
static int lfsck_orphan_it_next(const struct lu_env *env, struct dt_it *di) { ... rc = dt_xattr_get(env, obj, lfsck_buf_get(env, ff, sizeof(*ff)), XATTR_NAME_FID); if (rc == -ENODATA) { /* For the pre-created OST-object, update the bitmap to avoid * others LFSCK (second phase) iteration to touch it again. */ if (la->la_ctime == 0) { if (!test_and_set_bit(pos, lrn->lrn_accessed_bitmap)) atomic_inc(&lrn->lrn_accessed_count); /* For the race between repairing dangling referenced * MDT-object and unlink the file, it may left orphan * OST-object there. Destroy it now! */ if (unlikely(!(la->la_mode & S_ISUID))) { dt_read_unlock(env, obj); lfsck_layout_destroy_orphan(env, lfsck, obj); lfsck_object_put(env, obj); pos++; goto again1; } } else if (idx == 0) { /* If the orphan OST-object has no parent information, * regard it as referenced by the MDT-object on MDT0. */ fid_zero(&rec->lor_rec.lor_fid); rec->lor_rec.lor_uid = la->la_uid; rec->lor_rec.lor_gid = la->la_gid; memset(ol, 0, sizeof(*ol)); rec->lor_layout_version = 0; rec->lor_range = 0; GOTO(out, rc = 0); } dt_read_unlock(env, obj); lfsck_object_put(env, obj); pos++; goto again1;
Preallocated objects are created with the following attributes:
int ofd_precreate_objects(const struct lu_env *env, struct ofd_device *ofd, u64 id, struct ofd_seq *oseq, int nr, int sync) ... info->fti_attr.la_valid = LA_TYPE | LA_MODE; info->fti_attr.la_mode = S_IFREG | S_ISUID | S_ISGID | S_ISVTX | 0666; info->fti_dof.dof_type = dt_mode_to_dft(S_IFREG);