Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-21004

lfsck does not handle orphan preallocated objects

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Medium Medium
    • None
    • Lustre 2.15.8
    • lustre 2.15.8.ocean5 server+client.
      Test filesystem hosted at the CEA.
    • 3
    • 9223372036854775807

      After a maintenance window we observe the following on a test FS (feature testing, tools integrations) at the CEA:

      # lfs df -i /mnt/testfs
      UUID                      Inodes       IUsed       IFree IUse% Mounted on
      testfs-MDT0000_UUID      4194304         299     4194005   1% /ccc/testfs[MDT:0]
      testfs-MDT0001_UUID      4194304         275     4194029   1% /ccc/testfs[MDT:1]
      testfs-MDT0002_UUID      4194304         272     4194032   1% /ccc/testfs[MDT:2]
      testfs-MDT0003_UUID      4194304         277     4194027   1% /ccc/testfs[MDT:3]
      testfs-OST0000_UUID       768000      366999      401001  48% /ccc/testfs[OST:0]
      testfs-OST0001_UUID       768000      696622       71378  91% /ccc/testfs[OST:1]
      ...
      

      No inodes are used on the MDT side, but the OSTs are still heavily loaded with allocated objects, leaving OST0001 almost full.

      After some digging, the orphans seems to be preallocated objetcs that can't be cleaned:

      Oct 18 03:22:49 oss1 kernel: Lustre: testfs-OST0001: too large difference between MDS LAST_ID [0xxx0000401:0xxx776:0x0] (391030) and OST LAST_ID [0xxx0000401:0xxx9f7:0x0] (412151), trust the OST
      Oct 18 03:23:14 oss1 kernel: Lustre: testfs-OST0001: too large difference between MDS LAST_ID [0xxx0000401:0xxx776:0x0] (391030) and OST LAST_ID [0xxx0000401:0xxxa18:0x0] (412184), trust the OST
      Oct 18 03:23:20 oss1 kernel: Lustre: testfs-OST0001: too large difference between MDS LAST_ID [0xxx0000401:0xxx776:0x0] (391030) and OST LAST_ID [0xxx0000401:0xxxa39:0x0] (412217), trust the OST
      Oct 18 03:23:51 oss1 kernel: Lustre: testfs-OST0001: too large difference between MDS LAST_ID [0xxx0000401:0xxx776:0x0] (391030) and OST LAST_ID [0xxx0000401:0xxxa5a:0x0] (412250), trust the OST
      Oct 18 03:29:19 oss1 kernel: Lustre: testfs-OST0001: too large difference between MDS LAST_ID [0xxx0000401:0xxx776:0x0] (391030) and OST LAST_ID [0xxx0000401:0xxxa5a:0x0] (412250), trust the OST
      Oct 18 03:29:56 oss1 kernel: Lustre: testfs-OST0001: too large difference between MDS LAST_ID [0xxx0000401:0xxx776:0x0] (391030) and OST LAST_ID [0xxx0000401:0xxxa7b:0x0] (412283), trust the OST
      Oct 18 03:30:27 oss1 kernel: Lustre: testfs-OST0001: too large difference between MDS LAST_ID [0xxx0000401:0xxx776:0x0] (391030) and OST LAST_ID [0xxx0000401:0xxxa7b:0x0] (412283), trust the OST
      Oct 18 03:30:51 oss1 kernel: Lustre: testfs-OST0001: too large difference between MDS LAST_ID [0xxx0000401:0xxx776:0x0] (391030) and OST LAST_ID [0xxx0000401:0xxxa9c:0x0] (412316), trust the OST
      Oct 18 03:32:28 oss1 kernel: Lustre: testfs-OST0001: too large difference between MDS LAST_ID [0xxx0000401:0xxx776:0x0] (391030) and OST LAST_ID [0xxx0000401:0xxxabd:0x0] (412349), trust the OST
      Oct 18 03:33:35 oss1 kernel: Lustre: testfs-OST0001: too large difference between MDS LAST_ID [0xxx0000401:0xxx776:0x0] (391030) and OST LAST_ID [0xxx0000401:0xxxade:0x0] (412382), trust the OST
      Oct 18 03:35:00 oss1 kernel: Lustre: testfs-OST0001: too large difference between MDS LAST_ID [0xxx0000401:0xxx776:0x0] (391030) and OST LAST_ID [0xxx0000401:0xxxaff:0x0] (412415), trust the OST
      Oct 18 03:36:01 oss1 kernel: Lustre: testfs-OST0001: too large difference between MDS LAST_ID [0xxx0000401:0xxx776:0x0] (391030) and OST LAST_ID [0xxx0000401:0xxxb20:0x0] (412448), trust the OST
      Oct 18 03:38:09 oss1 kernel: Lustre: testfs-OST0001: too large difference between MDS LAST_ID [0xxx0000401:0xxx776:0x0] (391030) and OST LAST_ID [0xxx0000401:0xxxb41:0x0] (412481), trust the OST
      Oct 18 03:40:48 oss1 kernel: Lustre: testfs-OST0001: too large difference between MDS LAST_ID [0xxx0000401:0xxx776:0x0] (391030) and OST LAST_ID [0xxx0000401:0xxxb62:0x0] (412514), trust the OST
      Oct 18 03:41:06 oss1 kernel: Lustre: testfs-OST0001: too large difference between MDS LAST_ID [0xxx0000401:0xxx776:0x0] (391030) and OST LAST_ID [0xxx0000401:0xxxb62:0x0] (412514), trust the OST
      Oct 18 03:41:24 oss1 kernel: Lustre: testfs-OST0001: too large difference between MDS LAST_ID [0xxx0000401:0xxx776:0x0] (391030) and OST LAST_ID [0xxx0000401:0xxxb62:0x0] (412514), trust the OST
      

      The network was unstable, maybe the re-connections between MDT-OST explain this.

      We try to clean the orphan objects with LFSCK but those are ignored:

      # lctl lfsck_start -r -o -t layout -A
      # lctl get_param obdfilter.testfs-OST000[01].lfsck_layout
      obdfilter.testfs-OST0000.lfsck_layout=     
      name: lfsck_layout                         
      magic: 0xb1732fed                          
      version: 2                                 
      status: completed                          
      flags:                                     
      param: all_targets,orphan                  
      last_completed_time: 1792569531            
      time_since_last_completed: 14002 seconds   
      latest_start_time: 1792569522              
      time_since_latest_start: 14011 seconds     
      last_checkpoint_time: 1792569531           
      time_since_last_checkpoint: 14002 seconds  
      latest_start_position: 13                  
      last_checkpoint_position: 767996           
      first_failure_position: 0                  
      success_count: 7                           
      repaired_dangling: 0                       
      repaired_unmatched_pair: 0                 
      repaired_multiple_referenced: 0            
      repaired_orphan: 0                         
      repaired_inconsistent_owner: 0             
      repaired_others: 0                         
      skipped: 0                                 
      failed_phase1: 0                           
      failed_phase2: 0                           
      checked_phase1: 366633                     
      checked_phase2: 0                          
      run_time_phase1: 3 seconds                 
      run_time_phase2: 5 seconds                 
      average_speed_phase1: 122211 items/sec     
      average_speed_phase2: 0 objs/sec           
      real_time_speed_phase1: N/A                
      real_time_speed_phase2: N/A                
      current_position: N/A                      
      obdfilter.testfs-OST0001.lfsck_layout=     
      name: lfsck_layout                         
      magic: 0xb1732fed                          
      version: 2                                 
      status: completed                          
      flags:                                     
      param: all_targets,orphan                  
      last_completed_time: 1792569531            
      time_since_last_completed: 14002 seconds   
      latest_start_time: 1792569522              
      time_since_latest_start: 14011 seconds     
      last_checkpoint_time: 1792569531           
      time_since_last_checkpoint: 14002 seconds  
      latest_start_position: 13                  
      last_checkpoint_position: 696622           
      first_failure_position: 0                  
      success_count: 7                           
      repaired_dangling: 0                       
      repaired_unmatched_pair: 0                 
      repaired_multiple_referenced: 0            
      repaired_orphan: 0                         
      repaired_inconsistent_owner: 0             
      repaired_others: 0                         
      skipped: 0                                 
      failed_phase1: 0                           
      failed_phase2: 0                           
      checked_phase1: 696289                     
      checked_phase2: 0                          
      run_time_phase1: 7 seconds                 
      run_time_phase2: 2 seconds                 
      average_speed_phase1: 99469 items/sec      
      average_speed_phase2: 0 objs/sec           
      real_time_speed_phase1: N/A                
      real_time_speed_phase2: N/A                
      current_position: N/A                      
      

      On the cllient:

      # lfs df -i /mnt/testfs
      UUID                      Inodes       IUsed       IFree IUse% Mounted on
      testfs-MDT0000_UUID      4194304         299     4194005   1% /ccc/testfs[MDT:0]
      testfs-MDT0001_UUID      4194304         275     4194029   1% /ccc/testfs[MDT:1]
      testfs-MDT0002_UUID      4194304         272     4194032   1% /ccc/testfs[MDT:2]
      testfs-MDT0003_UUID      4194304         277     4194027   1% /ccc/testfs[MDT:3]
      testfs-OST0000_UUID       768000      366999      401001  48% /ccc/testfs[OST:0]
      testfs-OST0001_UUID       768000      696622       71378  91% /ccc/testfs[OST:1]
      # find /ccc/testfs/.lustre/lost+found/MDT*
      /mnt/testfs/.lustre/lost+found/MDT0000
      /mnt/testfs/.lustre/lost+found/MDT0001
      /mnt/testfs/.lustre/lost+found/MDT0002
      /mnt/testfs/.lustre/lost+found/MDT0003
      

      The preallocated objects seems to be intentionally ignored by the following:

      static int lfsck_orphan_it_next(const struct lu_env *env,
                                      struct dt_it *di)
      {
      ...
              rc = dt_xattr_get(env, obj, lfsck_buf_get(env, ff, sizeof(*ff)),
                                XATTR_NAME_FID);
              if (rc == -ENODATA) {
                      /* For the pre-created OST-object, update the bitmap to avoid
                       * others LFSCK (second phase) iteration to touch it again. */
                      if (la->la_ctime == 0) {
                              if (!test_and_set_bit(pos, lrn->lrn_accessed_bitmap))
                                      atomic_inc(&lrn->lrn_accessed_count);
      
                              /* For the race between repairing dangling referenced
                               * MDT-object and unlink the file, it may left orphan
                               * OST-object there. Destroy it now! */
                              if (unlikely(!(la->la_mode & S_ISUID))) {
                                      dt_read_unlock(env, obj);
                                      lfsck_layout_destroy_orphan(env, lfsck, obj);
                                      lfsck_object_put(env, obj);
                                      pos++;
                                      goto again1;
                              }
                      } else if (idx == 0) {
                              /* If the orphan OST-object has no parent information,
                               * regard it as referenced by the MDT-object on MDT0. */
                              fid_zero(&rec->lor_rec.lor_fid);
                              rec->lor_rec.lor_uid = la->la_uid;
                              rec->lor_rec.lor_gid = la->la_gid;
                              memset(ol, 0, sizeof(*ol));
                              rec->lor_layout_version = 0;
                              rec->lor_range = 0;
      
                              GOTO(out, rc = 0);
                      }
      
                      dt_read_unlock(env, obj);
                      lfsck_object_put(env, obj);
                      pos++;
                      goto again1;
      

      Preallocated objects are created with the following attributes:

      int ofd_precreate_objects(const struct lu_env *env, struct ofd_device *ofd,
                                u64 id, struct ofd_seq *oseq, int nr, int sync)
      ...
              info->fti_attr.la_valid = LA_TYPE | LA_MODE;
              info->fti_attr.la_mode = S_IFREG | S_ISUID | S_ISGID | S_ISVTX | 0666;
              info->fti_dof.dof_type = dt_mode_to_dft(S_IFREG);
      

            dongyang Dongyang Li
            eaujames Etienne Aujames
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated: