Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-21281

ldlm: cancelling many extent locks on one object is quadratic in ldlm_extent_shift_kms()

XMLWordPrintable

    • Icon: Improvement Improvement
    • Resolution: Unresolved
    • Icon: Medium Medium
    • None
    • None
    • None
    • 3
    • 9223372036854775807

      Description
      When a client holds many PR extent locks on one object, cancelling them is       
      quadratic in the number of locks. ldlm_extent_shift_kms() runs for every         
      cancel, under lock_res_and_lock(), and walks every lock in the extent tree       
      whenever no remaining lock protects the old KMS.                                 
                                                                                       
      With 20000 adjacent one-page PR locks, cancelling them all takes about 5.6 s     
      on average on test clients, and 0.19 s with the patch below (30x faster, see     
      "Measurements"). The cost grows with the square of the lock count, so it         
      gets much worse at the lock counts seen on shared files.                         
                                                                                       
      The same problem was seen together with client evictions during shared-file      
      IOR with unaligned records ("Observed in the field", below). The link to         
      the evictions is likely, but it is not proven. This ticket is about the          
      cancel cost, which is measured.

      Analysis                                                                                                                                              
      __osc_dlm_blocking_ast() calls ldlm_extent_shift_kms() for every cancel, under   
      lock_res_and_lock(). When no remaining lock protects the old KMS, the            
      function walks every lock in the extent tree. That is O(N) per cancel and        
      O(N^2) for a burst of N cancels. Other threads that need the resource lock       
      spin behind it.                                                                  
                                                                                       
      The client cancels its unused locks from the newest to the oldest. Locks taken   
      from low to high offsets are therefore cancelled highest first, which is the     
      worst case.                                                                      
                                                                                       
      Related tickets: LU-12832 and LU-12377 show the same symptoms. The               
      "end + 1 >= old_kms" change discussed in LU-12832 is already in the tree         
      (LU-13312), and the stacks below still show the scan. It is not LU-19721:        
      the client build has no invalidate_lock code in llite or osc.   

      Measurements                                                                     
      Seven clients sharing one striped file. Each client takes 20000 adjacent         
      one-page PR locks on the same object with "lfs ladvise -a lockahead", lowest     
      offset first, so the locks are cancelled highest first. The time is from         
      "lru_size=clear" to lock_count reaching 0 on that client. One run per build.     

                                                                             
      cancel 20000 locks    unpatched    patched    ratio                              
        sles160               5574 ms     130 ms     43x                               
        sles157               6518 ms     126 ms     52x                               
        rhel98                5245 ms     158 ms     33x                               
        rhel102               6435 ms     233 ms     28x                               
        debian137             5146 ms     214 ms     24x                               
        ubuntu2404            4943 ms     220 ms     22x                               
        ubuntu2604            5359 ms     219 ms     24x                               
        mean                  5603 ms     186 ms     30x                               
      

                                                                            
                                                                                       
      The same case on a 4 vCPU rocky9 VM (sanity 255e, new), one run each:            

                                                                             
        unpatched: lowest first 2083 ms, highest first 6630 ms  -> FAIL                
        patched:   lowest first 2137 ms, highest first 2137 ms  -> PASS                
      

                                                                            
                                                                                       
      kunit ldlm_extent test 3, cost per iteration (one lock added, the oldest         
      cancelled, and one ldlm_extent_shift_kms() call), on the same VM:                

                                                                             
        locks      unpatched   patched                                                 
        10         9.1 us      9.1 us                                                  
        1,000      28.1 us     9.3 us                                                  
        10,000     220.5 us    9.4 us                                                  
        ~27,000    ~346 us     n/a                                                     
        100,000    not reached 9.2 us                                                  
      

             

      Workaround (from LU-12832)                                 

                                                                             
      lctl set_param ldlm.namespaces.*osc*.lru_max_age=30000                           
      

                                                                            
      This shortens how long idle locks accumulate, at the cost of more lock RPCs      
      and more re-reads. It does not remove the cost of cancelling many live           
      locks at once.                                                          

            manhongdai Manhong Dai
            manhongdai Manhong Dai
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: