[LU-14974] Improve the repartition of requests between copytools Created: 30/Aug/21  Updated: 30/Aug/21

Status: Open
Project: Lustre
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Minor
Reporter: Guillaume Courrier Assignee: Guillaume Courrier
Resolution: Unresolved Votes: 0
Labels: hsm

Project: HSM
Rank (Obsolete): 9223372036854775807

 Description   

Currently, when reading new requests, the HSM coordinator builds *hsm_action_list*s ** by first trying to find a suitable list in the already constructed lists (i.e. with the same archive_id and enough space meaning less than LDLM_MAXREQSIZE). Then the request is added in the list if possible or a new list is allocated for it.

The issue with this approach is that if the number of requests is not large enough to fill an entire list, all the requests will be sent to one copytool only, leaving the other idles. If these few requests are concerning "big" files, this will create a huge load on one copytool and waste a lot of time.

The idea would be to build more *hsm_action_list*s based on the number of copytools per archive_id.

If we have, say, 3 copytools for the archive_id 1, we can build 3 lists and distribute the requests of archive_id 1 evenly in them. These lists will then be more likely to be sent to the 3 copytools depending on their current load. Once the 3 lists are full, we can allocate 3 new ones and continue the process.



 Comments   
Comment by Gerrit Updater [ 30/Aug/21 ]

"Guillaume Courrier <guillaume.courrier@cea.fr>" uploaded a new patch: https://review.whamcloud.com/44782
Subject: LU-14974: improve the repartition of requests between copytools
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 0b1f7d281831fab7e679b21dfc1ccc79af916648

Comment by Gerrit Updater [ 30/Aug/21 ]

"Guillaume Courrier <guillaume.courrier@cea.fr>" uploaded a new patch: https://review.whamcloud.com/44783
Subject: LU-14974: improve the repartition of requests between copytools
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: e3de948d66609856c2516fabc1dbfddc2388dcc5

Generated at Sat Feb 10 03:14:21 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.