[LU-14974] Improve the repartition of requests between copytools

• Type: Improvement
• Resolution: Unresolved
• Priority: Minor
• Labels: hsm
• Project: HSM
• Assignee: Guillaume Courrier
• Reporter: Guillaume Courrier

Description

Currently, when reading new requests, the HSM coordinator builds *hsm_action_list*s ** by first trying to find a suitable list in the already constructed lists (i.e. with the same archive_id and enough space meaning less than LDLM_MAXREQSIZE). Then the request is added in the list if possible or a new list is allocated for it.

The issue with this approach is that if the number of requests is not large enough to fill an entire list, all the requests will be sent to one copytool only, leaving the other idles. If these few requests are concerning "big" files, this will create a huge load on one copytool and waste a lot of time.

The idea would be to build more *hsm_action_list*s based on the number of copytools per archive_id.

If we have, say, 3 copytools for the archive_id 1, we can build 3 lists and distribute the requests of archive_id 1 evenly in them. These lists will then be more likely to be sent to the 3 copytools depending on their current load. Once the 3 lists are full, we can allocate 3 new ones and continue the process.

Comments

[Gerrit Updater]

"Guillaume Courrier <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/44783
Subject: LU-14974: improve the repartition of requests between copytools
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: e3de948d66609856c2516fabc1dbfddc2388dcc5

[Gerrit Updater]

"Guillaume Courrier <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/44782
Subject: LU-14974: improve the repartition of requests between copytools
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 0b1f7d281831fab7e679b21dfc1ccc79af916648