[LU-8626] limit number of items in HSM action queue

• Type: Improvement
• Resolution: Unresolved
• Priority: Major
• Labels: hsm
• Assignee: Quentin Bouget (Inactive)
• Reporter: Andreas Dilger

Description

Several presentations at RUG'16 mentioned that Lustre has poor performance when there are very large numbers of HSM actions outstanding on the coordinator.

Firstly, having a /proc file that exposes the number of entries currently in the HSM action list would allow RBH and monitoring scripts to easily monitor the number of enties.

Issue Links

• is related to: LU-7988 HSM: high lock contention for cdt_llog_lock

Comments

[Peter Jones]

Landed for 2.11

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/30336/
Subject: LU-8626 hsm: expose the number of active hsm requests per type
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 42e40555f250b83730d233dc5e22fd1f9396ccfe

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/28677/
Subject: LU-8626 hsm: count the number of started requests of each type
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 973759d1ff3bbcb217754bd9942fdf670dec2d96

[Gerrit Updater]

Quentin Bouget ([e-mail]) uploaded a new patch: https://review.whamcloud.com/30336
Subject: LU-8626 hsm: expose the number of active hsm requests per type
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 294a0e4ff77fb09ac643b5e15af224027ded4aee

[Gerrit Updater]

Quentin Bouget ([e-mail]) uploaded a new patch: https://review.whamcloud.com/28677
Subject: LU-8626 hsm: count the number of started requests of each type
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 4e84ba8aa669554b2d1b77459ebe79770aa4ad37

[Andreas Dilger]

I think we should stick with a single value per file, since this is required when moving stats into /sys/fs/lustre, so something like action_archive_started_count, action_archive_succeed_count, action_restore_started_count, action_restore_succeed_count.

[Stephane Thiell]

max_requests is the maximum number of active requests at the same time per coordinator, so it has nothing to do with the HSM action queue. By the way, the Lustre documentation for max_requests is correct.

actions is used to dump the action queue
(Note: it is also accessible through "lctl get_param mdt.lustre-MDT0000.hsm.actions")

Typical entries in actions look like:

lrh=[type=10680000 len=136 idx=342/13652] fid=[0x200034e87:0x8700:0x0] dfid=[0x200034e87:0x8700:0x0] compound/cookie=0x57d8a6d9/0x57d89f3e action=ARCHIVE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[]
lrh=[type=10680000 len=136 idx=342/13655] fid=[0x20002a3ec:0xf909:0x0] dfid=[0x20002a3ec:0xf909:0x0] compound/cookie=0x57d8a6dc/0x57d89f41 action=ARCHIVE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[]

To retrieve statistics from this file to put them into Graphite/Grafana, I use the following script that extracts the number of lines by grouping them by "action" and also "status":

prefix="$CARBON_PREFIX.$MDT.hsm.actions"

lctl get_param mdt.$MDT.hsm.actions | awk -v prefix=$prefix 'BEGIN { now = systime() } / status=/ { action=gensub(/action=(.+)/, "\\1", "g", $7); status=gensub(/status=(.+)/, "\\1", "g", $(NF-1)); arr[action"."status] += 1   } END { for (s in arr) { printf "%s.%s %s %d\n", prefix, s, arr[s], now} }'

So I get lines that are valid for Graphite that look like:

srcc.sherlock.lustre.mdt.regal-MDT0000.hsm.actions.ARCHIVE.SUCCEED 3 1474463707

I am sure you do something similar for IML as you have a graph for HSM, but it is also broken when too many hsm actions are present.

The problem is that when the actions file reaches 100K's entries, it takes so long it is not parsable in a timely manner anymore...

I think having counters per "action" and "status" could be very useful, perhaps something like:

actions_stats:

status=archive status=STARTED count=23
status=archive status=SUCCEED count=1234
status=restore status=STARTED count=0
status=restore status=SUCCEED count=1

About the way to limit the size of the queue, a tunable like max_actions and -EFBIG sounds good, but still there might be a problem if restore actions are triggered when the action queue is full. Maybe the best would be a max_actions per action (archive, restore, remove)?...

[Andreas Dilger]

Hopefully Stéphane or Patrick can answer your question here. I was just trying to record the issue raised during RUG. It appears this is related to LU-7988 also caused by having a large number of items in the action list.

[Bruno Faccini (Inactive)]

Andreas, you sure about the fact that [cdt_]max_requests only concerns requests from the coordinator to a single copytool? I thought (and seems confirmed with my related code reading) it is used to account and limit the whole set of "active" requests been handled by a CDT and regardless to the concerned Agent(s).

About the performance to access the hsm_actions LLOG content when there is a huge back-log, I think that there have been already some work done, and not only in LU-7988. I will try to gather these infos and add to this ticket.

And also, as a first step, could it be acceptable to implement a simple/basic limit of a maximum number of (active and not) requests to be handled by a single CDT and simply return an error (-EFBIG ?) when reached ?

[Andreas Dilger]

I don't know enough about the details to write a good description of what needs to be done here. There is a max_requests tunable, but that appears to control the number of requests outstanding from the coordinator to a single copytool (cdt_max_requests).

Stephane, could you please add some info here about what variable(s) should be exposed via /proc, and what should be used to limit the size of the queue? What error should be returned by the coordinator if the action queue size limit is exceeded, -EFBIG?