[LU-14793] HSM: record the index when scan HSM action llog for new HSM requests

Related — Fri, 14 Jan 2022 06:07:30 +0000

After analyze online for HSM workload, we found the following contention between HSM archive request and hsm_cdtr thread:

hsm_archive:
  mdt_hsm_add_actions
  mdt_hsm_register_hal
  mdt_agent_record_add
    down_write(&cdt->cdt_llog_lock);
    llog_cat_add()
    up_write(&cdt->cdt_llog_lock);


// hsm_cdtr kernel daemon thread:
mdt_coordinator()
  cdt_llog_process(mti->mti_env, mdt, mdt_coordinator_cb,
				      &hsd, 0, 0, WRITE);
    down_write(&cdt->cdt_llog_lock);
    llog_cat_process
    up_write(&cdt->cdt_llog_lock);

# top
PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 7675 root      20   0       0      0      0 R 100.0  0.0   8729:13 hsm_cdtr
    1 root      20   0   53984   6232   2604 S   0.0  0.0   3:17.86 systemd
    2 root      20   0       0      0      0 S   0.0  0.0   0:00.62 kthreadd
    4 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:0H

HSM archive and kernel thread hsm_cdtr are both contented for cdt->cdt_llog_lock to add or update the hsm_action llog.

The testing is use max_requests = 10000:

mdt.isg-tiny-MDT0000.hsm.max_requests=10000

It means "hsm_cdtr" kernel thread will scan 10000 llog actions with write lock held in the whole processing, reduce the max_requests to a small value (i.e. 256) should be mitigate the problem. Or we can release the write lock when scan certain number of hsm actions (128) in llog, resched and then re-acquire the write lock to go on scanning the hsm action llog.

And also when "hsm_cdtr" scan the hsm_action llog, we should record the latest cookie that has been handled, next scanning should be started from the previous end position .

Whamcloud Community JIRA

[LU-14793] HSM: record the index when scan HSM action llog for new HSM requests