[LU-14793] HSM: record the index when scan HSM action llog for new HSM requests

• Type: Bug
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.15.0
• Severity: 3
• Assignee: Qian Yingjin
• Reporter: Qian Yingjin

Description

After analyze online for HSM workload, we found the following contention between HSM archive request and hsm_cdtr thread:

hsm_archive:
  mdt_hsm_add_actions
  mdt_hsm_register_hal
  mdt_agent_record_add
    down_write(&cdt->cdt_llog_lock);
    llog_cat_add()
    up_write(&cdt->cdt_llog_lock);


// hsm_cdtr kernel daemon thread:
mdt_coordinator()
  cdt_llog_process(mti->mti_env, mdt, mdt_coordinator_cb,
				      &hsd, 0, 0, WRITE);
    down_write(&cdt->cdt_llog_lock);
    llog_cat_process
    up_write(&cdt->cdt_llog_lock);

# top
PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 7675 root      20   0       0      0      0 R 100.0  0.0   8729:13 hsm_cdtr
    1 root      20   0   53984   6232   2604 S   0.0  0.0   3:17.86 systemd
    2 root      20   0       0      0      0 S   0.0  0.0   0:00.62 kthreadd
    4 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:0H

HSM archive and kernel thread hsm_cdtr are both contented for cdt->cdt_llog_lock to add or update the hsm_action llog.

The testing is use max_requests = 10000:

mdt.isg-tiny-MDT0000.hsm.max_requests=10000

It means "hsm_cdtr" kernel thread will scan 10000 llog actions with write lock held in the whole processing, reduce the max_requests to a small value (i.e. 256) should be mitigate the problem. Or we can release the write lock when scan certain number of hsm actions (128) in llog, resched and then re-acquire the write lock to go on scanning the hsm action llog.

And also when "hsm_cdtr" scan the hsm_action llog, we should record the latest cookie that has been handled, next scanning should be started from the previous end position .

Issue Links

• is related to: LU-13689 Replace cdt_state_lock with cdt_llog_lock
• is related to: LU-13651 Conditionally skip finding compatible HSM requests

Comments

[Peter Jones]

Landed for 2.15

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/44077/
Subject: LU-14793 hsm: record index for further HSM action scanning
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: a15a5432f8063e3a04a87d74eafac0060a8f9d26

[Gerrit Updater]

Yingjin Qian ([e-mail]) uploaded a new patch: https://review.whamcloud.com/44077
Subject: LU-14793 hsm: record index for further HSM action scanning
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 1238bf2e3a1f29cf9eb7d1af3610a709fbce3865