https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-13689 Lustre <p>Merge cdt_llog_lock and cdt_state_lock and check cdt_state against CDT_STOPPED and CDT_INIT in mdt_agent_record_add(); this should allow for race-free checking of cdt_state with respect to cdt_state transitions and ensure that cdt_last_cookie has been properly set at CDT startup time, before it is used for new requests.</p> LU-13689

Replace cdt_state_lock with cdt_llog_lock

Bug Minor Open Unresolved Nikitas Angelinas Nikitas Angelinas Wed, 17 Jun 2020 11:47:57 +0000 Wed, 12 Oct 2022 19:25:42 +0000 0 4 <p>Nikitas Angelinas (nikitas.angelinas@hpe.com) uploaded a new patch: <a href="https://review.whamcloud.com/38963" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38963</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13689" title="Replace cdt_state_lock with cdt_llog_lock" class="issue-link" data-issue-key="LU-13689">LU-13689</a> hsm: replace cdt_state_lock with cdt_llog_lock<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 3123d6498c1310ab5472d44ea255b49aa62bda41</p> <p>The following shows a reproducer and that the fix addresses the issue:</p> <p>Reproducer:</p> <p>The reproducer depends on artificially-inserted delays in the relevant code paths to simulate the race condition between the threads. A 10 second delay is inserted in mdt_hsm_cdt_start() after the HSM coordinator is set to CDT_INIT and cdt_last_cookie is reset using ktime_get_real_seconds(), but before mdt_hsm_pending_restore() is called; this allows us to enlarge the race window artificially when restarting the HSM coordinator. A 5 second delay is added in mdt_hsm_add_actions(), after cdt_state is checked against CDT_STOPPED and CDT_INIT, but only after 1000 HSM requests have been handled; this allows the reproducer to enter the race window where cdt_state is checked and gives a small delay during which we can restart the HSM coordinator. Because the request-adding delay is smaller than the coordinator-restarting delay, the request-adding thread in mdt_hsm_add_actions() will check cdt_state and proceed since it is set to CDT_RUNNING. Once 1000 HSM requests have been handled by the HSM coordinator, we try to archive the 1001th request. The 5-second delay allows us to restart the coordinator quickly; this resets cdt_last_cookie. The request-adding thread wakes up quicker than the coordinator-restarting thread since it has a smaller delay and since cdt_last_cookie has not yet been incremented to an unused value by the coordinator-starting thread, it proceeds to add the HSM request to the llog using an already-used cookie value. This leads to the following assertions being hit in cdt_agent_record_hash_add():</p> <p>LASSERT(carl0-&gt;carl_cat_idx == carl1-&gt;carl_cat_idx);<br/> LASSERT(carl0-&gt;carl_rec_idx == carl1-&gt;carl_rec_idx);</p> <p>Applying the fix from <a href="https://review.whamcloud.com/#/c/38963/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/38963/</a> allows the HSM coordinator to return -EAGAIN, instead of hitting the assertions.</p> <p>Steps to reproduce:</p> <p>1. Set up a minimal Lustre cluster, one with llmount.sh and cfg/local.sh is sufficient; a copytool is not needed, but lhsmtool_posix can be used<br/> 2. Start the HSM coordinator<br/> 3. Set grace_delay to a large value, e.g. 1200; this allows HSM requests to remain in the llog long enough for the reproducer to take place and cause the assertions to be hit<br/> 4. Create 1001 files<br/> 5. Archive 1000 of the files<br/> 6. Archive the last file; notice that a small delay is incurred. During this delay window, shut down and enable the coordinator<br/> With a branch derived from somewhat recent master, the following output is observed:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># lfs hsm_archive /mnt/lustre/testfile1001 # dmesg -c ... kernel:LustreError: 30362:0:(mdt_hsm_cdt_actions.c:139:cdt_agent_record_hash_add()) ASSERTION( carl0-&gt;carl_rec_idx == carl1-&gt;carl_rec_idx ) failed: ... kernel:LustreError: 30362:0:(mdt_hsm_cdt_actions.c:139:cdt_agent_record_hash_add()) LBUG [root@c02sxfm0gtfm-mds1 tests]# dmesg -c [19605.583099] LustreError: 30362:0:(mdt_hsm_cdt_actions.c:139:cdt_agent_record_hash_add()) ASSERTION( carl0-&gt;carl_rec_idx == carl1-&gt;carl_rec_idx ) failed: [19605.583105] LustreError: 30362:0:(mdt_hsm_cdt_actions.c:139:cdt_agent_record_hash_add()) LBUG [19605.583107] Pid: 30362, comm: hsm_cdtr 3.10.0-1062.9.1.el7.x86_64 #1 SMP Fri Dec 6 15:49:49 UTC 2019 [19605.583108] Call Trace: [19605.583121] [&lt;ffffffffc06557cc&gt;] libcfs_call_trace+0x8c/0xc0 [libcfs] [19605.583131] [&lt;ffffffffc065587c&gt;] lbug_with_loc+0x4c/0xa0 [libcfs] [19605.583136] [&lt;ffffffffc11a64f5&gt;] cdt_agent_record_hash_add+0x355/0x3c0 [mdt] [19605.583154] [&lt;ffffffffc11b2969&gt;] mdt_cdt_waiting_cb.isra.26+0x729/0xcc0 [mdt] [19605.583167] [&lt;ffffffffc11b4772&gt;] mdt_coordinator_cb+0x162/0x290 [mdt] [19605.583180] [&lt;ffffffffc07d37ff&gt;] llog_process_thread+0x85f/0x1a20 [obdclass] [19605.583199] [&lt;ffffffffc07d4a8f&gt;] llog_process_or_fork+0xcf/0x520 [obdclass] [19605.583210] [&lt;ffffffffc07da1b9&gt;] llog_cat_process_cb+0x239/0x250 [obdclass] [19605.583222] [&lt;ffffffffc07d37ff&gt;] llog_process_thread+0x85f/0x1a20 [obdclass] [19605.583234] [&lt;ffffffffc07d4a8f&gt;] llog_process_or_fork+0xcf/0x520 [obdclass] [19605.583245] [&lt;ffffffffc07d6911&gt;] llog_cat_process_or_fork+0x1f1/0x370 [obdclass] [19605.583256] [&lt;ffffffffc07d6abe&gt;] llog_cat_process+0x2e/0x30 [obdclass] [19605.583268] [&lt;ffffffffc11a6a26&gt;] cdt_llog_process+0xc6/0x3a0 [mdt] [19605.583280] [&lt;ffffffffc11b06d1&gt;] mdt_coordinator+0x541/0x19f0 [mdt] [19605.583291] [&lt;ffffffff8ccc61f1&gt;] kthread+0xd1/0xe0 [19605.583296] [&lt;ffffffff8d38dd37&gt;] ret_from_fork_nospec_end+0x0/0x39 [19605.583300] [&lt;ffffffffffffffff&gt;] 0xffffffffffffffff </pre> </div></div> <p>An LBUG can be triggered due to this issue in cdt_agent_record_hash_add() on either carl_cat_idx or carl_rec_idx (as shown in the example above) depending on whether the two llog entries are in the same catalog entry or not. With the fix from <a href="https://review.whamcloud.com/#/c/38963/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/38963/</a> applied, this same code allows the MDT to return -EAGAIN, instead of triggering one of the two LBUGs:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># lfs hsm_archive /mnt/lustre/testfile1000 Cannot send HSM request (use of /mnt/lustre/testfile1000): Resource temporarily unavailable # dmesg -c [ 385.347189] Lustre: lustre-OST0000-osc-ffff97ad8ac4a800: reconnect after 16s idle [ 385.347324] Lustre: lustre-OST0000: Connection restored to b3fb17d8-3cf4-5172-6801-684ff0fc684a (at 192.168.42.90@tcp) </pre> </div></div> <p>-EAGAIN is returned from the following change that the fix introduces in mdt_agent_record_add():</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>if ((cdt-&gt;cdt_state == CDT_STOPPED || cdt-&gt;cdt_state == CDT_INIT) &amp;&amp; !(flags &amp; HAL_CDT_FORCE)) GOTO(unavail, rc = -EAGAIN); </pre> </div></div> Duplicate LU-16235 Related LU-11675 LU-14793 Development Rank 1|i012yn: Rank (Obsolete) 9223372036854775807 Severity