https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7986 Lustre <p>Some entries can end up being duplicated in "actions". For instance:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lrh=[type=10680000 len=136 idx=1/14] fid=[0x200000401:0x10:0x0] dfid=[0x200000401:0x10:0x0] compound/cookie=0x5703ea60/0x5703ea6c action=ARCHIVE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=1/51633] fid=[0x200000401:0x10:0x0] dfid=[0x200000401:0x10:0x0] compound/cookie=0x5703ea60/0x5703ea6c action=ARCHIVE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[] a </pre> </div></div> <p>As cdt_llog_lock is taken in mdt_hsm_actions_proc_show(), I think that could be a race between the coordinator/client and the proc_show() function.</p> LU-7986

HSM: Duplicated cookies in "actions"

Bug Minor Resolved Fixed John Hammond Frank Zago Tue, 5 Apr 2016 17:05:08 +0000 Mon, 10 Apr 2017 18:03:30 +0000 Thu, 21 Apr 2016 20:45:21 +0000 Lustre 2.9.0 0 7 <p>For my test setup, I create a new fs (llmount.sh), and then create 30000 files and hsm_archive them, without a copytool.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>#!/bin/sh mkdir /mnt/lustre/foo echo Start $(date "+%s") for x in {1..3000}; do echo loop $x $(date "+%s") mkdir /mnt/lustre/foo/d$x touch /mnt/lustre/foo/d$x/file{1..10} lfs hsm_archive /mnt/lustre/foo/d$x/file* done echo Done $(date "+%s") </pre> </div></div> <p>I then start the following command to watch "actions":</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>while [ 1 ]; do date "+%s" ; cat actions &gt; /root/copytool_test/actions.$(date "+%s") ; time wc -l actions ; sleep 1 ; done </pre> </div></div> <p>Then I start the copytool that will eventually succeed without any error or warnings.<br/> But I get puzzling output such as:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>30000 actions real 0m0.199s user 0m0.002s sys 0m0.197s 1459874583 30000 actions real 0m0.221s user 0m0.003s sys 0m0.218s 1459874584 30190 actions real 0m0.957s user 0m0.002s </pre> </div></div> <p>30190 actions? But only 30000 were posted.</p> <p>The worst I've seen is 34267 actions instead of 30000 max.</p> <p>Some FIDs/cookie even appear 3 times:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lrh=[type=10680000 len=136 idx=1/1415] fid=[0x200000401:0x605:0x0] dfid=[0x200000401:0x605:0x0] compound/cookie=0x5703eaea/0x5703efce action=ARCHIVE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=WAITING data=[] lrh=[type=10680000 len=136 idx=1/48820] fid=[0x200000401:0x605:0x0] dfid=[0x200000401:0x605:0x0] compound/cookie=0x5703eaea/0x5703efce action=ARCHIVE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=STARTED data=[] lrh=[type=10680000 len=136 idx=1/51221] fid=[0x200000401:0x605:0x0] dfid=[0x200000401:0x605:0x0] compound/cookie=0x5703eaea/0x5703efce action=ARCHIVE archive#=1 flags=0x0 extent=0x0-0xffffffffffffffff gid=0x0 datalen=0 status=SUCCEED data=[] </pre> </div></div> <p>actions.1459874574 is "actions" before starting the copytool<br/> actions.1459874610 is "actions" while the copytool is running</p> <p>The list of duplicated FIDs can be seen with the following command:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>sed -e "s/.*fid=\[\(.*\)\] dfid=.*/\1/" actions.1459874610 | sort | uniq -c | sort -r | less </pre> </div></div> <p>This comes from the way the llog record is updated. We could not update in place so we add a new one and remove the old one. This is done in mdt_agent_record_update_cb(): add a new record and ask the callback caller to invalid the old one.<br/> So if llog cleanning is not fast enough we can have multiple records for the same request.<br/> At HSM dev time, I did not find a way to update in place.</p> <p>May be that could explain why we're sometimes seeing replay of cookies. From our copytool traces, just after the hsm_recv:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Apr 5 19:20:23 [lustre] new request (cookie=57ac3d15, FID=0x2000d0ebb:0x4cd:0x0, DFID=0x2000d0ebb:0x4cd:0x0, len=72, action=20, extent=-1/0, gid=0, item=0) Apr 5 20:34:33 [lustre] new request (cookie=57ac3d15, FID=0x2000d0ebb:0x4cd:0x0, DFID=0x2000d0ebb:0x4cd:0x0, len=72, action=20, extent=-1/0, gid=0, item=0) Apr 5 20:37:31 [lustre] new request (cookie=57ac3d15, FID=0x2000d0ebb:0x4cd:0x0, DFID=0x2000d0ebb:0x4cd:0x0, len=72, action=20, extent=-1/0, gid=0, item=0) Apr 5 20:37:31 [lustre] new request (cookie=57ac3d15, FID=0x2000d0ebb:0x4cd:0x0, DFID=0x2000d0ebb:0x4cd:0x0, len=72, action=20, extent=-1/0, gid=0, item=0) </pre> </div></div> <p>Hi Frank,</p> <p>I'll push a patch that should address this. I'll push it after returning from LUG.</p> <p>Thanks.</p> <p>John L. Hammond (john.hammond@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/19447" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/19447</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7986" title="HSM: Duplicated cookies in &quot;actions&quot;" class="issue-link" data-issue-key="LU-7986"><del>LU-7986</del></a> hsm: update actions llog in place<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 54a07389e965a0cba7d246f8a6179591fb4b8798</p> <p>Frank Zago (fzago@cray.com) uploaded a new patch: <a href="http://review.whamcloud.com/19577" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/19577</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7986" title="HSM: Duplicated cookies in &quot;actions&quot;" class="issue-link" data-issue-key="LU-7986"><del>LU-7986</del></a> hsm: update actions llog in place<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: e4d96fb6b56a0d40aa721a67351bea6a62db97b9</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/19447/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/19447/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7986" title="HSM: Duplicated cookies in &quot;actions&quot;" class="issue-link" data-issue-key="LU-7986"><del>LU-7986</del></a> hsm: update actions llog in place<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 91144acb3dc1120c00797269afa621c94cb64e1e</p> <p>Landed to master for 2.9.0</p> Related LU-9312 Development Rank 1|hzy6tr: Rank (Obsolete) 9223372036854775807 Severity