https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5195 Lustre <p>Several times while testing HSM in a virtual environment (Centos 6.5 + Lustre 2.5.1 on clients and servers), we've observed what may be HSM_ACTIONS llog corruption.</p> <p>Here's our internal bug description:<br/> A Lustre filesystem where HSM and changelogs were used started misbehaving. The system was rebooted, and started spewing a lot of these traces in the system log:</p> <p>&lt;3&gt;LustreError: 2990:0:(mdt_hsm_cdt_actions.c:104:cdt_llog_process()) tas01-MDT0000: failed to process HSM_ACTIONS llog (rc=-2)<br/> &lt;3&gt;LustreError: 2990:0:(mdt_hsm_cdt_actions.c:104:cdt_llog_process()) Skipped 600 previous similar messages<br/> &lt;3&gt;LustreError: 2990:0:(llog_cat.c:192:llog_cat_id2handle()) tas01-MDD0000: error opening log id 0x1c:1:0: rc = -2<br/> &lt;3&gt;LustreError: 2990:0:(llog_cat.c:192:llog_cat_id2handle()) Skipped 600 previous similar messages<br/> &lt;3&gt;LustreError: 2990:0:(llog_cat.c:556:llog_cat_process_cb()) tas01-MDD0000: cannot find handle for llog 0x1c:1: -2<br/> &lt;3&gt;LustreError: 2990:0:(llog_cat.c:556:llog_cat_process_cb()) Skipped 600 previous similar messages<br/> &lt;3&gt;LustreError: 2990:0:(mdt_hsm_cdt_actions.c:104:cdt_llog_process()) tas01-MDT0000: failed to process HSM_ACTIONS llog (rc=-2)<br/> &lt;3&gt;LustreError: 2990:0:(mdt_hsm_cdt_actions.c:104:cdt_llog_process()) Skipped 600 previous similar messages<br/> &lt;3&gt;LustreError: 2990:0:(llog_cat.c:192:llog_cat_id2handle()) tas01-MDD0000: error opening log id 0x1c:1:0: rc = -2<br/> &lt;3&gt;LustreError: 2990:0:(llog_cat.c:192:llog_cat_id2handle()) Skipped 600 previous similar messages</p> <p>At that point the MDS would not accept any HSM request, nor would it deliver any.</p> <p>The MGT/MDT were unmounted and remounted as ldisk, and the file hsm_actions was deleted. Lustre was then remounted, and HSM became usable again.</p> <p>We do not have a simple reproducer for this, but it has happened several times.</p> LU-5195

HSM: mdt_hsm_cdt_actions.c:104:cdt_llog_process() failed to process HSM_ACTIONS llog

Bug Major Resolved Fixed James Nunez Patrick Farrell hsm patch Fri, 13 Jun 2014 18:13:45 +0000 Mon, 20 Apr 2015 17:21:46 +0000 Wed, 27 Aug 2014 17:15:03 +0000 Lustre 2.5.1 Lustre 2.7.0 Lustre 2.5.4 0 6 <p>Dump of the MDS is at:<br/> ftp.whamcloud.com:/uploads/<a href="https://jira.whamcloud.com/browse/LU-5195" title="HSM: mdt_hsm_cdt_actions.c:104:cdt_llog_process() failed to process HSM_ACTIONS llog" class="issue-link" data-issue-key="LU-5195"><del>LU-5195</del></a>/cdt_llog_process_HSM_ACTIONS_140613.tar.gz</p> <p>This issue occurred again on the same system. Here is what led up to the incident, according to the person who was working on the system:</p> <blockquote> <p>I was doing some archiving. The lustre client stopped working, so I rebooted both the client and the MDS.</p> <p>Now archiving is not working. On the client:</p> <p> &#35; cd /mnt/tas01/<br/> &#35; lfs hsm_archive fz<br/> Cannot send HSM request (use of fz): No such file or directory</p> <p>On the MDS:</p> <p> &#35; cat /proc/fs/lustre/mdt/tas01-MDT0000/hsm/actions<br/> cat: /proc/fs/lustre/mdt/tas01-MDT0000/hsm/actions: No such file or<br/> directory</p> <p>I don't think I did some unexpected actions from an admin point of view.</p></blockquote> <p>At this point, I got on the system and gathered as much relevant information as I could.</p> <p>I gathered full dk logs, the contents of the hsm_actions file on the MDT, the contents of the hsm proc files, and a dump of the system.</p> <p>I got the system working again by following the steps in this bug's description. That is,</p> <ol> <li>Unmount the Lustre MDT.</li> <li>Mount the MDT as ldiskfs.</li> <li>Remove the file hsm_actions from the root of the MDT.</li> <li>Unmount the MDT.</li> <li>Remount the MDT as Lustre. The LustreError messages stopped appearing in the console, and HSM was usable again.</li> </ol> <p>After I got the HSM working again, I checked what would happen if I replaced the hsm_actions file on the MDT with the "unhealthy" one which was in place when HSM was not working. When I did this and remounted the MDT as Lustre, I got the same LustreErrors in the console log again. Replacing the hsm_actions file with the one which was previously in place got it working again.</p> <p>The logs and dump mentioned in the above comment have been uploaded to the whamcloud ftp server.</p> <p>ftp.whamcloud.com:/uploads/<a href="https://jira.whamcloud.com/browse/LU-5195" title="HSM: mdt_hsm_cdt_actions.c:104:cdt_llog_process() failed to process HSM_ACTIONS llog" class="issue-link" data-issue-key="LU-5195"><del>LU-5195</del></a>/<a href="https://jira.whamcloud.com/browse/LU-5195" title="HSM: mdt_hsm_cdt_actions.c:104:cdt_llog_process() failed to process HSM_ACTIONS llog" class="issue-link" data-issue-key="LU-5195"><del>LU-5195</del></a>-logs.tar.gz</p> <p>That tar contains a README describing each file in it.</p> <p>This bug can be reproduced by inserting the failed hsm_actions on a healthy filesystem.</p> <p>Proposed fix: <a href="http://review.whamcloud.com/11419" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/11419</a></p> <p>Landed to master (2.7.0)</p> <p>Patch for b2_5 at <a href="http://review.whamcloud.com/#/c/11619/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/11619/</a></p> <p>Is there some reasons to prevent the b2_5 patch to also land? Seems an interesting fix, just missing a +2...</p> <p>Aurelien, </p> <p>When we start landing patches for 2.5.4, this patch will be considered for that release.</p> Related LU-6471 Development Rank 1|hzwp1z: Rank (Obsolete) 14513 Severity