https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-14399 Lustre <p>We observed that when mounting MDT with HSM enable, mount command take minutes compare to seconds as before. We saw this in the log</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [53618.238941] Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 [53618.624098] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [53720.390690] Lustre: 1722736:0:(mdt_coordinator.c:1114:mdt_hsm_cdt_start()) lustre-MDT0001: trying to init HSM before MDD [53720.392834] LustreError: 1722736:0:(mdt_coordinator.c:1125:mdt_hsm_cdt_start()) lustre-MDT0001: cannot take the layout locks needed <span class="code-keyword">for</span> registered restore: -2 [53720.398049] LustreError: 1722741:0:(mdt_coordinator.c:1090:mdt_hsm_cdt_start()) lustre-MDT0001: Coordinator already started or stopping [53720.400681] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [53720.424872] Lustre: lustre-MDT0001: in recovery but waiting <span class="code-keyword">for</span> the first client to connect [53720.953893] Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check [53722.067555] Lustre: DEBUG MARKER: </pre> </div></div> <p>Seems related to <a href="https://jira.whamcloud.com/browse/LU-13920" title="HSM: hsm_actions are not processed after MDS failover" class="issue-link" data-issue-key="LU-13920"><del>LU-13920</del></a></p> <a href="https://testing.whamcloud.com/test_sets/20fcc789-47a3-44a2-a17e-d6f39fbb3ec0">https://testing.whamcloud.com/test_sets/20fcc789-47a3-44a2-a17e-d6f39fbb3ec0</a> LU-14399

mount MDT takes very long with hsm enable

Bug Minor Resolved Fixed Sergey Cheremencev Minh Diep Wed, 3 Feb 2021 21:19:02 +0000 Mon, 14 Feb 2022 23:20:57 +0000 Tue, 18 Jan 2022 14:54:30 +0000 Lustre 2.15.0 0 9 <p>John L. Hammond (jhammond@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/41409" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/41409</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14399" title="mount MDT takes very long with hsm enable" class="issue-link" data-issue-key="LU-14399"><del>LU-14399</del></a> Revert "<a href="https://jira.whamcloud.com/browse/LU-13920" title="HSM: hsm_actions are not processed after MDS failover" class="issue-link" data-issue-key="LU-13920"><del>LU-13920</del></a> hsm: process hsm_actions only after mdd setup"<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 5b375e90e3cc3538f4dd92dc81f98fcf5b98e41e</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>        /* wait until MDD initialize hsm actions llog */         while (!test_bit(MDT_FL_CFGLOG, &amp;mdt-&gt;mdt_state) &amp;&amp; i &lt; obd_timeout) {                 schedule_timeout_interruptible(cfs_time_seconds(1));                 i++;         }         if (!test_bit(MDT_FL_CFGLOG, &amp;mdt-&gt;mdt_state))                 CWARN("%s: trying to init HSM before MDD\n", mdt_obd_name(mdt));</pre> </div></div> <p><a href="https://jira.whamcloud.com/browse/LU-13920" title="HSM: hsm_actions are not processed after MDS failover" class="issue-link" data-issue-key="LU-13920"><del>LU-13920</del></a> just waits until MDT_FL_CFGLOG will be set.<br/> This flag is set at the end of mdt_prepare. There is a chance mdt_prepare was stuck for some reasons.<br/> Anyway I need debug logs from MDT to say something.<br/> Is it possible to reproduce it again and gather debug logs?</p> <p>John L. Hammond (jhammond@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/41415" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/41415</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14399" title="mount MDT takes very long with hsm enable" class="issue-link" data-issue-key="LU-14399"><del>LU-14399</del></a>: Revert "<a href="https://jira.whamcloud.com/browse/LU-13651" title="Conditionally skip finding compatible HSM requests" class="issue-link" data-issue-key="LU-13651"><del>LU-13651</del></a> hsm: call hsm_find_compatible_cb() only for cancel"<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 4ed2cd37d3331511ffff1fcd2bbe53c9b2513502</p> <p>here is the sequence that I did to hit this bug</p> <ol> <li>mkfs.lustre --mdt --mgsnode ....</li> <li>tunefs.lustre --param mdt.hsm_control=enabled ...</li> <li>mount -t lustre ...</li> </ol> <blockquote><p>here is the sequence that I did to hit this bug</p></blockquote> <p>Thank you. Now the problem is clear.</p> <p>I'll provide a fix in a few days.</p> <p>Sergey Cheremencev (sergey.cheremencev@hpe.com) uploaded a new patch: <a href="https://review.whamcloud.com/41445" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/41445</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14399" title="mount MDT takes very long with hsm enable" class="issue-link" data-issue-key="LU-14399"><del>LU-14399</del></a> hsm: process hsm_actions in coordinator<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: ec9b9b8d2e05568f75ca75e596585503ae0d4216</p> <p>Sergey Cheremencev (sergey.cheremencev@hpe.com) uploaded a new patch: <a href="https://review.whamcloud.com/42005" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/42005</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14399" title="mount MDT takes very long with hsm enable" class="issue-link" data-issue-key="LU-14399"><del>LU-14399</del></a> tests: hsm_actions after failover<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 87493ef365d9faaf1f6c1e1a40f65157d37f72dc</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/41445/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/41445/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14399" title="mount MDT takes very long with hsm enable" class="issue-link" data-issue-key="LU-14399"><del>LU-14399</del></a> hsm: process hsm_actions in coordinator<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: e26d7cc3992252e5fce5a51aee716f933b04c13a</p> <p>Landed for 2.15</p> <p>the patch just landed fails every run on my setup:</p> <blockquote> <p>...<br/> Writing CONFIGS/mountdata<br/> start mds service on tmp.BKaRODHgLn<br/> Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1<br/> Started lustre-MDT0000<br/> conf-sanity test_132: @@@@@@ FAIL: Can not take the layout lock <br/> Trace dump:<br/> = ./../tests/test-framework.sh:6389:error()<br/> = conf-sanity.sh:9419:test_132()<br/> = ./../tests/test-framework.sh:6693:run_one()<br/> = ./../tests/test-framework.sh:6740:run_one_logged()<br/> = ./../tests/test-framework.sh:6581:run_test()<br/> = conf-sanity.sh:9422:main()<br/> Dumping lctl log to /tmp/ltest-logs/conf-sanity.test_132.*.1642612854.log<br/> Dumping logs only on local client.<br/> FAIL 132 (84s)</p></blockquote> <p>Hello Alex,</p> <p>what setup did you use? It doesn't fail on my local VM on the latest master(0feec5a3).</p> <p>well, essentially just a local VM:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">FSTYPE=ldiskfs MDSCOUNT=2 MDSSIZE=300000 OSTSIZE=400000 OSTCOUNT=2 LOGDIR=/tmp/ltest-logs REFORMAT=yes HONOR_EXCEPT=y bash conf-sanity.sh </pre> </div></div> <p>Alex, it still doesn't fail on my local VM.<br/> I can you look into the logs if you attach them to the ticket.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [ 8878.042110] Lustre: Found index 0 <span class="code-keyword">for</span> lustre-MDT0000, updating log [ 8878.044766] Lustre: Modifying parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 [ 8898.880071] Lustre: 497204:0:(mdt_coordinator.c:1145:mdt_hsm_cdt_start()) lustre-MDT0000: trying to init HSM before MDD [ 8898.888824] LustreError: 497204:0:(mdt_coordinator.c:1156:mdt_hsm_cdt_start()) lustre-MDT0000: cannot take the layout locks needed <span class="code-keyword">for</span> registered restore: -2 [ 8899.150565] Lustre: DEBUG MARKER: conf-sanity test_132: @@@@@@ FAIL: Can not take the layout lock </pre> </div></div> <p>I guess "trying to init HSM before MDD" is a hint?</p> <blockquote><blockquote><p> I guess "trying to init HSM before MDD" is a hint?</p></blockquote></blockquote> <p>IMO, it hints your build doesn't include "<a href="https://jira.whamcloud.com/browse/LU-14399" title="mount MDT takes very long with hsm enable" class="issue-link" data-issue-key="LU-14399"><del>LU-14399</del></a> hsm: process hsm_actions in coordinator".</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>static void cdt_start_pending_restore(struct mdt_device *mdt,                       struct coordinator *cdt) {     struct mdt_thread_info *cdt_mti;     unsigned int i = 0;     int rc;    /* wait until MDD initialize hsm actions llog */     while (!test_bit(MDT_FL_CFGLOG, &amp;mdt-&gt;mdt_state) &amp;&amp; i &lt; obd_timeout) {         schedule_timeout_interruptible(cfs_time_seconds(1));         i++;     }     if (!test_bit(MDT_FL_CFGLOG, &amp;mdt-&gt;mdt_state))         CWARN("%s: trying to init HSM before MDD\n", mdt_obd_name(mdt)); </pre> </div></div> <p>"trying to init HSM before MDD" message should be printed by cdt_start_pending_restore, while in your case it is mdt_hsm_cdt_start.</p> <p>well, I'm running master branch, so clearly <a href="https://jira.whamcloud.com/browse/LU-14399" title="mount MDT takes very long with hsm enable" class="issue-link" data-issue-key="LU-14399"><del>LU-14399</del></a> is in. this is what helps:\</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> - <span class="code-keyword">while</span> (!test_bit(MDT_FL_CFGLOG, &amp;mdt-&gt;mdt_state) &amp;&amp; i &lt; obd_timeout) { + <span class="code-keyword">while</span> (!test_bit(MDT_FL_CFGLOG, &amp;mdt-&gt;mdt_state) &amp;&amp; i &lt; obd_timeout * 2) { </pre> </div></div> <p>not sure how good it is...</p> Related LU-13920 Development Rank 1|i01lh3: Rank (Obsolete) 9223372036854775807 Severity