https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5420 Lustre <p>This issue was created by maloo for sarah &lt;sarah@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.hpdd.intel.com/test_sets/16302020-14ed-11e4-bb6a-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/16302020-14ed-11e4-bb6a-5254006e85c2</a>.</p> <p>The sub-test test_17m failed with the following error:</p> <blockquote> <p>test failed to respond and timed out</p></blockquote> <p>Hit this bug in many tests, the env is configured as 1 MDS with 2 MDTs. Didn't hit this error when the configuration is 2 MDSs with 2 MDTs<br/> client console:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>CMD: onyx-46vm7 mkdir -p /mnt/mds1 CMD: onyx-46vm7 test -b /dev/lvm-Role_MDS/P1 Starting mds1: /dev/lvm-Role_MDS/P1 /mnt/mds1 CMD: onyx-46vm7 mkdir -p /mnt/mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/mds1 onyx-46vm7: mount.lustre: mount /dev/mapper/lvm--Role_MDS-P1 at /mnt/mds1 failed: Input/output error onyx-46vm7: Is the MGS running? Start of /dev/lvm-Role_MDS/P1 on mds1 failed 5 </pre> </div></div> client and server: lustre-b2_6-rc2 RHEL6 ldiskfs DNE mode LU-5420

Failure on test suite sanity test_17m: mount MDS failed, Input/output error

Bug Blocker Resolved Fixed Di Wang Sarah Liu HB dne patch Sat, 26 Jul 2014 19:08:35 +0000 Tue, 19 Mar 2019 15:15:57 +0000 Mon, 11 May 2015 17:36:00 +0000 Lustre 2.6.0 Lustre 2.7.0 Lustre 2.8.0 0 16 <p>mds console</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>11:01:42:Lustre: DEBUG MARKER: mkdir -p /mnt/mds1; mount -t lustre /dev/lvm-Role_MDS/P1 /mnt/mds1 11:01:42:LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. quota=on. Opts: 11:01:42:LustreError: 166-1: MGC10.2.4.243@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail 11:01:43:Lustre: Evicted from MGS (at MGC10.2.4.243@tcp_0) after server handle changed from 0x5efc70d9d01b7154 to 0x5efc70d9d01e1f23 11:01:43:LustreError: 18197:0:(obd_mount_server.c:1165:server_register_target()) lustre-MDT0000: error registering with the MGS: rc = -108 (not fatal) 11:01:45:LustreError: 15c-8: MGC10.2.4.243@tcp: The configuration from log 'lustre-MDT0000' failed (-5). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information. 11:01:45:Lustre: MGC10.2.4.243@tcp: Connection restored to MGS (at 0@lo) 11:01:46:LustreError: 18197:0:(obd_mount_server.c:1297:server_start_targets()) failed to start server lustre-MDT0000: -5 11:01:46:LustreError: 18197:0:(obd_mount_server.c:1769:server_fill_super()) Unable to start targets: -5 11:01:46:LustreError: 18197:0:(obd_mount_server.c:1496:server_put_super()) no obd lustre-MDT0000 11:01:47:LustreError: 18197:0:(obd_mount.c:1342:lustre_fill_super()) Unable to mount (-5) 11:01:47:Lustre: DEBUG MARKER: lctl set_param -n mdt.lustre*.enable_remote_dir=1 11:01:47:LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 10.2.4.244@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. 11:01:47:LustreError: Skipped 11 previous similar messages </pre> </div></div> <p>Sigh, this is brought in by <a href="http://review.whamcloud.com/#/c/9967/6" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/9967/6</a></p> <p><a href="https://jira.whamcloud.com/browse/LU-4913" title="mgc import reconnect race" class="issue-link" data-issue-key="LU-4913"><del>LU-4913</del></a> mgc: mgc import reconnect race</p> <p>mgc import can be reconnected by pinger or<br/> ptlrpc_reconnect_import().<br/> ptlrpc_invalidate_import() isn't protected against<br/> alteration of imp_invalid state. Import can be<br/> reconnected by pinger which makes imp_invalid<br/> equal to false. Thus LASSERT(imp-&gt;imp_invalid) fails<br/> in ptlrpc_invalidate_import().</p> <p>It is safe to call ptlrpc_invalidate_import() when<br/> import is deactivated, but ptlrpc_reconnect_import() doesn't<br/> deactivate it.<br/> Let's use only pinger when available to reconnect import</p> <p>Hmm, in ptlrpc_reconnect_import, the patch does not force the the import to reconnect to the server, instead it only check the import status, which seems wrong. Given that the import status might change later soon. I think the intention here to make the import is refreshed and connected after this call. </p> <p>Though I am not so sure what the patch are trying to resolve here, since comment is a bit confused to me. I think there two options to fix the problem</p> <p>1. revert the patch 9967, or just revert the change in ptlrpc_reconnect_import to force reconnect anyway.<br/> 2. mgc should retry to enqueue and get log when it meet invalid import (by checking the return value -ESHUTDOWN, see ptlrpc_import_delay_req).</p> <p>I will cook these two patches to see which one is better. </p> <p>Btw: We hit this until now because our current maloo DNE configuration is a bit different then this final FULL release test. Here we are using 2 MDTs in MDS, where in maloo DNE test, we also use 2 MDSes, but 1 MDS only 1 MDTs, the other MDS has 3 MDTs. </p> <p>This failure will only be hit when there are multiple MDTs on the first MDS. </p> <p>Option 1: <a href="http://review.whamcloud.com/11241" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/11241</a></p> <p>Option 2: <a href="http://review.whamcloud.com/11240" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/11240</a></p> <p>Personally I like option 1, but I am not sure whether reverting the patch will bring <a href="https://jira.whamcloud.com/browse/LU-4913" title="mgc import reconnect race" class="issue-link" data-issue-key="LU-4913"><del>LU-4913</del></a> back, the patch description is a bit confused to me.</p> <p>The option #2 patch is testing well on my local system (single-node 2x MDT, 3x OST, client) which was having solid test failures in sanity.sh test_17m and test_17o (which I'd incorrectly attributed to <a href="https://jira.whamcloud.com/browse/LU-1538" title="cleanup test scripts" class="issue-link" data-issue-key="LU-1538"><del>LU-1538</del></a> patch <a href="http://review.whamcloud.com/10481" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/10481</a> that was reverted).</p> <p>I've pushed an updated version of the 11240 patch at <a href="http://review.whamcloud.com/11258" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/11258</a> with improved comments and removing some noise from the console. Since this might be a blocker I didn't refresh the original 11240 patch so that it could continue testing, but I'd prefer that the 11258 version land if it is ready.</p> <p>It seems that this patch is repeatedly failing insanity, even when it is running on b2_6. The failures are marked as <a href="https://jira.whamcloud.com/browse/LU-5077" title="insanity test_1: out of memory on MDT in crypto_create_tfm()" class="issue-link" data-issue-key="LU-5077"><del>LU-5077</del></a>, but I don't think that is the real reason. I suspect there is some other problem with this patch that needs to be investigated.</p> <p>I verified that virtually all of the test failures marked <a href="https://jira.whamcloud.com/browse/LU-5077" title="insanity test_1: out of memory on MDT in crypto_create_tfm()" class="issue-link" data-issue-key="LU-5077"><del>LU-5077</del></a> are actually from the three versions of the <a href="https://jira.whamcloud.com/browse/LU-5420" title="Failure on test suite sanity test_17m: mount MDS failed, Input/output error" class="issue-link" data-issue-key="LU-5420"><del>LU-5420</del></a> patches, which fail "insanity" and "conf-sanity" repeatedly. Due to the presence of <a href="https://jira.whamcloud.com/browse/LU-5425" title="Test failure on test suite conf-sanity, subtest test_32a (mount OST failed)" class="issue-link" data-issue-key="LU-5425"><del>LU-5425</del></a>, I'm not 100% positive that those are caused by this patch, but definitely the insanity failures.</p> <p>Hmm, I think insanity failures should be related with the fix from <a href="https://jira.whamcloud.com/browse/LU-5420" title="Failure on test suite sanity test_17m: mount MDS failed, Input/output error" class="issue-link" data-issue-key="LU-5420"><del>LU-5420</del></a>. I am looking at it now. </p> <p><span class="error">&#91;7/30/14, 12:54:36 PM&#93;</span> wangdi: the failure of insanity is because of with the fix of <a href="https://jira.whamcloud.com/browse/LU-5425" title="Test failure on test suite conf-sanity, subtest test_32a (mount OST failed)" class="issue-link" data-issue-key="LU-5425"><del>LU-5425</del></a>, MDT will insist MGS must be started, i.e. MDS setup process will wait there until MGS is setup. But in insanity test_1, it start mdt2 first, then mdt1/mgs, that is why the test will fail<br/> <span class="error">&#91;7/30/14, 12:55:32 PM&#93;</span> wangdi: so can we just fix the test case here, because it seems to me MGS must be setup first</p> <p>Sigh, most of insanity test starts MDT or OST before MGS, that is why it cause so many insanity failure with this patch. So if "starting mgs before other targets" is a requirement then we need fix insanity.</p> <p>Test specific issues aside, we need to fix this as putting 2 MDTs from a FS on a single node will be a likely failover configuration.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>t:lustre-release# export LUSTRE=$HOME/lustre-release/lustre t:lustre-release# export MDSCOUNT=2 t:lustre-release# llmount.sh ... Starting mds1: -o loop /tmp/lustre-mdt1 /mnt/mds1 Started lustre-MDT0000 Starting mds2: -o loop /tmp/lustre-mdt2 /mnt/mds2 Started lustre-MDT0001 Starting ost1: -o loop /tmp/lustre-ost1 /mnt/ost1 Started lustre-OST0000 Starting ost2: -o loop /tmp/lustre-ost2 /mnt/ost2 Started lustre-OST0001 Starting client: t: -o user_xattr,flock t@tcp:/lustre /mnt/lustre Using TIMEOUT=20 seting jobstats to procname_uid Setting lustre.sys.jobid_var from disable to procname_uid Waiting 90 secs for update Updated after 3s: wanted 'procname_uid' got 'procname_uid' disable quota as required t:lustre-release# umount /mnt/mds1 t:lustre-release# mount /tmp/lustre-mdt1 /mnt/mds1 -o loop -t lustre mount.lustre: mount /dev/loop0 at /mnt/mds1 failed: Input/output error Is the MGS running? </pre> </div></div> <p>Without this patch I'm also not able to test past sanity.sh test_17m and test_17n without a shared MGS+MDS failing to mount due to -EIO and causing testing to hang until I remount the MDS. I'm able to mount it manually after 2 or 3 tries, so there must be some kind of startup race between the MDS and the MGS. Once I applied this patch I made it through all of sanity.sh and sanityn.sh with multiple MDS remounts without problems until I hit a memory allocation deadlock running dbench that looks unrelated.</p> <p>The patch is still failing with a hang at unmount time (this failed in four separate conf-sanity runs in different subtests):</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>01:52:25:INFO: task umount:26263 blocked for more than 120 seconds. 01:52:25: Tainted: G W --------------- 2.6.32-431.23.3.el6_lustre.g9f5284f.x86_64 #1 01:52:26:umount D 0000000000000000 0 26263 26262 0x00000080 01:52:27:Call Trace: 01:52:27: [&lt;ffffffff8152b6e5&gt;] rwsem_down_failed_common+0x95/0x1d0 01:52:27: [&lt;ffffffff8152b843&gt;] rwsem_down_write_failed+0x23/0x30 01:52:28: [&lt;ffffffff8128f7f3&gt;] call_rwsem_down_write_failed+0x13/0x20 01:52:28: [&lt;ffffffffa0b13cd1&gt;] client_disconnect_export+0x61/0x460 [ptlrpc] 01:52:28: [&lt;ffffffffa058975a&gt;] lustre_common_put_super+0x28a/0xbf0 [obdclass] 01:52:28: [&lt;ffffffffa05bc508&gt;] server_put_super+0x198/0xe50 [obdclass] 01:52:29: [&lt;ffffffff8118b23b&gt;] generic_shutdown_super+0x5b/0xe0 01:52:29: [&lt;ffffffff8118b326&gt;] kill_anon_super+0x16/0x60 01:52:29: [&lt;ffffffffa0580d06&gt;] lustre_kill_super+0x36/0x60 [obdclass] 01:52:29: [&lt;ffffffff8118bac7&gt;] deactivate_super+0x57/0x80 01:52:29: [&lt;ffffffff811ab4cf&gt;] mntput_no_expire+0xbf/0x110 01:52:29: [&lt;ffffffff811ac01b&gt;] sys_umount+0x7b/0x3a0 </pre> </div></div> <p>Just updated the patch.</p> <p>Hello</p> <p>We hit these problem in xyratex and have another solution <a href="http://review.whamcloud.com/#/c/12515/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/12515/</a>.<br/> Hope it could be helpful.</p> <p>Sergey Cheremencev (sergey_cheremencev@xyratex.com) uploaded a new patch: <a href="http://review.whamcloud.com/12515" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12515</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5420" title="Failure on test suite sanity test_17m: mount MDS failed, Input/output error" class="issue-link" data-issue-key="LU-5420"><del>LU-5420</del></a> mgc: process config logs only in mgc_requeue_thread()<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 3<br/> Commit: 1c3148dd8645cfa94bf3c36cfbe41176334ad4c5</p> <p>While running replay-dual tests on master branch with MDSCOUNT=4, the same failure occurred:<br/> <a href="https://testing.hpdd.intel.com/test_sets/33dfc794-6dba-11e4-9d65-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/33dfc794-6dba-11e4-9d65-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/5cb7b7f8-6dba-11e4-9d65-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/5cb7b7f8-6dba-11e4-9d65-5254006e85c2</a></p> <p>In seagate these bug is occurred only when MDT and MGS are on same node, in case when mdt starts earlier than mgs.</p> <p>When MDT is on separate node it uses LOCAL configuration if can't retrieve it from MGS.<br/> But when MDT and MGS are on same node MDT can't use LOCAL configuration:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> /* Copy the setup log locally if we can. Don't mess around if we're * running an MGS though (logs are already local). */ if (lctxt &amp;&amp; lsi &amp;&amp; IS_SERVER(lsi) &amp;&amp; !IS_MGS(lsi) &amp;&amp; cli-&gt;cl_mgc_configs_dir != NULL &amp;&amp; lu2dt_dev(cli-&gt;cl_mgc_configs_dir-&gt;do_lu.lo_dev) == lsi-&gt;lsi_dt_dev) { .... } else { if (local_only) /* no local log at client side */ GOTO(out_pop, rc = -EIO); } </pre> </div></div> <p>Hmm, I thought there are two problems here<br/> 1. it is not just MGS and MDT are not share the same node, if several targets are sharing the same mgc, you will meet similar problem, because after <a href="http://review.whamcloud.com/#/c/9967" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/9967</a> is landed, we can not make sure the import is FULL before mgc enqueue lock and retrieve logs, unless the MGC is new.<br/> 2. How can we make sure the local config log is stale or not. I think that is the reason we saw <a href="https://jira.whamcloud.com/browse/LU-5658" title="sanity test_17n: destroy remote dir error 0" class="issue-link" data-issue-key="LU-5658"><del>LU-5658</del></a>, where the local config is stale.</p> <p>As a note I don't see this is my regular RHEL testing but I can constantly reproduce this problem with my 3.12 kernel setup. This is with the MGS and MDS each being on separate nodes.</p> <p>Alexey Lyashkov (alexey.lyashkov@seagate.com) uploaded a new patch: <a href="http://review.whamcloud.com/13693" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13693</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5420" title="Failure on test suite sanity test_17m: mount MDS failed, Input/output error" class="issue-link" data-issue-key="LU-5420"><del>LU-5420</del></a> mgc: fix reconnect<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: ccfca18ad2ae9acb84dbfc4c0b2217bd10a0589d</p> <p><a href="http://review.whamcloud.com/#/c/12515/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/12515/</a></p> <p>Oleg Drokin (oleg.drokin@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/13832" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13832</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5420" title="Failure on test suite sanity test_17m: mount MDS failed, Input/output error" class="issue-link" data-issue-key="LU-5420"><del>LU-5420</del></a> revert part of <a href="https://jira.whamcloud.com/browse/LU-4913" title="mgc import reconnect race" class="issue-link" data-issue-key="LU-4913"><del>LU-4913</del></a><br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 260e150f98f07fa68fb124348ca9540e77fed100</p> <p>Oleg Drokin (oleg.drokin@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/13838" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13838</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5420" title="Failure on test suite sanity test_17m: mount MDS failed, Input/output error" class="issue-link" data-issue-key="LU-5420"><del>LU-5420</del></a> revert part of <a href="https://jira.whamcloud.com/browse/LU-4913" title="mgc import reconnect race" class="issue-link" data-issue-key="LU-4913"><del>LU-4913</del></a><br/> Project: fs/lustre-release<br/> Branch: b2_7<br/> Current Patch Set: 1<br/> Commit: 77856caa2468dd69cfa5796bceb22c32aacf402f</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/13838/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13838/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5420" title="Failure on test suite sanity test_17m: mount MDS failed, Input/output error" class="issue-link" data-issue-key="LU-5420"><del>LU-5420</del></a> ptlrpc: revert ptlrpc_reconnect_import() changes<br/> Project: fs/lustre-release<br/> Branch: b2_7<br/> Current Patch Set: <br/> Commit: 02739a078f54b5ccdf49456fd0d1daea90472a8d</p> <p>Patches landed to Master.</p> <p>Actually Jodi the patches for master are still in flight. It is simply a workaround fix that has landed to b2_7</p> <p>Yes, my apologies.</p> <p>I see many patches for this. Which patches are valid?</p> <p>For now, you can use this <a href="http://review.whamcloud.com/13838/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13838/</a> , but that only reverts the patch (<a href="http://review.whamcloud.com/#/c/9967/)which" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/9967/)which</a> cause the problem, and not real fix. </p> <p>There are patches trying to fix this problem, but none of them are satisfied by everyone. So leave it to 2.8 for now.<br/> <a href="http://review.whamcloud.com/13693" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13693</a><br/> <a href="http://review.whamcloud.com/11258" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/11258</a></p> <p>Andreas Dilger (andreas.dilger@intel.com) merged in patch <a href="http://review.whamcloud.com/11258/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/11258/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5420" title="Failure on test suite sanity test_17m: mount MDS failed, Input/output error" class="issue-link" data-issue-key="LU-5420"><del>LU-5420</del></a> mgc: MGC should retry for invalid import<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 77d406a0699307e8e633ef41f8984f45c09db9b8</p> <p>Landed for 2.8</p> Duplicate Related LU-4913 LU-5404 LU-5077 LU-5407 LU-5130 LU-8206 Development Rank 1|hzwse7: Rank (Obsolete) 15076 Severity