https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4433 Lustre <p>While running mds-survey in DNE mode, test 1 failed as follows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== mds-survey test 1: Metadata survey with zero-stripe == 18:52:57 (1388717577) CMD: client-7vm3 lctl dl + file_count=102962 thrlo=1 thrhi=8 dir_count=4 layer=mdd stripe_count=0 rslt_loc=/tmp targets="client-7vm3:lustre-MDT0000 lustre-MDT0001" /usr/bin/mds-survey Warning: Permanently added 'client-7vm3,10.10.4.236' (RSA) to the list of known hosts. OST lustre-MDT0001 not setup procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 3 0 0 1279000 126456 355052 0 0 6 25 490 332 1 6 90 2 1 mds-survey test_1: @@@@@@ FAIL: mds-survey failed </pre> </div></div> <p>Maloo report: <a href="https://maloo.whamcloud.com/test_sets/64322640-749a-11e3-8b21-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/64322640-749a-11e3-8b21-52540035b04c</a></p> <br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_5/5/">http://build.whamcloud.com/job/lustre-b2_5/5/</a><br/> Distro/Arch: RHEL6.4/x86_64<br/> MDSCOUNT=2<br/> LU-4433

mds-survey test_1: FAIL: OST lustre-MDT0001 not setup

Bug Major Resolved Fixed nasf Jian Yu dne Sat, 4 Jan 2014 15:17:29 +0000 Mon, 20 Nov 2017 23:19:47 +0000 Mon, 22 Aug 2016 15:39:11 +0000 Lustre 2.5.0 Lustre 2.6.0 Lustre 2.5.1 Lustre 2.5.3 Lustre 2.9.0 Lustre 2.9.0 0 12 <p>In DNE mode, mds-survey test 2 failed as follows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== mds-survey test 2: Metadata survey with stripe_count = 1 == 18:53:03 (1388717583) CMD: client-7vm3 lctl dl + file_count=102962 thrlo=1 thrhi=8 dir_count=4 layer=mdd stripe_count=1 rslt_loc=/tmp targets="client-7vm3:lustre-MDT0000 lustre-MDT0001" /usr/bin/mds-survey Need obdfilter to test stripe_count cat: /tmp/mds_survey*: No such file or directory mds-survey test_2: @@@@@@ FAIL: mds-survey failed </pre> </div></div> <p>Maloo report: <a href="https://maloo.whamcloud.com/test_sets/64322640-749a-11e3-8b21-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/64322640-749a-11e3-8b21-52540035b04c</a></p> <p>Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_5/39/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_5/39/</a> (2.5.1 RC1) <br/> MDSCOUNT=2</p> <p>The same failure occurred:<br/> <a href="https://maloo.whamcloud.com/test_sets/8aea0bde-a557-11e3-a61d-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/8aea0bde-a557-11e3-a61d-52540035b04c</a></p> <p>Hit this in b2_6-rc2 DNE testing </p> <p>client and server: lustre-b2_6-rc2 1 MDS with 2 MDTs</p> <p><a href="https://testing.hpdd.intel.com/test_sets/b99b1d4e-14ef-11e4-bb6a-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/b99b1d4e-14ef-11e4-bb6a-5254006e85c2</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== mds-survey test 1: Metadata survey with zero-stripe == 22:17:04 (1406351824) CMD: onyx-46vm7 lctl dl + file_count=102960 thrlo=1 thrhi=8 dir_count=4 layer=mdd stripe_count=0 rslt_loc=/tmp targets="onyx-46vm7:lustre-MDT0000 lustre-MDT0001" /usr/bin/mds-survey Warning: Permanently added 'onyx-46vm7' (RSA) to the list of known hosts. OST lustre-MDT0001 not setup procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 2 0 0 1386232 136280 238416 0 0 14 114 1219 687 1 10 85 4 0 mds-survey test_1: @@@@@@ FAIL: mds-survey failed </pre> </div></div> <p>Lustre Build: <a href="https://build.hpdd.intel.com/job/lustre-b2_5/86/" class="external-link" target="_blank" rel="nofollow noopener">https://build.hpdd.intel.com/job/lustre-b2_5/86/</a> (2.5.3 RC1)</p> <p>The same failure occurred: <a href="https://testing.hpdd.intel.com/test_sets/3a8c8fc2-308a-11e4-a3d9-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/3a8c8fc2-308a-11e4-a3d9-5254006e85c2</a></p> <p>same issue hit on master/#3324 DNE mode<br/> <a href="https://testing.hpdd.intel.com/test_sets/6cf67e46-e7db-11e5-afa2-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/6cf67e46-e7db-11e5-afa2-5254006e85c2</a></p> <p>Hi Di,</p> <p>Does /usr/bin/mds-survey support multiple MDT targets?</p> <p>While running the following command on master branch:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>file_count=117549 thrlo=1 thrhi=8 dir_count=4 layer=mdd stripe_count=0 rslt_loc=/tmp targets="eagle-36vm6:lustre-MDT0000 eagle-36vm6:lustre-MDT0002 eagle-37vm5:lustre-MDT0001 eagle-37vm5:lustre-MDT0003" /usr/bin/mds-survey </pre> </div></div> <p>I hit the following errors:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Mon Apr 11 01:28:35 UTC 2016 /usr/bin/mds-survey from eagle-30vm6 error: test_mkdir: Object is remote ERROR: fail test_mkdir created directories on eagle-36vm6:lustre-MDT0002_ecc failed program exited with error =======&gt; Create 4 directories on eagle-36vm6:lustre-MDT0000_ecc =======&gt; Create 4 directories on eagle-36vm6:lustre-MDT0002_ecc error: test_mkdir: Object is remote Mon Apr 11 01:28:35 UTC 2016 /usr/bin/mds-survey from eagle-30vm6 created directories on eagle-36vm6:lustre-MDT0002_ecc failed </pre> </div></div> <p>Hmm, if I remember correctly, b2_5 should support mds_survey on multiple MDT. Master definitely does. Maybe there is a bug in 2_5.</p> <p>Hi Di,<br/> The above command was run on master branch. It failed.</p> <p>Jian Yu (jian.yu@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/19437" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/19437</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-4433" title="mds-survey test_1: FAIL: OST lustre-MDT0001 not setup" class="issue-link" data-issue-key="LU-4433"><del>LU-4433</del></a> tests: fix mds-survey.sh to support multiple MDTs<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 2589497e3b2ea3b1d09e35742a12cf54e79ceeb7</p> <p>Hi Di,</p> <p>The above patch just fixes mds-survey.sh test script to support multiple MDTs. But for mds-survey, it still failed with multiple MDTs.</p> <p>Could you please re-run with debug-level = -1? thanks.</p> <p>Hi Di,</p> <p>Here is the report with debug=-1 and debug_mb=150:<br/> <a href="https://testing.hpdd.intel.com/test_sets/9bb65320-011a-11e6-9ccf-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/9bb65320-011a-11e6-9ccf-5254006e85c2</a></p> <p>According to the debug log, creation fails in lod_declare_object_create()</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000004:00000001:0.0:1460512310.071969:0:15039:0:(lod_object.c:3470:lod_declare_object_create()) Process leaving via out (rc=18446744073709551550 : -66 : 0xffffffffffffffbe) 00000004:00000001:0.0:1460512310.071972:0:15039:0:(lod_object.c:3508:lod_declare_object_create()) Process leaving (rc=18446744073709551550 : -66 : ffffffffffffffbe) 00000004:00000001:0.0:1460512310.071974:0:15039:0:(mdd_object.c:374:mdd_declare_object_create_internal()) Process leaving (rc=18446744073709551550 : -66 : ffffffffffffffbe) </pre> </div></div> <p>So in lod_declare_object_create()</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>static int lod_declare_object_create(const struct lu_env *env, struct dt_object *dt, struct lu_attr *attr, struct dt_allocation_hint *hint, struct dt_object_format *dof, struct thandle *th) { ........... /* If the parent has default stripeEA, and client * did not find it before sending create request, * then MDT will return -EREMOTE, and client will * retrieve the default stripeEA and re-create the * sub directory. * * Note: if dah_eadata != NULL, it means creating the * striped directory with specified stripeEA, then it * should ignore the default stripeEA */ if (hint != NULL &amp;&amp; hint-&gt;dah_eadata == NULL) { if (OBD_FAIL_CHECK(OBD_FAIL_MDS_STALE_DIR_LAYOUT)) GOTO(out, rc = -EREMOTE); ........ } </pre> </div></div> <p>If the create request does not have EA, and also the parent does not have default stripe EA, then the new created child has to be in the same MDT with its parent, otherwise it will return -EREMOTE, see <a href="https://jira.whamcloud.com/browse/LU-6341" title="Do not check security when accessing default stripeEA" class="issue-link" data-issue-key="LU-6341"><del>LU-6341</del></a> (<a href="http://review.whamcloud.com/13990" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13990</a>). Unfortunately, this did not consider echo client. </p> <p>So there are two options to fix the problem</p> <p>1. If echo client want to create a remote directory, then add lum into the creation spec, see echo_create_md_object(). <br/> 2. Or add some flags in spec and hint, so let lod_declare_object_create() skip the this check for echo client.</p> <p>Either way is fine to me. </p> <p>Reassigning to Lai</p> <p>Take it as Peter suggested.</p> <p>Updated the patch <a href="http://review.whamcloud.com/#/c/19437/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/19437/</a>. Hope it can pass Maloo test.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/19437/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/19437/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-4433" title="mds-survey test_1: FAIL: OST lustre-MDT0001 not setup" class="issue-link" data-issue-key="LU-4433"><del>LU-4433</del></a> tests: fix mds-survey.sh to support multiple MDTs<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 530b16b187de9a27723205d9d759f260bfd350b8</p> <p>The patch has been landed to master.</p> <p>James Simmons (uja.ornl@yahoo.com) uploaded a new patch: <a href="http://review.whamcloud.com/23146" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/23146</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-4433" title="mds-survey test_1: FAIL: OST lustre-MDT0001 not setup" class="issue-link" data-issue-key="LU-4433"><del>LU-4433</del></a> obd: use ktime_t for calculating elapsed time<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: c52ac644ab7bf1d3d9fe3fac5918ca2d2d584b80</p> <p>Sorry mistyped</p> Duplicate Related LU-6341 Development Rank 1|hzwc6v: Rank (Obsolete) 12177 Severity