https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9248 Lustre <p>This issue was created by maloo for Joe Gmitter &lt;joseph.gmitter@intel.com&gt;</p> <p>conf-sanity: test_55 fails with lov_objid size has to be 8192, not 8192</p> <p>Starting client: trevis-33vm1.trevis.hpdd.intel.com: -o user_xattr,flock trevis-33vm7@tcp:/lustre /mnt/lustre<br/> CMD: trevis-33vm1.trevis.hpdd.intel.com mkdir -p /mnt/lustre<br/> CMD: trevis-33vm1.trevis.hpdd.intel.com mount -t lustre -o user_xattr,flock trevis-33vm7@tcp:/lustre /mnt/lustre<br/> checking size of lov_objid for ost index 1023<br/> CMD: trevis-33vm7 debugfs -R 'stat lov_objid' /dev/lvm-Role_MDS/P1 2&gt;/dev/null<br/> conf-sanity test_55: @@@@@@ FAIL: lov_objid size has to be 8192, not 8192 </p> <p>This issue relates to the following test suite run: <a href="https://testing.hpdd.intel.com/test_sets/3e251b4e-0f04-11e7-9053-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/3e251b4e-0f04-11e7-9053-5254006e85c2</a>.</p> LU-9248

conf-sanity: test_55 fails with lov_objid size has to be 8192, not 8192

Bug Minor Open Unresolved WC Triage Maloo tests Thu, 23 Mar 2017 17:18:04 +0000 Tue, 19 Mar 2019 18:00:30 +0000 Lustre 2.7.0 Lustre 2.9.0 Lustre 2.10.0 Lustre 2.10.4 0 5 <p>2.9.57, b3575:<br/> <a href="https://testing.hpdd.intel.com/test_sessions/0800ff00-8d4c-4627-878e-566e8a697c01" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/0800ff00-8d4c-4627-878e-566e8a697c01</a></p> <p>This looks like some kind of bash comparison bug in the test that could be easily fixed:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> LOV_OBJID_SIZE=$(do_facet mds1 "$DEBUGFS -R 'stat lov_objid' $mdsdev 2&gt;/dev/null" | grep ^User | awk -F 'Size: ' '{print $2}') if [ "$LOV_OBJID_SIZE" != $(lov_objid_size $i) ]; then error "lov_objid size has to be $(lov_objid_size $i), not $LOV_OBJID_SIZE" </pre> </div></div> <p>It might be that the quoted <tt>"$LOV_OBJID_SIZE"</tt> doesn't compare nicely with the unquoted <tt>$(lov_objid_size $i)</tt>? Try removing the double quotes, and using <tt>if [[ ... ]]</tt> instead? It would also be useful to add single quotes around the values in the error message, in case there are spaces around the values (which would also cause problems with the quoted value).</p> <p>We haven’t see conf-sanity test 55 fail with</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lov_objid size has to be 8192, not 8192 </pre> </div></div> <p>for over a year. I’ve gone back to January of 2017 and don’t see this error message for this test.</p> <p>What we do see frequently is the error</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lov_objid size has to be 8192, not 0 </pre> </div></div> <p>which is not the same issue. We see this error message only during interop testing; when a client with version 2.9.56 (actually 2.9.55.36) or earlier runs against a server with version 2.9.57 (actually ~2.9.55.38) or later. For example, we see this failure with the following Lustre client/server conbinations:</p> <p>2.9.0 clients and 2.11.50.52 servers<br/> 2.9.0 clients and 2.10.52.75 servers<br/> 2.5.5-RC2 clients and  2.10.52.97 servers<br/> 2.7.3 (2_7_fe) clients and 2.9.55.41 servers<br/> 2.9.0 clients and 2.9.56.11 servers</p> <p>The issue with interop testing is that the patch for <a href="https://jira.whamcloud.com/browse/LU-4017" title="Add project quota support feature" class="issue-link" data-issue-key="LU-4017"><del>LU-4017</del></a>, commit 91fbc94f3eabe9a, changed the following in conf-sanity test 55</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">                echo checking size of lov_objid <span class="code-keyword">for</span> ost index $i -               LOV_OBJID_SIZE=$(do_facet mds1 <span class="code-quote">"$DEBUGFS -R <span class="code-quote">'stat lov_objid'</span> $mdsdev 2&gt;/dev/<span class="code-keyword">null</span>"</span> | grep ^User | awk <span class="code-quote">'\{print $6}'</span>) +               LOV_OBJID_SIZE=$(do_facet mds1 <span class="code-quote">"$DEBUGFS -R <span class="code-quote">'stat lov_objid'</span> $mdsdev 2&gt;/dev/<span class="code-keyword">null</span>"</span> | +                                grep ^User | awk -F <span class="code-quote">'Size: '</span> <span class="code-quote">'\{print $2}'</span>)                 <span class="code-keyword">if</span> [ <span class="code-quote">"$LOV_OBJID_SIZE"</span> != $(lov_objid_size $i) ]; then                         error <span class="code-quote">"lov_objid size has to be $(lov_objid_size $i), not $LOV_OBJID_SIZE"</span>                 <span class="code-keyword">else</span> </pre> </div></div> <p>Looking at a master, 2.11.50, MDS, on a running system, we see</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># debugfs -R 'stat lov_objid' /dev/vda3 | grep ^User debugfs 1.42.13.wc6 (05-Feb-2017) User:     0   Group:     0   Project:     0   Size: 32 </pre> </div></div> <p>Using the “old”, pre 2.9.56 grep/awk commands printing $6, we get</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># debugfs -R 'stat lov_objid' /dev/vda3 | grep ^User | awk '\{print $6}' debugfs 1.42.13.wc6 (05-Feb-2017) 0 </pre> </div></div> <p>which explains the output we see with interop testing.</p> <p>Thus, if we want to "fix" this issue, we would need to change what parameter is printed based on the server version number for all client from 2.9.0 and before which seems unlikely. </p> <p> </p> <p>It would be better to do something like:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>do_facet mds1 "$DEBUGFS -R 'stat lov_objid' $mdsdev 2&gt;/dev/null" | grep "^User" | sed -e 's/.*Size: //' -e 's/ [A-Z].*//') </pre> </div></div> <p>That will drop everything before and including "{{Size: }}", and then (just in case this changes again in the future) drop anything after the actual size. That should work with both old and new debugfs output, and be flexible in the future. We might even consider to replace the use of "^User" with "Size: " so that it is triggered on the actual data that we want rather than an unrelated value that just happens to exist on the same line.</p> <p>+1 on 2.10.3 <a href="https://testing.hpdd.intel.com/test_sets/61ca2062-5067-11e8-abc3-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/61ca2062-5067-11e8-abc3-52540065bddc</a></p> Development Rank 1|hzz85b: Rank (Obsolete) 9223372036854775807 Severity