https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5375 Lustre <p>This issue was created by maloo for sarah &lt;sarah@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.hpdd.intel.com/test_sets/97672104-0dca-11e4-b3f5-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/97672104-0dca-11e4-b3f5-5254006e85c2</a>.</p> <p>The sub-test test_151 failed with the following error:</p> <blockquote> <p>roc_hit is not safe to use</p></blockquote> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== sanity test 151: test cache on oss and controls ================================= 19:31:03 (1405477863) CMD: onyx-40vm8 /usr/sbin/lctl get_param -n obdfilter.lustre-OST*.read_cache_enable osd-*.lustre-OST*.read_cache_enable 2&gt;&amp;1 CMD: onyx-40vm8 /usr/sbin/lctl get_param -n obdfilter.lustre-OST*.read_cache_enable osd-*.lustre-OST*.read_cache_enable 2&gt;&amp;1 CMD: onyx-40vm8 /usr/sbin/lctl set_param -n obdfilter.lustre-OST*.writethrough_cache_enable=1 osd-*.lustre-OST*.writethrough_cache_enable=1 2&gt;&amp;1 CMD: onyx-40vm8 /usr/sbin/lctl get_param -n obdfilter.lustre-OST*.writethrough_cache_enable osd-*.lustre-OST*.writethrough_cache_enable 2&gt;&amp;1 4+0 records in 4+0 records out 16384 bytes (16 kB) copied, 0.00947514 s, 1.7 MB/s CMD: onyx-40vm8 /usr/sbin/lctl get_param -n obdfilter.*OST*0000.stats osd-*.*OST*0000.stats 2&gt;&amp;1 CMD: onyx-40vm8 /usr/sbin/lctl get_param -n obdfilter.*OST*0000.stats osd-*.*OST*0000.stats 2&gt;&amp;1 BEFORE:11 AFTER:12 sanity test_151: @@@@@@ FAIL: roc_hit is not safe to use </pre> </div></div> client and server: lustre-b2_6-rc2 ldiskfs <br/> client is SLES11 SP3 LU-5375

Failure on test suite sanity test_151 test_156: roc_hit is not safe to use

Bug Minor Resolved Cannot Reproduce WC Triage Maloo Sat, 19 Jul 2014 00:22:45 +0000 Sun, 27 Jan 2019 21:10:56 +0000 Wed, 12 Sep 2018 00:28:57 +0000 Lustre 2.6.0 Lustre 2.8.0 0 9 <p>So this failing code was added as part of <a href="https://jira.whamcloud.com/browse/LU-2902" title="sanity test_156: NOT IN CACHE: before: , after: " class="issue-link" data-issue-key="LU-2902"><del>LU-2902</del></a> so whoever is going to look at it, you might want to look there too.</p> <p>Also seen this during rolling upgrade from 2.5 ldiskfs to 2.6. <br/> After MDS and OSS were upgraded to 2.6, both clients keep 2.5 and then run the sanity test_151 failed as the same error.</p> <p>before upgrade: 2.5.2<br/> after upgrade: b2_6-rc2</p> <p>Haven't seen this in a long time.</p> <p>hit this again in interop testing with lustre-master server(EL7) and 2.5.3 client</p> <p><a href="https://testing.hpdd.intel.com/test_sets/e006681c-1250-11e5-bec9-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/e006681c-1250-11e5-bec9-5254006e85c2</a></p> <p>This is failing between 0-5 times per day, maybe twice per day on average. It looks like most of these recent failures (excluding those attributable to <a href="https://jira.whamcloud.com/browse/LU-5030" title="&quot;lctl {get,set}_param&quot; should also check in /sys/fs/{lnet,lustre}" class="issue-link" data-issue-key="LU-5030"><del>LU-5030</del></a> breaking /proc access completely) are of the form:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>BEFORE:18720 AFTER:18721 sanity test_151: @@@@@@ FAIL: roc_hit is not safe to use </pre> </div></div> <p>so the before/after values are only off by one. I suspect this is just a problem with the test script - the <tt>roc_hit_init()</tt> function is just using <tt>cat $DIR/$tfile</tt> to read the file and with proper readahead of files smaller than <tt>max_readahead_whole</tt> it should only do a single read. So <tt>roc_hit_init()</tt> should be changed to use something like:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">if</span> (( AFTER - BEFORE == 0 || AFTER - BEFORE &gt; 4)); then rm -rf $dir error <span class="code-quote">"roc_hit is not safe to use: BEFORE=$BEFORE, AFTER=$AFTER"</span> fi </pre> </div></div> <p>The <tt>rm -rf $dir</tt> at the end of <tt>roc_hit_init()</tt> should also be changed to just use <tt>rmdir $dir</tt> since this directory should be empty at this point because $file is deleted for each loop.</p> <p>also hit this issue after rolling downgrade from master/3264 RHEL6.7 to 2.5.5 RHEL6.6</p> <p>Encountered another instance for Interop master&lt;-&gt;2.5.5<br/> Server: Master, Build# 3266, Tag 2.7.64 <br/> Client: 2.5.5, b2_5_fe/62<br/> <a href="https://testing.hpdd.intel.com/test_sets/ac332386-9fcc-11e5-a33d-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/ac332386-9fcc-11e5-a33d-5254006e85c2</a></p> <p>Server: Master, Build# 3266, Tag 2.7.64 , RHEL 7 <br/> Client: 2.5.5, b2_5_fe/62<br/> <a href="https://testing.hpdd.intel.com/test_sets/e4f27f18-9fff-11e5-a33d-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/e4f27f18-9fff-11e5-a33d-5254006e85c2</a></p> <p>I know why this test is failing. Symlinks are being created by the obdfilter into the osd-* layer for writehthrough_cache_enable, read cache_max_filesize, read_cache_enable and brw_stats. This works for ldiskfs but not ZFS. ZFS only has brw_stats but lacks the rest. This is why sanity test 151 fails for ZFS.</p> <p>James, wouldn't that cause review-zfs to fail all the time? It appears that test_151 has checks for <tt>read_cache_enable</tt> and <tt>writethrough_cache_enable</tt>, though it does "return 0" instead of "skip" as it probably should.</p> <p>The source of the failure is get_osd_param(). It should be reporting that those parameters don't exist. Its doing a 'grep -v 'Found no match' but that is not being reported by lctl get_param. Running the command manually gives:</p> <p>root@ninja11 lustre-OST0000]# lctl get_param -n obdfilter.lustre-OST0000.read_cache_enable<br/> error: list_param: obdfilter/lustre-OST0000/read_cache_enable: No such file or directory</p> <p>Ah yes I moved from the custom globerrstr() to the standard strerror(...). The failure in this case is due to the <a href="https://jira.whamcloud.com/browse/LU-5030" title="&quot;lctl {get,set}_param&quot; should also check in /sys/fs/{lnet,lustre}" class="issue-link" data-issue-key="LU-5030"><del>LU-5030</del></a> changes.</p> <p>I'm thinking the "grep -v 'Found no match'" test might not always work. I'm exploring testing the return value "$?" of the command. I like to test to see if "$?" is less than zero. Would something like this work?</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>do_nodes $nodes "$LCTL set_param -n obdfilter.$device.$name=$value \ - osd-*.$device.$name=$value 2&gt;&amp;1" | grep -v 'Found no match' + osd-*.$device.$name=$value 2&gt;&amp;1" || return [ $? -lt 0 ] </pre> </div></div> <p>Sorry not the greatest bash scripter.</p> <p>No, because bash always returns positive error numbers and not negative ones. You could check <tt>[ $? -ne 0 ]</tt> but that might as well just be <tt>return $?</tt>, which is also the default behaviour when returning from a function - to return the exit code from the last function. The other question is whether "lctl set_param" actually returns an error code on errors, or just prints a message?</p> <p>In this case, you might be better off using <tt>| egrep -v 'Found no match|no such file or directory'</tt> or similar, to ensure it works for both old and new lctl, since this will also run in interop mode with servers that do not have your patches. Is there a reason you got rid of globerrstr() and went to strerror()?</p> <p>globstrerr only handled 3 error cases. The move to cfs_get_paths() expanded the possible errors. I have a working solution now. Just pushed the patch.</p> <p>Another instance found for interop : EL6.7 Server/2.5.5 Client<br/> Server: master, build# 3303, RHEL 6.7<br/> Client: 2.5.5, b2_5_fe/62<br/> <a href="https://testing.hpdd.intel.com/test_sets/24b4b54e-bad6-11e5-9137-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/24b4b54e-bad6-11e5-9137-5254006e85c2</a></p> <p>Another instance found for interop tag 2.7.66 - EL6.7 Server/2.5.5 Client, build# 3316<br/> <a href="https://testing.hpdd.intel.com/test_sets/9ed7c1d8-cc9f-11e5-963e-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/9ed7c1d8-cc9f-11e5-963e-5254006e85c2</a></p> <p>Another instance found for interop tag 2.7.66 - EL7 Server/2.5.5 Client, build# 3316<br/> <a href="https://testing.hpdd.intel.com/test_sets/5ea975e2-cc46-11e5-901d-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/5ea975e2-cc46-11e5-901d-5254006e85c2</a></p> <p>Another instance found for interop - EL6.7 Server/2.5.5 Client, tag 2.7.90. <br/> <a href="https://testing.hpdd.intel.com/test_sessions/f99a2d60-d567-11e5-bc47-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/f99a2d60-d567-11e5-bc47-5254006e85c2</a><br/> Another instance found for interop - EL7 Server/2.5.5 Client, tag 2.7.90. <br/> <a href="https://testing.hpdd.intel.com/test_sessions/93baffee-d2ae-11e5-8697-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/93baffee-d2ae-11e5-8697-5254006e85c2</a></p> <p>Can we close this?</p> <p>Recent failures reporting this ticket were caused by <a href="https://jira.whamcloud.com/browse/LU-11347" title="Do not use pagecache for SSD I/O when read/write cache are disabled" class="issue-link" data-issue-key="LU-11347"><del>LU-11347</del></a> patch, this hasn't been hit in a long time.</p> <p>I'm seeing this bug again <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/sad.png" height="16" width="16" align="absmiddle" alt="" border="0"/> in 2.12.50 testing.</p> <p>Recent failures were triggered by <a href="https://jira.whamcloud.com/browse/LU-11607" title="Reduce repeated function calls in Lustre test suites" class="issue-link" data-issue-key="LU-11607"><del>LU-11607</del></a> patch landing, but it turns out the problem was in the original <a href="https://jira.whamcloud.com/browse/LU-2261" title="Add cache stats to zfs osd" class="issue-link" data-issue-key="LU-2261"><del>LU-2261</del></a> patch. See <a href="https://jira.whamcloud.com/browse/LU-11889" title="sanity test 156 fails on ZFS: roc_hit not safe to use" class="issue-link" data-issue-key="LU-11889">LU-11889</a> for details.</p> Related LU-11889 LU-2902 LU-11347 LU-2261 LU-11607 Development Rank 1|hzwrtr: Rank (Obsolete) 14980 Severity