https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1957 Lustre <p>This issue was created by maloo for Li Wei &lt;liwei@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://maloo.whamcloud.com/test_sets/f1cbcf24-fe85-11e1-b4cd-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/f1cbcf24-fe85-11e1-b4cd-52540035b04c</a>.</p> <p>The sub-test test_180b failed with the following error:</p> <blockquote> <p>test_180b failed with 1</p></blockquote> <p>From the test output:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== sanity test 180b: test obdecho directly on obdfilter == 03:16:00 (1347617760) CMD: client-26vm4 lsmod | grep -q obdecho || { insmod /usr/lib64/lustre/obdecho/obdecho.ko || modprobe obdecho; } CMD: client-26vm4 /usr/sbin/lctl dl CMD: client-26vm4 /usr/sbin/lctl attach echo_client ec ec_uuid CMD: client-26vm4 /usr/sbin/lctl --device ec setup lustre-OST0000 CMD: client-26vm4 /usr/sbin/lctl --device ec create 1 client-26vm4: error: create: #1 - Operation not supported New object id is CMD: client-26vm4 /usr/sbin/lctl --device ec cleanup CMD: client-26vm4 /usr/sbin/lctl --device ec detach obecho_create_test failed: 3 CMD: client-26vm4 rmmod obdecho sanity test_180b: @@@@@@ FAIL: test_180b failed with 1 </pre> </div></div> <p>This was b2_3 with OFD and ZFS OSTs.</p> <p>Info required for matching: sanity 180b</p> LU-1957

Test failure on test suite sanity, subtest test_180b

Bug Blocker Resolved Fixed WC Triage Maloo ldiskfs Sun, 16 Sep 2012 22:40:36 +0000 Fri, 3 Jan 2020 23:53:24 +0000 Wed, 18 Sep 2019 21:41:06 +0000 Lustre 2.3.0 Lustre 2.4.0 Lustre 2.13.0 Lustre 2.12.4 0 7 <p><a href="https://maloo.whamcloud.com/test_sets/29df0cfe-fedc-11e1-b4cd-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/29df0cfe-fedc-11e1-b4cd-52540035b04c</a></p> <p>This was b2_3 with OFD and ZFS OSTs.</p> <p>Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_3/19" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_3/19</a><br/> USE_OFD=yes<br/> OSTFSTYPE=zfs<br/> LOAD_MODULES_REMOTE=true<br/> <a href="https://maloo.whamcloud.com/test_sets/ff130926-0241-11e2-ab94-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/ff130926-0241-11e2-ab94-52540035b04c</a></p> <p><a href="https://maloo.whamcloud.com/test_sets/cbb8e36a-0490-11e2-bfd4-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/cbb8e36a-0490-11e2-bfd4-52540035b04c</a></p> <p>This was master with OFD and LDiskFS.</p> <p>The zfs portion of this bug is possibly handled by <a href="https://jira.whamcloud.com/browse/LU-2803" title="sanity.sh test_180 fails with zfs" class="issue-link" data-issue-key="LU-2803"><del>LU-2803</del></a></p> <p>So from looking at maloo test 180b is still failing. An ldiskfs can be seen as <a href="https://maloo.whamcloud.com/sub_tests/33b45e16-7a59-11e2-b916-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/sub_tests/33b45e16-7a59-11e2-b916-52540035b04c</a> (it failed 4 times in the last 4 weeked on the ldiskfs patch review jobs)</p> <p>For both zfs and ldiskfs it seems there is some fun error like</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>CMD: wtm-19vm4 lsmod | grep -q obdecho || { insmod /usr/lib64/lustre/obdecho/obdecho.ko || modprobe obdecho; } wtm-19vm4: insmod: can't read '/usr/lib64/lustre/obdecho/obdecho.ko': No such file or directory </pre> </div></div> <p>There is no module?</p> <p>Nope there is a module is just seems to not want to load on the OST</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>12:11:05:Lustre: DEBUG MARKER: /usr/sbin/lctl --device ec create 1 12:11:05:LustreError: 21642:0:(ofd_obd.c:1191:ofd_create()) lustre-OST0000: Can't find FID Sequence 0x2: rc = -22 12:11:05:LustreError: 21642:0:(echo_client.c:2306:echo_create_object()) Cannot create objects: rc = -22 12:11:06:LustreError: 21642:0:(echo_client.c:2330:echo_create_object()) create object failed with: rc = -22 </pre> </div></div> <p>Ok current update. The "insmod: can't read " thing is not part of the issue. It is just the way the test is written. It seem obdecho can be in /usr/lib64/lustre/obdecho/ sometimes and sometimes it is with the kernel. It is not quite clear to me yet of the module from /usr/lib64/lustre/obdecho/ would be different then the one that modprobe returns. </p> <p>In any case in the main error path is the one seen above where "/usr/sbin/lctl --device ec create 1 " fails to find the FID Sequence. This is the real error. </p> <p>I am running endless testing to recreate the FID sequence error. I will submit an autotest job tomorrow I am unable to re-pro overnight. I would say it is a rare error at this point in time. </p> <p>So far 11k Iterations of the test without a repro. </p> <p>It seems <a href="http://review.whamcloud.com/5307" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5307</a> was landed on Feb 14th. The last know ldiskfs error was Feb 18th so I doubt the error has been encountered since the patch landed. </p> <p>Patch 5307 is "<a href="https://jira.whamcloud.com/browse/LU-2775" title="Interop 2.1.4&lt;-&gt;2.4 failure on test suite lustre-initialization-1: ASSERTION( fid_seq_is_mdt(loi-&gt;loi_oi.oi_seq) ) failed" class="issue-link" data-issue-key="LU-2775"><del>LU-2775</del></a> osp: enable fid-on-OST only for DNE." Basically now we only use use FID for DNE, and autotest runs should not be using FID sequence on the OST. </p> <p>I am inclined to say this issue has been fixed. I have emailed Wang Di.</p> <p>Yes, with patch 5307, Normal FID will be only used when DNE is enabled. But I do not understand why this can fix this problem, echo client should always use seq 2, no matter OST FID is enabled or not. Probably I miss sth here. The debug log from those failure links are so less. Keith, Do you have debug log you can post here? Thanks. </p> <p>It seems the error has not occurred for over two weeks the last know error was Feb18th. I do not have any local debug logs as I have not been only the autotests logs. </p> <p>Below is the OST debug log from the Feb 18th run. <br/> <a href="https://maloo.whamcloud.com/test_logs/1dcc313a-7a5b-11e2-b916-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_logs/1dcc313a-7a5b-11e2-b916-52540035b04c</a></p> <p>In general I don't see anytime more interesting than </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000100:00100000:0.0:1360080302.580440:0:13196:0:(service.c:1976:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_002:lustre-MDT0000-mdtlov_UUID+5:3022:x1426144806071293:12345-10.10.16.242@tcp:400 00000100:00100000:0.0:1360080302.580447:0:13196:0:(service.c:2020:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_002:lustre-MDT0000-mdtlov_UUID+5:3022:x1426144806071293:12345-10.10.16.242@tcp:400 Request procesed in 7us (156us total) trans 0 rc 0/0 00000100:00100000:0.0:1360080302.580448:0:13196:0:(nrs_fifo.c:245:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.10.16.242@tcp, seq: 251 00000100:00100000:0.0:1360080302.580450:0:13196:0:(nrs_fifo.c:223:nrs_fifo_req_start()) NRS start fifo request from 12345-10.10.16.242@tcp, seq: 252 00000100:00100000:0.0:1360080302.580451:0:13196:0:(service.c:1976:ptlrpc_server_handle_request()) Handling RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_002:lustre-MDT0000-mdtlov_UUID+5:3022:x1426144806071295:12345-10.10.16.242@tcp:400 00000100:00100000:0.0:1360080302.580456:0:13196:0:(service.c:2020:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc ll_ost00_002:lustre-MDT0000-mdtlov_UUID+5:3022:x1426144806071295:12345-10.10.16.242@tcp:400 Request procesed in 5us (161us total) trans 0 rc 0/0 00000100:00100000:0.0:1360080302.580458:0:13196:0:(nrs_fifo.c:245:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.10.16.242@tcp, seq: 252 00002000:00100000:0.0:1360080302.620394:0:21137:0:(ofd_obd.c:135:ofd_parse_connect_data()) lustre-OST0000: cli ECHO_UUID/ffff88006041b800 ocd_connect_flags: 0x405000000068 ocd_version: 2033c00 ocd_grant: 0 ocd_index: 0 ocd_group 2 00002000:00100000:0.0:1360080302.620403:0:21137:0:(ofd_obd.c:234:ofd_parse_connect_data()) lustre-OST0000: cli (no nid) does not support OBD_CONNECT_CKSUM, CRC32 will be used 00002000:00080000:0.0:1360080302.620456:0:21137:0:(ofd_obd.c:317:ofd_obd_connect()) lustre-OST0000: get connection from MDS 2 00000001:02000400:0.0:1360080302.715088:0:21160:0:(debug.c:445:libcfs_debug_mark_buffer()) DEBUG MARKER: /usr/sbin/lctl --device ec create 1 00002000:00020000:0.0:1360080302.803633:0:21184:0:(ofd_obd.c:1178:ofd_create()) lustre-OST0000: Can't find oseq 0x2: -22 00008000:00020000:0.0:1360080302.803636:0:21184:0:(echo_client.c:2300:echo_create_object()) Cannot create objects: rc = -22 00008000:00020000:0.0:1360080302.804450:0:21184:0:(echo_client.c:2324:echo_create_object()) create object failed with: rc = -22 00000001:02000400:0.0:1360080302.902789:0:21207:0:(debug.c:445:libcfs_debug_mark_buffer()) DEBUG MARKER: /usr/sbin/lctl --device ec cleanup 00000100:00100000:0.0:1360080302.936335:0:5561:0:(client.c:1418:ptlrpc_send_new_req()) Sending RPC pname:cluuid:pid:xid:nid:opc ptlrpcd_0:29659cc7-7815-f2e4-6cf2-2103848e55b6:5561:1426144820745791:10.10.16.242@tcp:400 </pre> </div></div> <p>in the logs. </p> <p>In my maloo search, it does appear that sanity.sh test_180b is failing several times a day:</p> <p><a href="https://maloo.whamcloud.com/sub_tests/query?utf8=%E2%9C%93&amp;test_set[test_set_script_id]=f9516376-32bc-11e0-aaee-52540025f9ae&amp;sub_test[sub_test_script_id]=14b0513a-32be-11e0-b685-52540025f9ae&amp;sub_test[status]=FAIL&amp;sub_test[query_bugs]=&amp;test_session[test_host]=&amp;test_session[test_group]=&amp;test_session[user_id]=&amp;test_session[query_date]=&amp;test_session[query_recent_period]=2419200&amp;test_node[os_type_id]=&amp;test_node[distribution_type_id]=&amp;test_node[architecture_type_id]=&amp;test_node[file_system_type_id]=&amp;test_node[lustre_branch_id]=&amp;test_node_network[network_type_id]=&amp;commit=Update+results" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/sub_tests/query?utf8=%E2%9C%93&amp;test_set[test_set_script_id]=f9516376-32bc-11e0-aaee-52540025f9ae&amp;sub_test[sub_test_script_id]=14b0513a-32be-11e0-b685-52540025f9ae&amp;sub_test[status]=FAIL&amp;sub_test[query_bugs]=&amp;test_session[test_host]=&amp;test_session[test_group]=&amp;test_session[user_id]=&amp;test_session[query_date]=&amp;test_session[query_recent_period]=2419200&amp;test_node[os_type_id]=&amp;test_node[distribution_type_id]=&amp;test_node[architecture_type_id]=&amp;test_node[file_system_type_id]=&amp;test_node[lustre_branch_id]=&amp;test_node_network[network_type_id]=&amp;commit=Update+results</a></p> <p>The most recent failures are at:<br/> <a href="https://maloo.whamcloud.com/sub_tests/0b93da12-833a-11e2-85c9-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/sub_tests/0b93da12-833a-11e2-85c9-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/sub_tests/e26a1e3c-8242-11e2-ba47-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/sub_tests/e26a1e3c-8242-11e2-ba47-52540035b04c</a></p> <p>Looks like all of the failures on ZFS.</p> <p>Correct fails ZFS alot. There has not been an Ldiskfs Failure since Feb 18th. </p> <p><a href="https://jira.whamcloud.com/browse/LU-2803" title="sanity.sh test_180 fails with zfs" class="issue-link" data-issue-key="LU-2803"><del>LU-2803</del></a> sanity/180 fail with zfs : is a separate LU that tracks the zfs issue. Alex has a Patch out for the issue. I have taken this LU to mean Ldiskfs. </p> <p>Are we all ok to bring this down out of blocker state? </p> <p>Perhaps a close as unreproducible for ldiskfs? </p> <p>Fixed for ldiskfs, use <a href="https://jira.whamcloud.com/browse/LU-2803" title="sanity.sh test_180 fails with zfs" class="issue-link" data-issue-key="LU-2803"><del>LU-2803</del></a> for the current ZFS failures. </p> <p>sanity.sh test_180 is still being skipped on ZFS filesystems due to this issue. If it was fixed by <a href="https://jira.whamcloud.com/browse/LU-2803" title="sanity.sh test_180 fails with zfs" class="issue-link" data-issue-key="LU-2803"><del>LU-2803</del></a> then a patch should be submitted to re-enable the test.</p> <p>Really old blocker for unsupported version</p> <p>Reopen to clear flags.</p> <p>We still always skip test_180 for ZFS targets due to <tt>ALWAYS_EXCEPT</tt>. A patch should be submitted to remove the subtests from <tt>ALWAYS_EXCEPT</tt> on the assumption that <a href="https://jira.whamcloud.com/browse/LU-2803" title="sanity.sh test_180 fails with zfs" class="issue-link" data-issue-key="LU-2803"><del>LU-2803</del></a> fixed that problem.</p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/35930" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35930</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-1957" title="Test failure on test suite sanity, subtest test_180b" class="issue-link" data-issue-key="LU-1957"><del>LU-1957</del></a> tests: remove sanity test 180 from ALWAYS_EXCEPT<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 2b0119932f5abfd85f01ffac5614f41d5b9fe559</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/35930/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35930/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-1957" title="Test failure on test suite sanity, subtest test_180b" class="issue-link" data-issue-key="LU-1957"><del>LU-1957</del></a> tests: remove sanity test 180 from ALWAYS_EXCEPT<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 72b59b85a253e508ec1b192fbf8cad840ca6ff2c</p> <p>Bug was fixed in 2.4.0, test enabled in 2.13.0.</p> <p>James Nunez (jnunez@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/36930" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36930</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-1957" title="Test failure on test suite sanity, subtest test_180b" class="issue-link" data-issue-key="LU-1957"><del>LU-1957</del></a> tests: remove sanity test 180 from ALWAYS_EXCEPT<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: 3e47ed64ed481d59140fd74ff92d9f774d0e39da</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/36930/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36930/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-1957" title="Test failure on test suite sanity, subtest test_180b" class="issue-link" data-issue-key="LU-1957"><del>LU-1957</del></a> tests: remove sanity test 180 from ALWAYS_EXCEPT<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: c3d53269c5133e938b90f0f0488cddf29c35701b</p> Related LU-2803 Development Rank 1|hzv3in: Rank (Obsolete) 4069 Severity