https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4600 Lustre <p>This issue was created by maloo for wangdi &lt;di.wang@intel.com&gt;</p> <p>This issue relates to the following test suite run: <a href="http://maloo.whamcloud.com/test_sets/994ad81c-8fbc-11e3-92cc-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">http://maloo.whamcloud.com/test_sets/994ad81c-8fbc-11e3-92cc-52540035b04c</a>.</p> <p>The sub-test test_50h failed with the following error:</p> <blockquote> <p>some OSC imports are still not connected</p></blockquote> <p>Info required for matching: conf-sanity 50h</p> LU-4600

Test failure on test suite conf-sanity, subtest test_50h "some OSC imports are still not connected"

Bug Critical Closed Cannot Reproduce Alex Zhuravlev Maloo Fri, 7 Feb 2014 21:37:38 +0000 Thu, 16 Apr 2020 08:17:27 +0000 Thu, 16 Apr 2020 08:17:27 +0000 Lustre 2.7.0 Lustre 2.5.3 Lustre 2.9.0 Lustre 2.10.0 0 14 <p>It looks like this problem has only been hit once with the patch 7196 applied. Until seen elsewhere we should assume this problem is caused by the patch. </p> <p>seen in b2_4<br/> <a href="https://maloo.whamcloud.com/test_sets/450f675e-b8b6-11e3-bc82-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/450f675e-b8b6-11e3-bc82-52540035b04c</a></p> <p>Seen on master (2.6-pre): <a href="https://maloo.whamcloud.com/test_sets/3cd0ba26-e70e-11e3-badc-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/3cd0ba26-e70e-11e3-badc-52540035b04c</a></p> <p>I hit this on master (2.7): <a href="https://testing.hpdd.intel.com/test_sets/2046baf4-11e3-11e4-90ac-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/2046baf4-11e3-11e4-90ac-5254006e85c2</a></p> <p>In the client log, I see </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>open(/mnt/lustre/d50h.conf-sanity/2/f50h.conf-sanity-0) error: File too large </pre> </div></div> <p>Maybe related to <a href="https://jira.whamcloud.com/browse/LU-4340" title="conf-sanity test_69: error: File too large" class="issue-link" data-issue-key="LU-4340"><del>LU-4340</del></a>?</p> <p>Lustre Build: <a href="https://build.hpdd.intel.com/job/lustre-b2_5/84/" class="external-link" target="_blank" rel="nofollow noopener">https://build.hpdd.intel.com/job/lustre-b2_5/84/</a><br/> Distro/Arch: RHEL6.5/x86_64<br/> Network: o2ib</p> <p><a href="https://testing.hpdd.intel.com/test_sets/9c8fcf9a-2b62-11e4-8687-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/9c8fcf9a-2b62-11e4-8687-5254006e85c2</a></p> <p>Dmesg on MDS:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LustreError: 14985:0:(osp_precreate.c:719:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -5 Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre-OST0000.osc.active='1' Lustre: Permanently reactivating lustre-OST0000 LustreError: 14718:0:(lod_qos.c:946:lod_alloc_specific()) can't lstripe objid [0x200000bd0:0x5:0x0]: have 0 want 1 LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. Lustre: DEBUG MARKER: /usr/sbin/lctl mark conf-sanity test_50h: @@@@@@ FAIL: some OSC imports are still not connected </pre> </div></div> <p>Seen in master (2.7): <a href="https://testing.hpdd.intel.com/test_sets/218a95fe-3e0f-11e4-b06a-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/218a95fe-3e0f-11e4-b06a-5254006e85c2</a></p> <p>another seen on b2_5:<br/> <a href="https://testing.hpdd.intel.com/test_sets/2cdb6ca2-ca74-11e4-9330-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/2cdb6ca2-ca74-11e4-9330-5254006e85c2</a></p> <p><a href="https://testing.hpdd.intel.com/test_sets/786f02ae-6255-11e5-8cee-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/786f02ae-6255-11e5-8cee-5254006e85c2</a></p> <p>There are two different error messages found in the test_log for the failures listed in this ticket. </p> <p>Some of the logs listed here have the following error in the test_log:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>open(/mnt/lustre/d50h.conf-sanity/2/f50h.conf-sanity-0) error: No space left on device </pre> </div></div> <p>which may be related to <a href="https://jira.whamcloud.com/browse/LU-7309" title="replay-single test_70b: no space left on device" class="issue-link" data-issue-key="LU-7309"><del>LU-7309</del></a>.</p> <p>Logs with this failure are at:<br/> 2015-11-22 10:46:30 - <a href="https://testing.hpdd.intel.com/test_sets/7a1ec004-9134-11e5-b507-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/7a1ec004-9134-11e5-b507-5254006e85c2</a><br/> 2016-01-05 07:30:40 - <a href="https://testing.hpdd.intel.com/test_sets/33dac2da-b3aa-11e5-8114-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/33dac2da-b3aa-11e5-8114-5254006e85c2</a><br/> 2016-02-01 12:07:07 - <a href="https://testing.hpdd.intel.com/test_sets/9a834e80-c908-11e5-aaa9-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/9a834e80-c908-11e5-aaa9-5254006e85c2</a></p> <p>Others test_logs mentioned in this ticket have the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>open(/mnt/lustre/d50h.conf-sanity/2/f50h.conf-sanity-0) error: File too large </pre> </div></div> <p>another on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/9fdfff5e-0751-11e6-9e5d-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/9fdfff5e-0751-11e6-9e5d-5254006e85c2</a></p> <p>another on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/245c2112-de9e-11e6-a08b-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/245c2112-de9e-11e6-a08b-5254006e85c2</a></p> <p>another on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/d599a2c8-e2f0-11e6-84f4-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/d599a2c8-e2f0-11e6-84f4-5254006e85c2</a></p> <p>+1 on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/40074146-ea7c-11e6-be3b-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/40074146-ea7c-11e6-be3b-5254006e85c2</a></p> <p>+1 on master: <a href="https://testing.hpdd.intel.com/test_sets/657e0ec0-ee71-11e6-b34d-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/657e0ec0-ee71-11e6-b34d-5254006e85c2</a></p> <p>I’ve looked at the past 17 failures and they all take place on review-zfs-part-2. The MDS and OSS logs all have the same errors.</p> <p>In a recent failure, <a href="https://testing.hpdd.intel.com/test_sets/1a98b6a8-0686-11e7-98e7-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/1a98b6a8-0686-11e7-98e7-5254006e85c2</a>, the MDS console log shows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>15:36:09:[10168.048147] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request 15:36:09:[10168.051350] LustreError: 29278:0:(osp_precreate.c:616:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 15:36:09:[10168.055582] LustreError: 29278:0:(osp_precreate.c:1264:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 15:36:09:[10176.020524] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre-OST0000.osc.active='1' 15:36:09:[10176.166752] Lustre: Permanently reactivating lustre-OST0000 15:36:09:[10177.906960] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. 15:36:09:[10177.917400] LustreError: 29278:0:(osp_precreate.c:914:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -22 15:36:09:[10178.926677] LustreError: 29278:0:(osp_precreate.c:914:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -22 15:36:09:[10179.936774] LustreError: 29278:0:(osp_precreate.c:914:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -22 15:36:09:[10181.949675] LustreError: 29278:0:(osp_precreate.c:914:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -22 15:36:09:[10181.954467] LustreError: 29278:0:(osp_precreate.c:914:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message 15:37:05:[10185.969586] LustreError: 29278:0:(osp_precreate.c:914:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -22 15:37:06:[10185.976987] LustreError: 29278:0:(osp_precreate.c:914:osp_precreate_cleanup_orphans()) Skipped 3 previous similar messages 15:37:06:[10186.327376] LustreError: 29258:0:(lod_qos.c:1273:lod_alloc_specific()) can't lstripe objid [0x200000bd0:0x3:0x0]: have 1 want 2 15:37:06:[10186.535832] Lustre: DEBUG MARKER: /usr/sbin/lctl mark conf-sanity test_50h: @@@@@@ FAIL: some OSC imports are still not connected </pre> </div></div> <p>On the OSS, we see:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>15:36:09:[10171.051192] LustreError: 18033:0:(ofd_dev.c:1688:ofd_create_hdl()) lustre-OST0000: invalid precreate request for 0x0:33, last_id 65. Likely MDS last_id corruption 15:36:09:[10172.060525] LustreError: 18033:0:(ofd_dev.c:1688:ofd_create_hdl()) lustre-OST0000: invalid precreate request for 0x0:33, last_id 65. Likely MDS last_id corruption 15:36:09:[10173.070515] LustreError: 18033:0:(ofd_dev.c:1688:ofd_create_hdl()) lustre-OST0000: invalid precreate request for 0x0:33, last_id 65. Likely MDS last_id corruption 15:36:09:[10175.080586] LustreError: 18033:0:(ofd_dev.c:1688:ofd_create_hdl()) lustre-OST0000: invalid precreate request for 0x0:33, last_id 65. Likely MDS last_id corruption 15:36:09:[10175.085663] LustreError: 18033:0:(ofd_dev.c:1688:ofd_create_hdl()) Skipped 1 previous similar message 15:36:09:[10179.096640] LustreError: 18033:0:(ofd_dev.c:1688:ofd_create_hdl()) lustre-OST0000: invalid precreate request for 0x0:33, last_id 65. Likely MDS last_id corruption 15:36:09:[10179.104152] LustreError: 18033:0:(ofd_dev.c:1688:ofd_create_hdl()) Skipped 3 previous similar messages 15:36:09:[10179.697594] Lustre: DEBUG MARKER: /usr/sbin/lctl mark conf-sanity test_50h: @@@@@@ FAIL: some OSC imports are still not connected </pre> </div></div> <p>+1 on master <a href="https://testing.hpdd.intel.com/sub_tests/b7ede5e6-0ef5-11e7-9053-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/b7ede5e6-0ef5-11e7-9053-5254006e85c2</a></p> <p>+1 on master<br/> <a href="https://testing.hpdd.intel.com/test_sessions/5f184773-253a-47ab-8546-f4d4361b7ad1" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/5f184773-253a-47ab-8546-f4d4361b7ad1</a></p> <p>Another on master:</p> <p><a href="https://testing.hpdd.intel.com/test_sessions/079dc3b3-ee4c-46f6-8d64-1d7fc536c744" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/079dc3b3-ee4c-46f6-8d64-1d7fc536c744</a></p> <p>+1 on master</p> <p><a href="https://testing.hpdd.intel.com/test_sets/c0c038c2-29df-11e7-9073-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/c0c038c2-29df-11e7-9073-5254006e85c2</a> on master</p> <p>Haven't seen this in months.</p> <p>+1 on 2.10.x</p> <p><a href="https://testing.hpdd.intel.com/test_sets/d3cd7fc8-f01e-11e7-8c23-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/d3cd7fc8-f01e-11e7-8c23-52540065bddc</a></p> Duplicate Related LU-3529 LU-9225 Development Rank 1|hzwem7: Rank (Obsolete) 12585 Severity