[LU-4600] Test failure on test suite conf-sanity, subtest test_50h "some OSC imports are still not connected"

• Type: Bug
• Resolution: Cannot Reproduce
• Priority: Critical
• Affects Version/s: Lustre 2.7.0, Lustre 2.5.3, Lustre 2.9.0, Lustre 2.10.0
• Severity: 3
• Assignee: Alex Zhuravlev
• Reporter: Maloo

Description

This issue was created by maloo for wangdi <[e-mail]>

This issue relates to the following test suite run: http://maloo.whamcloud.com/test_sets/994ad81c-8fbc-11e3-92cc-52540035b04c.

The sub-test test_50h failed with the following error:

some OSC imports are still not connected

Info required for matching: conf-sanity 50h

Issue Links

• is related to: LU-9225 conf-sanity test_50g: /usr/bin/lfs setstripe failed: error on ioctl 0x4008669a: No data available
• is related to: LU-3529 create striped directory

Comments

[Minh Diep]

+1 on 2.10.x

https://testing.hpdd.intel.com/test_sets/d3cd7fc8-f01e-11e7-8c23-52540065bddc

[Andreas Dilger]

Haven't seen this in months.

[Mikhail Pershin]

https://testing.hpdd.intel.com/test_sets/c0c038c2-29df-11e7-9073-5254006e85c2 on master

[John Hammond]

+1 on master

[Steve Guminski (Inactive)]

Another on master:

https://testing.hpdd.intel.com/test_sessions/079dc3b3-ee4c-46f6-8d64-1d7fc536c744

[Shuichi Ihara (Inactive)]

+1 on master
https://testing.hpdd.intel.com/test_sessions/5f184773-253a-47ab-8546-f4d4361b7ad1

[Shuichi Ihara (Inactive)]

+1 on master https://testing.hpdd.intel.com/sub_tests/b7ede5e6-0ef5-11e7-9053-5254006e85c2

[James Nunez (Inactive)]

I’ve looked at the past 17 failures and they all take place on review-zfs-part-2. The MDS and OSS logs all have the same errors.

In a recent failure, https://testing.hpdd.intel.com/test_sets/1a98b6a8-0686-11e7-98e7-5254006e85c2, the MDS console log shows:

15:36:09:[10168.048147] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request
15:36:09:[10168.051350] LustreError: 29278:0:(osp_precreate.c:616:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5
15:36:09:[10168.055582] LustreError: 29278:0:(osp_precreate.c:1264:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5
15:36:09:[10176.020524] Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre-OST0000.osc.active='1'
15:36:09:[10176.166752] Lustre: Permanently reactivating lustre-OST0000
15:36:09:[10177.906960] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
15:36:09:[10177.917400] LustreError: 29278:0:(osp_precreate.c:914:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -22
15:36:09:[10178.926677] LustreError: 29278:0:(osp_precreate.c:914:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -22
15:36:09:[10179.936774] LustreError: 29278:0:(osp_precreate.c:914:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -22
15:36:09:[10181.949675] LustreError: 29278:0:(osp_precreate.c:914:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -22
15:36:09:[10181.954467] LustreError: 29278:0:(osp_precreate.c:914:osp_precreate_cleanup_orphans()) Skipped 1 previous similar message
15:37:05:[10185.969586] LustreError: 29278:0:(osp_precreate.c:914:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -22
15:37:06:[10185.976987] LustreError: 29278:0:(osp_precreate.c:914:osp_precreate_cleanup_orphans()) Skipped 3 previous similar messages
15:37:06:[10186.327376] LustreError: 29258:0:(lod_qos.c:1273:lod_alloc_specific()) can't lstripe objid [0x200000bd0:0x3:0x0]: have 1 want 2
15:37:06:[10186.535832] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  conf-sanity test_50h: @@@@@@ FAIL: some OSC imports are still not connected 

On the OSS, we see:

15:36:09:[10171.051192] LustreError: 18033:0:(ofd_dev.c:1688:ofd_create_hdl()) lustre-OST0000: invalid precreate request for 0x0:33, last_id 65. Likely MDS last_id corruption
15:36:09:[10172.060525] LustreError: 18033:0:(ofd_dev.c:1688:ofd_create_hdl()) lustre-OST0000: invalid precreate request for 0x0:33, last_id 65. Likely MDS last_id corruption
15:36:09:[10173.070515] LustreError: 18033:0:(ofd_dev.c:1688:ofd_create_hdl()) lustre-OST0000: invalid precreate request for 0x0:33, last_id 65. Likely MDS last_id corruption
15:36:09:[10175.080586] LustreError: 18033:0:(ofd_dev.c:1688:ofd_create_hdl()) lustre-OST0000: invalid precreate request for 0x0:33, last_id 65. Likely MDS last_id corruption
15:36:09:[10175.085663] LustreError: 18033:0:(ofd_dev.c:1688:ofd_create_hdl()) Skipped 1 previous similar message
15:36:09:[10179.096640] LustreError: 18033:0:(ofd_dev.c:1688:ofd_create_hdl()) lustre-OST0000: invalid precreate request for 0x0:33, last_id 65. Likely MDS last_id corruption
15:36:09:[10179.104152] LustreError: 18033:0:(ofd_dev.c:1688:ofd_create_hdl()) Skipped 3 previous similar messages
15:36:09:[10179.697594] Lustre: DEBUG MARKER: /usr/sbin/lctl mark  conf-sanity test_50h: @@@@@@ FAIL: some OSC imports are still not connected 

[Andreas Dilger]

+1 on master: https://testing.hpdd.intel.com/test_sets/657e0ec0-ee71-11e6-b34d-5254006e85c2

[nasf (Inactive)]

+1 on master:
https://testing.hpdd.intel.com/test_sets/40074146-ea7c-11e6-be3b-5254006e85c2