[LU-4039] Failure on test suite replay-single test_90: wrong stripe: f0, uuid: lustre-OST0000_UUID

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.5.0, Lustre 2.9.0
• Fix Version/s: Lustre 2.9.0
• Type: Bug
• Priority: Critical
• Reporter: Maloo
• Assignee: Yang Sheng
• Resolution: Fixed
• Labels: zfs

Environment

server and client: lustre-master build # 1687

• Severity: 3

Description

This issue was created by maloo for sarah <[e-mail]>

This issue relates to the following test suite run: http://maloo.whamcloud.com/test_sets/f9c487da-26c9-11e3-83d1-52540035b04c.

The sub-test test_90 failed with the following error:

Create the files
/mnt/lustre/d0.replay-single/d90/f0
lmm_stripe_count: 1
lmm_stripe_size: 1048576
lmm_pattern: 1
lmm_layout_gen: 0
lmm_stripe_offset: 1
obdidx objid objid group
1 30476 0x770c 0

replay-single test_90: @@@@@@ FAIL: wrong stripe: f0, uuid: lustre-OST0000_UUID

Info required for matching: replay-single 90

Issue Links

• is related to: LU-8544 recovery-double-scale test_pairwise_f...
• is related to: LU-7759 umount hanging in modern distros when...
• is related to: LU-8526 replay-single test_90: @@@@@@ FAIL: w...

Comments

[Yang Sheng]

This issue was caused by test_89 failed OST0000 and then import still is not in FULL state while test_90 start. So stripe alloced wrong. I'll push a patch to fix it.

/proc/fs/lustre/osp/lustre-MDT0000-osp-MDT0001/state:current_state: FULL
/proc/fs/lustre/osp/lustre-MDT0001-osp-MDT0000/state:current_state: FULL
/proc/fs/lustre/osp/lustre-OST0000-osc-MDT0000/state:current_state: REPLAY_WAIT
/proc/fs/lustre/osp/lustre-OST0000-osc-MDT0001/state:current_state: REPLAY_WAIT
/proc/fs/lustre/osp/lustre-OST0001-osc-MDT0000/state:current_state: FULL
/proc/fs/lustre/osp/lustre-OST0001-osc-MDT0001/state:current_state: FULL

[Gerrit Updater]

Yang Sheng ([e-mail]) uploaded a new patch: http://review.whamcloud.com/20931
Subject: LU-4039 tests: ensure osc import in FULL state
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: d03dbc49b9ee5da93341a5d4ddc42a8915293612

[Sebastien Buisson (Inactive)]

Several recent hits on master, like:
https://testing.hpdd.intel.com/test_sets/ae363b24-4205-11e6-a0ce-5254006e85c2

[Andreas Dilger]

It appears that this test only started failing again on 2016-06-28, so it is very likely a regression caused by some patch that recently landed. It seems that the MDS is not enforcing setstripe requests that specify a starting OST index, possibly if that OST does not have any precreated objects. One candidate is patch http://review.whamcloud.com/19195 "LU-7759 llite: handle inactive OSTs better in statfs" though I can't see why it would be a problem...

[Andreas Dilger]

Looking at the patch testing history for http://review.whamcloud.com/19195 it appears that it was failing replay-single test_90 on a regular basis, except for the very last version of the patch, which was landed.

[Gerrit Updater]

Andreas Dilger ([e-mail]) merged in patch http://review.whamcloud.com/20931/
Subject: LU-4039 tests: ensure osc import in FULL state
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: eb2657a654a449f2c5504fa4c5ba4b7e1413ddea

[Yang Sheng]

Patch landed. Close ticket.

[Andreas Dilger]

It looks like this patch did not fix the problem. There were two recent patch tests that failed replay-single.sh even though they included the latest patch:

http://review.whamcloud.com/18758

• https://testing.hpdd.intel.com/test_sets/30c3291a-4320-11e6-bbf5-5254006e85c2

http://review.whamcloud.com/16105

• https://testing.hpdd.intel.com/test_sets/2cde8cee-4353-11e6-acf3-5254006e85c2

[Yang Sheng]

Looks like almost tests failed on:

replay-single test_90: @@@@@@ FAIL: wrong stripe: all, uuid: lustre-OST0000_UUID 

and test_89 is skipped. I'll push a debug patch for it.

[Gerrit Updater]

Yang Sheng ([e-mail]) uploaded a new patch: http://review.whamcloud.com/21175
Subject: LU-4039 tests: debug patch
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 927531c4e0981990861cb2a82319816e2f913fab

[James Nunez (Inactive)]

Increased priority of ticket because replay-single test 90 is failing on master multiple times a day.

[Jian Yu]

This is blocking patch review testing on master branch:
https://testing.hpdd.intel.com/test_sets/becbbe9c-3fed-11e6-80b9-5254006e85c2
https://testing.hpdd.intel.com/test_sets/828464ba-40e8-11e6-acf3-5254006e85c2
https://testing.hpdd.intel.com/test_sets/e00ada42-4020-11e6-acf3-5254006e85c2

[Gerrit Updater]

James Nunez ([e-mail]) uploaded a new patch: http://review.whamcloud.com/21224
Subject: LU-4039 tests: EXCEPT replay-single test 90
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 27688be30c7c58343ec4eec88798b0588771f196

[Gerrit Updater]

Andreas Dilger ([e-mail]) merged in patch http://review.whamcloud.com/21224/
Subject: LU-4039 tests: EXCEPT replay-single test 90
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 95dbe2cbd2e8829356fc8287366b04fed7131ada

[Gerrit Updater]

Yang Sheng ([e-mail]) uploaded a new patch: http://review.whamcloud.com/21736
Subject: LU-4039 tests: enable test_90 for replay-single
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 5762823612887e81bc9d60874028f97cc441419b

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/21736/
Subject: LU-4039 tests: enable test_90 for replay-single
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: f49e518491e8cc96ca720e22e3f62443148cb102

[Peter Jones]

Test is active again so reclosing. Let's open a new ticket if any furhter failures are found for this test

[nasf (Inactive)]

Hit it again on master:
https://testing.hpdd.intel.com/test_sets/a102421e-5ff5-11e6-b2e2-5254006e85c2

[nasf (Inactive)]

Open new ticket LU-8526 for new failures on master recently.