[LU-4366] Test failure sanity test_63b: sync didn't return ENOMEM

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.6.0, Lustre 2.5.1, Lustre 2.7.0
• Fix Version/s: Lustre 2.8.0
• Type: Bug
• Priority: Major
• Reporter: Maloo
• Assignee: Zhenyu Xu
• Resolution: Fixed
• Severity: 3

Description

This issue was created by maloo for Nathaniel Clark <[e-mail]>

This issue relates to the following test suite run:
https://maloo.whamcloud.com/test_sets/4a6ee06c-4c50-11e3-9d6e-52540035b04c
https://maloo.whamcloud.com/test_sets/5de5880c-5ead-11e3-8901-52540035b04c

The sub-test test_63b failed with the following error:

sync didn't return ENOMEM

Info required for matching: sanity 63b

Comments

[Jian Yu]

Lustre Build: http://build.whamcloud.com/job/lustre-b2_5/16/
Distro/Arch: RHEL6.4/x86_64
FSTYPE=ldiskfs

sanity test 63b also hit the same failure:

== sanity test 63b: async write errors should be returned to fsync ===== 13:06:32 (1390424792)
debug=-1
1+0 records in
1+0 records out
4096 bytes (4.1 kB) copied, 0.00599668 s, 683 kB/s
fail_loc=0x80000406
 sanity test_63b: @@@@@@ FAIL: sync didn't return ENOMEM 

Dmesg on client showed that:

Lustre: DEBUG MARKER: == sanity test 63b: async write errors should be returned to fsync ===== 13:06:32 (1390424792)
Lustre: *** cfs_fail_loc=406, val=0***
LustreError: 21803:0:(osc_request.c:2161:osc_build_rpc()) prep_req failed: -12
LustreError: 21803:0:(osc_cache.c:2118:osc_check_rpcs()) Write request failed with -12
Lustre: DEBUG MARKER: /usr/sbin/lctl mark  sanity test_63b: @@@@@@ FAIL: sync didn\'t return ENOMEM 
Lustre: DEBUG MARKER: sanity test_63b: @@@@@@ FAIL: sync didn't return ENOMEM

Maloo reports:
https://maloo.whamcloud.com/test_sets/1a99648c-84ab-11e3-bab5-52540035b04c
https://maloo.whamcloud.com/test_sets/9f4fe770-84cc-11e3-81a1-52540035b04c

This is a regression introduced by Lustre b2_5 build #16 or #15 (not tested). The issue did not occur on Lustre b2_5 build #14 and previous builds.

[Jian Yu]

The failure did not occur regularly on Lustre b2_5 branch. Two more test sessions showed that the same test passed on build #16 and #15:
https://maloo.whamcloud.com/test_sessions/59e43998-8616-11e3-a2cb-52540035b04c (build #16)
https://maloo.whamcloud.com/test_sessions/66025174-8634-11e3-8155-52540035b04c (build #15)

[Jian Yu]

One more instance on Lustre b2_5 branch:
https://testing.hpdd.intel.com/test_sets/49a6765c-5ce6-11e4-8561-5254006e85c2

[nasf (Inactive)]

Another failure instance:
https://testing.hpdd.intel.com/test_sets/a73d4fc4-6dbd-11e4-9d65-5254006e85c2

[Jian Yu]

One more instance on Lustre b2_5 branch:
https://testing.hpdd.intel.com/test_sets/fbf560ce-7a24-11e4-807e-5254006e85c2

[Blake Caldwell]

A failure instance on master branch:
https://testing.hpdd.intel.com/test_sets/09be8858-87ee-11e4-86dc-5254006e85c2

[Bob Glossman (Inactive)]

seen on master:
https://testing.hpdd.intel.com/test_sets/4172ec0c-abad-11e4-992b-5254006e85c2

[Jian Yu]

One more instance on master branch:
https://testing.hpdd.intel.com/test_sets/5ce530f8-ad92-11e4-87bd-5254006e85c2

[Gerrit Updater]

Bobi Jam ([e-mail]) uploaded a new patch: http://review.whamcloud.com/14658
Subject: LU-4366 test: sync didn't return ENOMEM
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 0e3cc8919084eb2e9895b0fb30ff129a6e2d4808

[Gerrit Updater]

Andreas Dilger ([e-mail]) merged in patch http://review.whamcloud.com/14658/
Subject: LU-4366 test: sync didn't return ENOMEM
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: efd7a91040fc87cc73c3b68c7a8ac970606cb9cd

[Andreas Dilger]

Patch landed to master for 2.8.0.