[LU-3027] Failure on test suite parallel-scale test_write_disjoint: invalid file size 140329 instead of 160376 = 20047 * 8

• Type: Bug
• Resolution: Fixed
• Priority: Blocker
• Fix Version/s: Lustre 2.4.2
• Affects Version/s: Lustre 2.4.0, Lustre 2.4.1, Lustre 2.5.0
• Severity: 3
• Assignee: Oleg Drokin
• Reporter: Maloo

Description

This issue was created by maloo for sarah <[e-mail]>

This issue relates to the following test suite run: https://maloo.whamcloud.com/test_sets/2ed1fef2-94bd-11e2-93c6-52540035b04c.

The sub-test test_write_disjoint failed with the following error:

write_disjoint failed! 1

test log shows:

librdmacm: Fatal: no RDMA devices found
librdmacm: Fatal: no RDMA devices found
librdmacm: Fatal: no RDMA devices found
librdmacm: Fatal: no RDMA devices found
librdmacm: Fatal: no RDMA devices found
librdmacm: Fatal: no RDMA devices found
librdmacm: Fatal: no RDMA devices found
loop 0: chunk_size 103399
[client-27vm6.lab.whamcloud.com:00935] 7 more processes have sent help message help-mpi-btl-base.txt / btl:no-nics
[client-27vm6.lab.whamcloud.com:00935] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
loop 79: chunk_size 71702, file size was 573616
rank 2, loop 80: invalid file size 140329 instead of 160376 = 20047 * 8
loop 79: chunk_size 71702, file size was 573616
rank 4, loop 80: invalid file size 140329 instead of 160376 = 20047 * 8
loop 79: chunk_size 71702, file size was 573616
rank 6, loop 80: invalid file size 140329 instead of 160376 = 20047 * 8
loop 79: chunk_size 71702, file size was 573616
rank 0, loop 80: invalid file size 140329 instead of 160376 = 20047 * 8
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 4 in communicator MPI_COMM_WORLD 
with errorcode -1.

Looks like LU-2453 is a similar issue seen in b2_1 branch

Issue Links

• is duplicated by: LU-701 parallel-scale test_write_disjoint fails due to invalid file size
• is related to: LU-3889 LBUG: (osc_lock.c:497:osc_lock_upcall()) ASSERTION( lock->cll_state >= CLS_QUEUING )
• is related to: LU-3874 parallel-scale-nfsv3 test_iorfpp: ERROR in aiori-POSIX.c (line 235): seek failed
• is related to: LU-3889 LBUG: (osc_lock.c:497:osc_lock_upcall()) ASSERTION( lock->cll_state >= CLS_QUEUING )

Comments

[Patrick Farrell (Inactive)]

Interesting - Thanks, Jian.

[Jian Yu]

Hi Patrick,

Lustre b2_4 build #48 contains those two patches: http://build.whamcloud.com/job/lustre-b2_4/48/. And since that build, parallel-scale test write_disjoint has passed:
https://maloo.whamcloud.com/test_sets/35b41124-45a7-11e3-b22a-52540035b04c
https://maloo.whamcloud.com/test_sets/484f9162-4590-11e3-8713-52540035b04c
https://maloo.whamcloud.com/test_sets/c5f43484-4602-11e3-b5e8-52540035b04c
https://maloo.whamcloud.com/test_sets/1c537816-47f3-11e3-bc81-52540035b04c

[Patrick Farrell (Inactive)]

Jian - Are you saying that when you added 7569 and 7841 to b2_4 you no longer see the failure?

I do not see those patches as landed to b2_4 in Gerrit, and Cray is still seeing the closely related LU-3889 with both patches landed in 2.4.

[Jian Yu]

Patches http://review.whamcloud.com/7569 and http://review.whamcloud.com/7841 landed on Lustre b2_4 branch. The failure was fixed.

[Jian Yu]

Lustre build: http://build.whamcloud.com/job/lustre-b2_4/47/

The failure still occurred regularly on Lustre b2_4 branch:
https://maloo.whamcloud.com/test_sets/0039b106-4332-11e3-9490-52540035b04c
https://maloo.whamcloud.com/test_sets/762409ba-4333-11e3-8676-52540035b04c

[Peter Jones]

Landed for 2.5

[Sarah Liu]

Here is the result of parallel-scale test_write_disjoint:

https://maloo.whamcloud.com/test_sessions/5602b272-303b-11e3-b28a-52540035b04c

[Jinshan Xiong (Inactive)]

patch is at: http://review.whamcloud.com/7841, please give it a try.

[Sarah Liu]

hmm for LU-3027, tag-2.4.93 (http://build.whamcloud.com/job/lustre-master/1687/) should have the fix but I still hit this error:

https://maloo.whamcloud.com/test_sets/9a628942-272b-11e3-88c6-52540035b04c

[Peter Jones]

Patch landed for 2.5.0