[LU-9340] PFL fails performance testsSpirit

• Type: Bug
• Resolution: Fixed
• Priority: Blocker
• Fix Version/s: Lustre 2.10.0
• Affects Version/s: Lustre 2.10.0
• Labels: pfl
• Environment: Spirit performance cluster
• Severity: 3
• Assignee: Jinshan Xiong (Inactive)
• Reporter: Cliff White (Inactive)

Description

Attempting to run P02 and P03 performance tests, with striping set as:
$LFS setstripe $testdir --pool $ior_ostPool -E 64M -c 1 -E 4G -c 4 -E -1 -c -1I

Immediate MPI failures with IOR

 Commencing write performance test: Thu Apr 13 21:04:16 2017
024: ior ERROR: write() failed, errno 61, No data available (aiori-POSIX.c:335)
024: --------------------------------------------------------------------------
024: MPI_ABORT was invoked on rank 24 in communicator MPI_COMM_WORLD
--
..........
231: ior ERROR: write() failed, errno 61, No data available (aiori-POSIX.c:335)
088: In: PMI_Abort(-1, N/A)
287: ior ERROR: write() failed, errno 61, No data available (aiori-POSIX.c:335)
134: In: PMI_Abort(-1, N/A)
057: --------------------------------------------------------------------------
057: MPI_ABORT was invoked on rank 57 in communicator MPI_COMM_WORLD 
--
057: 
057: NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
057: You may or may not see output from other processes, depending on
057: exactly when Open MPI kills them.
057: -------------------------------------------

Lustre Errors on all nodes attached.

Issue Links

• is related to: LU-8494 IOR Aborts with EINTR in 2.7 but not 2.5
• is related to: LU-8998 Progressive File Layout (PFL)
• is related to: LU-9349 PFL known issues tracking ticket

Comments

[Peter Jones]

Landed for 2.10

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/27097/
Subject: LU-9340 lov: Initialize component extents unconditionally
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: df6e700c80f2c216270ca499db7373752f252166

[Gerrit Updater]

Andreas Dilger ([e-mail]) merged in patch https://review.whamcloud.com/27116/
Subject: LU-9340 lov: Initialize component extents unconditionally
Project: fs/lustre-release
Branch: pfl
Current Patch Set:
Commit: 683fb75906cc47fc6aa8c06d47cb672add9a608a

[Gerrit Updater]

Jinshan Xiong ([e-mail]) uploaded a new patch: https://review.whamcloud.com/27116
Subject: LU-9340 lov: Initialize component extents unconditionally
Project: fs/lustre-release
Branch: pfl
Current Patch Set: 1
Commit: 89375c8baccebf3cac1cfa3fd5f8ed1579fc9880

[Jinshan Xiong (Inactive)]

James - this patch won't address any performance issues.

[James A Simmons]

I'm preparing our small test system to see if this patch fixes the 50% drop in performance we see in our PFL testing.

[Gerrit Updater]

Jinshan Xiong ([e-mail]) uploaded a new patch: https://review.whamcloud.com/27097
Subject: LU-9340 lov: Initialize component extents unconditionally
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: d9fd41dc5c644479f740adab77381c33bf22d9dc

[Jinshan Xiong (Inactive)]

cliff - are you able to reproduce this issue on spirit?

[James Nunez (Inactive)]

We need to reopen this ticket because with the patch that landed did not fix the issue on Spirit.

[Peter Jones]

Landed for 2.10