[LU-10132] IO execvp errors 2.10 client/EE3.1.1 server

• Status: Resolved
• Project: Lustre
• Fix Version/s: Lustre 2.11.0, Lustre 2.10.2
• Type: Bug
• Priority: Minor
• Reporter: Trent Geerdes (Inactive)
• Assignee: John Hammond
• Resolution: Fixed
• Severity: 3

Description

Our users are seeing consistent IO errors at very early attempts to scale jobs that access executables on lustre volumes. The errors hang the entire job. These instances are 2.10 client and EE 3.1.1 server scenarios. I've been able to replicate the errors on several different filesystems. The errors do not occur when loading the EE 3.1.1/2.7 client stack to run the same jobs. Is there any expectation of 2.10 client compatibility with EE 3.1.1 servers?

Example job:

$ mpirun -perhost 1 -np 12 -host ekf067,ekf082,ekf087,ekf095,ekf194,ekf195,ekf355,ekf358,ekf359,ekf361,ekf364 -PSM2 /lfs/lfs11/tgeerdes/hello
Hello world from processor ekf087, rank 2 out of 12 processors
Hello world from processor ekf067, rank 11 out of 12 processors
Hello world from processor ekf082, rank 1 out of 12 processors
Hello world from processor ekf195, rank 5 out of 12 processors
Hello world from processor ekf095, rank 3 out of 12 processors
Hello world from processor ekf194, rank 4 out of 12 processors
Hello world from processor ekf355, rank 6 out of 12 processors
Hello world from processor ekf358, rank 7 out of 12 processors
Hello world from processor ekf364, rank 10 out of 12 processors
Hello world from processor ekf361, rank 9 out of 12 processors
Hello world from processor ekf359, rank 8 out of 12 processors
Hello world from processor ekf067, rank 0 out of 12 processors

$mpirun -perhost 2 -np 24 -host ekf067,ekf082,ekf087,ekf095,ekf194,ekf195,ekf355,ekf358,ekf359,ekf361,ekf364 -PSM2 /lfs/lfs11/tgeerdes/hello
[proxy:0:5@ekf195] HYDU_create_process (../../utils/launch/launch.c:825): execvp error on file /lfs/lfs11/tgeerdes/hello (Input/output error)

Comments

[Peter Jones]

Hi Trent

We would expect this combination to interoperate and it is included in our regular release tesitng. When you say 2.10 do you mean 2.10.0 or 2.10.1 clients?

Peter

[Trent Geerdes (Inactive)]

Hi Peter,
2.10.0 clients and this is on Endeavour.
Also the only FS I'm not able to replicate it on is our highest performing SSD based EE 3.1.1 FS. Smaller SSD based and large HDD based all exhibit the issue.

[John Hammond]

Hi Trent, can you attach logs from the clients that experienced the error, the MDT(s), and any OSTs that contained stripes from the executable.

[Trent Geerdes (Inactive)]

Client, MDT, OST's don't log any messages related to the failures. Just the typical, unrelated client disconnects, reconnects.

[John Hammond]

How easy is it to reproduce this?

[Trent Geerdes (Inactive)]

Pretty easy. Many of our customers have hit it and I didn't have trouble reproducing when trying.

[John Hammond]

Could you run the following on each client:

lctl set_param debug_mb=256
lctl set_param debug="vfstrace rpctrace dlmtrace net neterror ha trace"
lctl clear

Then run your reproducer

date +%s
mpirun -perhost 2 -np 24 -host ...

Then run lctl dk > INTL-313.log on one of the clients where execvp() fails and attach the log file here?

You will probably want to restore the settings to debug and debug_mb afterwards.

[Trent Geerdes (Inactive)]

I've set the params and captured the log. Attached.

[John Hammond]

Trent, do you mind if I move this to the LU project?

Thanks to your logs I can reproduce it locally and now understand where this is coming from.

[Trent Geerdes (Inactive)]

Sure, go ahead. Thank you.

[John Hammond]

This is from:

static int ll_xattr_cache_refill(struct inode *inode)
{
        ...
        /* Matched but no cache? Cancelled on error by a parallel refill. */
        if (unlikely(req == NULL)) {
                CDEBUG(D_CACHE, "cancelled by a parallel getxattr\n");
                ll_intent_drop_lock(&oit);
                GOTO(err_unlock, rc = -EIO);
        }
        ...
}

Looking at the code here, we should be returning -EAGAIN instead of -EIO so that ll_getxattr_common() will handle the race.

This affects master as well.

[Gerrit Updater]

John L. Hammond ([e-mail]) uploaded a new patch: https://review.whamcloud.com/29654
Subject: LU-10132 llite: handle xattr cache refill race
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 9cc8c3a20c547ec75325dde3dd17f4b1dcc66348

[Gerrit Updater]

Minh Diep ([e-mail]) uploaded a new patch: https://review.whamcloud.com/29795
Subject: LU-10132 llite: handle xattr cache refill race
Project: fs/lustre-release
Branch: b2_10
Current Patch Set: 1
Commit: f9a2f8bc817829416646dc7d3ea3add16055cefe

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/29654/
Subject: LU-10132 llite: handle xattr cache refill race
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 3dcb7d098759614ae7deb532e1555bd82dac7936

[Peter Jones]

Landed for 2.11

[Gerrit Updater]

John L. Hammond ([e-mail]) merged in patch https://review.whamcloud.com/29795/
Subject: LU-10132 llite: handle xattr cache refill race
Project: fs/lustre-release
Branch: b2_10
Current Patch Set:
Commit: 78a5d681932e30797775aa10d22fc25b20aa58f7