[LU-10132] IO execvp errors 2.10 client/EE3.1.1 server

• Type: Bug
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.11.0, Lustre 2.10.2
• Severity: 3
• Assignee: John Hammond
• Reporter: Trent Geerdes (Inactive)

Description

Our users are seeing consistent IO errors at very early attempts to scale jobs that access executables on lustre volumes. The errors hang the entire job. These instances are 2.10 client and EE 3.1.1 server scenarios. I've been able to replicate the errors on several different filesystems. The errors do not occur when loading the EE 3.1.1/2.7 client stack to run the same jobs. Is there any expectation of 2.10 client compatibility with EE 3.1.1 servers?

Example job:

$ mpirun -perhost 1 -np 12 -host ekf067,ekf082,ekf087,ekf095,ekf194,ekf195,ekf355,ekf358,ekf359,ekf361,ekf364 -PSM2 /lfs/lfs11/tgeerdes/hello
Hello world from processor ekf087, rank 2 out of 12 processors
Hello world from processor ekf067, rank 11 out of 12 processors
Hello world from processor ekf082, rank 1 out of 12 processors
Hello world from processor ekf195, rank 5 out of 12 processors
Hello world from processor ekf095, rank 3 out of 12 processors
Hello world from processor ekf194, rank 4 out of 12 processors
Hello world from processor ekf355, rank 6 out of 12 processors
Hello world from processor ekf358, rank 7 out of 12 processors
Hello world from processor ekf364, rank 10 out of 12 processors
Hello world from processor ekf361, rank 9 out of 12 processors
Hello world from processor ekf359, rank 8 out of 12 processors
Hello world from processor ekf067, rank 0 out of 12 processors

$mpirun -perhost 2 -np 24 -host ekf067,ekf082,ekf087,ekf095,ekf194,ekf195,ekf355,ekf358,ekf359,ekf361,ekf364 -PSM2 /lfs/lfs11/tgeerdes/hello
[proxy:0:5@ekf195] HYDU_create_process (../../utils/launch/launch.c:825): execvp error on file /lfs/lfs11/tgeerdes/hello (Input/output error)

Comments

[Gerrit Updater]

John L. Hammond ([e-mail]) merged in patch https://review.whamcloud.com/29795/
Subject: LU-10132 llite: handle xattr cache refill race
Project: fs/lustre-release
Branch: b2_10
Current Patch Set:
Commit: 78a5d681932e30797775aa10d22fc25b20aa58f7

[Peter Jones]

Landed for 2.11

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/29654/
Subject: LU-10132 llite: handle xattr cache refill race
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 3dcb7d098759614ae7deb532e1555bd82dac7936

[Gerrit Updater]

Minh Diep ([e-mail]) uploaded a new patch: https://review.whamcloud.com/29795
Subject: LU-10132 llite: handle xattr cache refill race
Project: fs/lustre-release
Branch: b2_10
Current Patch Set: 1
Commit: f9a2f8bc817829416646dc7d3ea3add16055cefe

[Gerrit Updater]

John L. Hammond ([e-mail]) uploaded a new patch: https://review.whamcloud.com/29654
Subject: LU-10132 llite: handle xattr cache refill race
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 9cc8c3a20c547ec75325dde3dd17f4b1dcc66348

[John Hammond]

This is from:

static int ll_xattr_cache_refill(struct inode *inode)
{
        ...
        /* Matched but no cache? Cancelled on error by a parallel refill. */
        if (unlikely(req == NULL)) {
                CDEBUG(D_CACHE, "cancelled by a parallel getxattr\n");
                ll_intent_drop_lock(&oit);
                GOTO(err_unlock, rc = -EIO);
        }
        ...
}

Looking at the code here, we should be returning -EAGAIN instead of -EIO so that ll_getxattr_common() will handle the race.

This affects master as well.

[Trent Geerdes (Inactive)]

Sure, go ahead. Thank you.

[John Hammond]

Trent, do you mind if I move this to the LU project?

Thanks to your logs I can reproduce it locally and now understand where this is coming from.

[Trent Geerdes (Inactive)]

I've set the params and captured the log. Attached.

[John Hammond]

Could you run the following on each client:

lctl set_param debug_mb=256
lctl set_param debug="vfstrace rpctrace dlmtrace net neterror ha trace"
lctl clear

Then run your reproducer

date +%s
mpirun -perhost 2 -np 24 -host ...

Then run lctl dk > INTL-313.log on one of the clients where execvp() fails and attach the log file here?

You will probably want to restore the settings to debug and debug_mb afterwards.