[LU-16362] landing async readhead caused a "uncovered page" panic during sanityN runs

• Status: Open
• Project: Lustre
• Type: Bug
• Priority: Critical
• Reporter: Alexey Lyashkov
• Assignee: Alexey Lyashkov
• Resolution: Unresolved
• Severity: 3

Description

Async readhead have skip a "locking" phase

c2791674260 (Wang Shilong 2019-01-21 20:23:47 +0800 658) io->ci_state = CIS_LOCKED;

But separate cl io created, this caused pages sends outside of original page lock and parallel blocking AST caused an "uncovered page" panic hit.

Issue Links

• is related to: LU-16332 LBUG with osc_req_attr_set() uncovere...
• is related to: LU-16401 various crashes with cl_page_discard ...

Comments

[Alexander Zarochentsev]

LU-16332 is a similar issue.

[Gerrit Updater]

"Alexey Lyashkov <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/51677
Subject: LU-16362 lov: dont't allow RA outside of active stripe
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 01966622f7b4f1c7626bc5980f9adc0d01338e1c

[Alexey Lyashkov]

once first patch version found an issues, I spent more time to investing it.
Current picture is:

application started to read a data from a file. It's multi stripe file, but read want to read just first stripe only, and cl io had created to offset range assigned to first stripe.
once stripe size is 4M now - RA started to full rpc with this size and finished at end of stripe offset.
On next read, RA window moved outside of IO region requested by cl_io and outside of active locks pined by this IO.
In the current code, it caused a situation osc will search an old lock for the second stripe. lock might exist from past, it's OK. but RA had pined this lock just to send a prepare read for one page - and release it after it via cl_read_ahead_release().
It caused a situation when some pages prepared to send read before lock cancel, but some not and error returned from cl_io_read_ahead(), but previous pages still in osc extents and ready to send rpc. It caused a panic in the osc_req_attr_set when osc have a time to send.
This is easy reproduce with simple assertion and sanity 44A+44a (START_AT=44A).

@@ -1195,22 +1198,27 @@ static int lov_io_read_ahead(const struct lu_env *env,
        if (unlikely(!r0->lo_sub[stripe]))
                RETURN(-EIO);

        sub = lov_sub_get(env, lio, lov_comp_index(index, stripe));
        if (IS_ERR(sub))
                RETURN(PTR_ERR(sub));

        /* no RA outside of active stripe */
+       if (sub->sub_io.ci_state != CIS_IO_GOING &&
+           sub->sub_io.ci_state != CIS_LOCKED)
+               LBUG();

I'm not sure what is best in this situation - just disable an RA outside of active region. or make lock pinning better to avoid situation with submitting without lock held.

[Alexey Lyashkov]

I looks I understand an issue finally.
readahead loop collects a locked pages during loop, but ldlm lock released after each iteration. So just PG_lock prevents a lock canceling during submit, bit osc_make_rpc had release a page lock after rpc created but don't send. it open window when lock might be canceled and osc_req_attr_set complains lock don't exist.
Lock might be canceled due aging or conlfict on the different node.
Right code should create an special cl_io to collect a lock and cl_io_submit_sync should be called during lock held. osc extent will hold a ldlm lock (after Bobi Jam patch LU-16160 osc: take ldlm lock when queue sync pages) it will prevent to cancel a lock until it will don't needs.

[Alexey Lyashkov]

It might be artifact of PG_writeback change. page don't unlocked until IO done before it and panic don't hit.

[Gerrit Updater]

"Alexey Lyashkov <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/53635
Subject: LU-16362 readahead: simplification
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: b41d1363be6ee45e2b19ec708be43954dde69e86