[LU-8680] replay-single test_20b: BUG: soft lockup - osc_makes_rpc()

• Type: Bug
• Resolution: Fixed
• Priority: Blocker
• Fix Version/s: Lustre 2.9.0
• Affects Version/s: Lustre 2.9.0
• Severity: 3
• Assignee: Zhenyu Xu
• Reporter: Maloo

Description

This issue was created by maloo for Niu Yawei <[e-mail]>

This issue relates to the following test suite run: https://testing.hpdd.intel.com/test_sets/a265b3dc-8b49-11e6-ad53-5254006e85c2.

The sub-test test_20b failed with the following error:

test failed to respond and timed out

Please provide additional information about the failure here.

Info required for matching: replay-single 20b

Comments

[Peter Jones]

Landed for 2.9

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/23326/
Subject: LU-8680 osc: soft lock - osc_makes_rpc()
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: a687000d2400fee88f122526444700502cb57fe4

[Gerrit Updater]

Bobi Jam ([e-mail]) uploaded a new patch: http://review.whamcloud.com/23326
Subject: LU-8680 osc: soft lock - osc_makes_rpc()
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 4b891b4752273c0852bec39e188a6aecfec800de

[Jinshan Xiong (Inactive)]

Indeed. It occurred to me why I would like to have

osc_max_write_chunks()
{
        return PTLRPC_MAX_BRW_SIZE >> cli->cl_chunkbits;
}

instead of the current implementation.

[Niu Yawei (Inactive)]

One thing I can't see from the patch of LU-8135 is that: when will the large extent (exceeding chunk limitation) being flushed back?

[Niu Yawei (Inactive)]

This could be a regression caused by LU-8135, that patch limited chunk number in a write RPC, so for a extent with large number of chunk, osc_check_rpcs() will run into a loop and never break.

see osc_check_rpcs()

                if (osc_makes_rpc(cli, osc, OBD_BRW_WRITE)) {
                        rc = osc_send_write_rpc(env, cli, osc);
                        if (rc < 0) {
                                CERROR("Write request failed with %d\n", rc);

                                /* osc_send_write_rpc failed, mostly because of
                                 * memory pressure.
                                 *
                                 * It can't break here, because if:
                                 *  - a page was submitted by osc_io_submit, so
                                 *    page locked;
                                 *  - no request in flight
                                 *  - no subsequent request
                                 * The system will be in live-lock state,
                                 * because there is no chance to call
                                 * osc_io_unplug() and osc_check_rpcs() any
                                 * more. pdflush can't help in this case,
                                 * because it might be blocked at grabbing
                                 * the page lock as we mentioned.
                                 *
                                 * Anyway, continue to drain pages. */
                                /* break; */
                        }
                }

With the fix of LU-8135, osc_send_write_rpc() will do nothing when the extent is too large, and osc_check_rpcs() won't break loop but continue to try on the same object.

[Joseph Gmitter (Inactive)]

Hi Bobijam,

Could you please look into this issue?

Thanks.
Joe

[Niu Yawei (Inactive)]

Looks all these failures happened in interop testing, Sarah, did you ever observe such failure during interop testing?

[Niu Yawei (Inactive)]

I searched maloo, and seems it appeared since around Sep 24 (not sure why some results don't have console logs, so it's hard to determine the exact date when it appeared first time), and happens quite often.

[Jinshan Xiong (Inactive)]

not to my knowledge.