[LU-12704] racer test_1: Invalid layout: The component end must be aligned by the stripe size

• Type: Bug
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.14.0
• Affects Version/s: Lustre 2.13.0
• Severity: 3
• Assignee: Vladimir Saveliev
• Reporter: Maloo

Description

This issue was created by maloo for jianyu <[e-mail]>

This issue relates to the following test suite run: https://testing.whamcloud.com/test_sets/4c310714-c728-11e9-9fc9-52540065bddc

test_1 failed with the following error:

layout: raid0 raid0 pfl pfl pfl dom dom dom flr flr flr
Invalid layout: The component end must be aligned by the stripe size

<<Please provide additional information about the failure here>>

VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV
racer test_1 - Timeout occurred after 833 mins, last suite running was racer, restarting cluster to continue tests

Issue Links

• is duplicated by: LU-13928 racer/file_create.sh passing bad PFL arguments

Comments

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/36368/
Subject: LU-12704 lov: check all entries in lov_flush_composite
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 44460570fd21a91002190c8a0620923125135b52

[Vladimir Saveliev]

(defect) I think that ignore_layout shouldn't be used here as its ignoring layout locking in LOV, as mentioned in lov_io_init() it is used along with CIT_MISC from OSC usually, because OSC object pins layout already. In our case we want cl_object_flush() be protected from layout change so shouldn't set ci_ignore_layout

Mike, it looks like we are in trouble with this patch as io->ci_ignore_layout = 0; as well as io->ci_ignore_layout = 1; leads to a problem. Any idea?

[Vladimir Saveliev]

+»       io->ci_ignore_layout = 1;

(defect) I think that ignore_layout shouldn't be used here as its ignoring layout locking in LOV, as mentioned in lov_io_init() it is used along with CIT_MISC from OSC usually, because OSC object pins layout already. In our case we want cl_object_flush() be protected from layout change so shouldn't set ci_ignore_layout

Mike, yes, you are right, io->ci_ignore_layout set to 1 leads to race between layout change and cl_io_init. Something like

00020000:00040000:0.0:1571007482.482526:0:28488:0:(lov_io.c:318:lov_io_mirror_init()) ASSERTION( comp->lo_preferred_mirror == 0 ) failed:

has been seen few times.

However, there is also a problem with io->ci_ignore_layout set to 0.

Namely, the below lockup has beed observed:

[<ffffffffc096f065>] ldlm_completion_ast+0x4e5/0x860 [ptlrpc]
[<ffffffffc0970e2a>] ldlm_cli_enqueue_fini+0x63a/0xef0 [ptlrpc]
[<ffffffffc0973b71>] ldlm_cli_enqueue+0x451/0xa60 [ptlrpc]
[<ffffffffc0ba7730>] mdc_enqueue_base+0x330/0x1c40 [mdc]
[<ffffffffc0ba9a85>] mdc_intent_lock+0x135/0x560 [mdc]
[<ffffffffc0be6742>] lmv_intent_lock+0x402/0xa20 [lmv]
[<ffffffffc0c0eb1d>] ll_layout_intent+0x1dd/0x720 [lustre]
[<ffffffffc0c1fa6c>] ll_layout_refresh+0x30c/0x900 [lustre]
[<ffffffffc0c62ea7>] vvp_io_init+0x347/0x460 [lustre]
[<ffffffffc076bf4b>] cl_io_init0.isra.15+0x8b/0x160 [obdclass]
[<ffffffffc076c0e3>] cl_io_init+0x43/0x80 [obdclass]
[<ffffffffc0c41fe5>] ll_lock_cancel_bits+0x625/0xca0 [lustre]
[<ffffffffc0c42a5c>] ll_md_blocking_ast+0x24c/0x2b0 [lustre]
[<ffffffffc09626ba>] ldlm_cancel_callback+0x8a/0x330 [ptlrpc]
[<ffffffffc096e311>] ldlm_cli_cancel_local+0xd1/0x420 [ptlrpc]
[<ffffffffc097294a>] ldlm_cli_cancel_list_local+0xea/0x280 [ptlrpc]
[<ffffffffc0972c6b>] ldlm_cancel_resource_local+0x18b/0x2a0 [ptlrpc]
[<ffffffffc0ba02ac>] mdc_resource_get_unused_res+0x10c/0x250 [mdc]
[<ffffffffc0bb1057>] mdc_enqueue_send+0x557/0x710 [mdc]
[<ffffffffc0bb14b2>] mdc_lock_enqueue+0x2a2/0x6f2 [mdc]
[<ffffffffc07696d5>] cl_lock_enqueue+0x65/0x120 [obdclass]
[<ffffffffc0ab51e5>] lov_lock_enqueue+0x95/0x150 [lov]
[<ffffffffc07696d5>] cl_lock_enqueue+0x65/0x120 [obdclass]
[<ffffffffc0769c67>] cl_lock_request+0x67/0x1f0 [obdclass]
[<ffffffffc076d9cb>] cl_io_lock+0x2bb/0x3d0 [obdclass]
[<ffffffffc076dcfa>] cl_io_loop+0xba/0x1c0 [obdclass]
[<ffffffffc0c5981f>] cl_setattr_ost+0x25f/0x3d0 [lustre]
[<ffffffffc0c34b28>] ll_setattr_raw+0xcc8/0x1060 [lustre]
[<ffffffffc0c34f23>] ll_setattr+0x63/0xc0 [lustre]
[<ffffffffbc260524>] notify_change+0x2c4/0x420
[<ffffffffbc23f335>] do_truncate+0x75/0xc0

mdc_enqueue_send() tries to do early cancel, ll_lock_cancel_bits()->ll_dom_lock_cancel() initializes cl_io->ci_ignore_layout to 0, so vvp_io_init() does ll_layout_refresh() and takes lli->lli_layout_mutex and sends enqueue rpc. Server sends blocking ast back to the client, so another ll_lock_cancel_bits() gets to run and stuck on trying to lock the mutex lli->lli_layout_mutex:

int ll_layout_refresh(struct inode *inode, __u32 *gen)
{
...
    mutex_lock(&lli->lli_layout_mutex);
...

[<ffffffffc0c1f94e>] ll_layout_refresh+0x1ee/0x900 [lustre]
[<ffffffffc0c62ea7>] vvp_io_init+0x347/0x460 [lustre]
[<ffffffffc076bf4b>] cl_io_init0.isra.15+0x8b/0x160 [obdclass]
[<ffffffffc076c0e3>] cl_io_init+0x43/0x80 [obdclass]
[<ffffffffc0c41fe5>] ll_lock_cancel_bits+0x625/0xca0 [lustre]
[<ffffffffc0c42a99>] ll_md_blocking_ast+0x289/0x2b0 [lustre]
[<ffffffffc0978bdd>] ldlm_handle_bl_callback+0xed/0x4e0 [ptlrpc]
[<ffffffffc09797d0>] ldlm_bl_thread_main+0x800/0xa40 [ptlrpc]
[<ffffffffbc0c1c71>] kthread+0xd1/0xe0

[Peter Jones]

Moving to 2.14 until we can understand the crashes in Olegtest

[Andreas Dilger]

Mike, Oleg is still hitting crashes with https://review.whamcloud.com/36300 so it can't land as-is. Does it make sense to rebase your patch to be directly on master so that it can land independently, or does it depend on 36300 in order to work properly?

[Mikhail Pershin]

Vladimir, do you think https://review.whamcloud.com/#/c/36300 is still needed? I was thinking that we are safe from layout change while LSM is referenced and no need to initialize IO in llite.

[Vladimir Saveliev]

Mike, wouldn't it be better to have  https://review.whamcloud.com/36368 on top of https://review.whamcloud.com/#/c/36300?

[Mikhail Pershin]

Vladimir, yes, I think it is what we need to add

P.S. I've updated patch with that code

[Vladimir Saveliev]

It seems that all elements of lov_dispatch[] need all members of struct lov_layout_operations initialized.

So, how about defining lov_dispatch[LLT_EMPTY].llo_flush and lov_dispatch[LLT_RELEASED].llo_flush to a function like

static int lov_flush_empty()
{
   return 0;
}

Mike, would that work?

[Vladimir Saveliev]

Could you check if patch above fixes that problem?

[ 2091.709375] BUG: unable to handle kernel NULL pointer dereference at           (null)
[ 2091.712428] IP: [<          (null)>]           (null)
...
[ 2091.736582] CPU: 0 PID: 20963 Comm: ldlm_bl_08 Kdump: loaded Tainted: G           OE  ------------   3.10.0-957.5.1.el7.x86_64 #1
...
[ 2091.773122] Call Trace:
[ 2091.775002]  [<ffffffffc0c9a1a2>] ? lov_object_flush+0x22/0x60 [lov]
[ 2091.777577]  [<ffffffffc093e193>] cl_object_flush+0x63/0x120 [obdclass]
[ 2091.780136]  [<ffffffffc0e1c408>] ll_lock_cancel_bits+0x9b8/0xc00 [lustre]

According to crash dump (from Cray's test system), the BUG happened when lov_object_flush() tried to dispatch to lov_flush_composite(), so probably the change in lov_flush_composite() will not help.
However, lov_object has LLT_EMPTY layout already:

crash> lov_object.lo_type,lo_lsm 0xffff8f46e1af42e0
  lo_type = LLT_EMPTY
  lo_lsm = 0x0

That probably happened earlier when layout lock was canceled:

00000080:00010000:0.0:1569576242.662719:0:21009:0:(namei.c:248:ll_lock_cancel_bits()) ### to cancel bits 0x19 ns: lustre-MDT0001-mdc-ffff8f4777b0e000 lock: ffff8f4765861d40/0x7909e24de166dec lrc: 3/0,0 mode: PR/PR res: [0x240000406:0x5278:0x0].0x0 bits 0x19/0x19 rrc: 3 type: IBT flags: 0x429400000000 nid: local remote: 0x96dffd219df32662 expref: -99 pid: 3158 timeout: 0 lvb_type: 0
00000080:00200000:0.0:1569576242.662727:0:21009:0:(vvp_object.c:140:vvp_conf_set()) [0x240000406:0x5278:0x0]: losing layout lock

Then on canceling DoM lock ldlm_bl_08 thread faced with empty layout:

00000080:00010000:0.0:1569576242.728877:0:20963:0:(namei.c:248:ll_lock_cancel_bits()) ### to cancel bits 0x40 ns: lustre-MDT0001-mdc-ffff8f4777b0e000 lock: ffff8f47663ee000/0x7909e24de166f19 lrc: 2/0,0 mode: PR/PR res: [0x240000406:0x5278:0x0].0x0 bits 0x48/0x40 rrc: 3 type: IBT flags: 0x460400000000 nid: local remote: 0x96dffd219df32c12 expref: -99 pid: 3288 timeout: 0 lvb_type: 3

[Mikhail Pershin]

I was fixing DOM entry checking in lov_flush_composite() and thought that taking LSM reference should protect us from layout change. Could you check if patch above fixes that problem?