[LU-12397] osp: race around opd_new_connection

• Status: Resolved
• Project: Lustre
• Fix Version/s: Lustre 2.14.0
• Type: Bug
• Priority: Major
• Reporter: Sergey Cheremencev
• Assignee: Sergey Cheremencev
• Resolution: Fixed
• Labels: osp
• Severity: 3

Description

There is a race around opd_new_connection causing osp_precreate_thread to infinite wait :

                        l_wait_event(d->opd_pre_waitq,
                                     !osp_precreate_running(d) ||
                                     d->opd_new_connection,
                                     &lwi); 

Issue Links

• duplicates: LU-16578 osp prealloc_status stuck at -11 afte...

Comments

[Sergey Cheremencev]

Below is scenario how this race can happen.

Disk error caused to hung one of OSTs:

Mar 11 19:59:39 lstrn09 kernel: Call Trace:
Mar 11 19:59:39 lstrn09 kernel:  [<ffffffff8163a089>] schedule+0x29/0x70
Mar 11 19:59:39 lstrn09 kernel:  [<ffffffff814ba2a5>] md_make_request+0xb5/0x370
Mar 11 19:59:39 lstrn09 kernel:  [<ffffffff812c7342>] generic_make_request+0xe2/0x130
Mar 11 19:59:39 lstrn09 kernel:  [<ffffffff812c7401>] submit_bio+0x71/0x150
Mar 11 19:59:39 lstrn09 kernel:  [<ffffffffa157612c>] osd_submit_bio+0x1c/0x60 [osd_ldiskfs]
Mar 11 19:59:39 lstrn09 kernel:  [<ffffffffa1578577>] osd_do_bio.isra.26+0x3d7/0x830 [osd_ldiskfs]
Mar 11 19:59:39 lstrn09 kernel:  [<ffffffffa15790f2>] osd_write_commit+0x322/0x900 [osd_ldiskfs]
Mar 11 19:59:39 lstrn09 kernel:  [<ffffffffa15f1ec0>] ofd_commitrw_write.isra.32+0xc60/0x1c20 [ofd]
Mar 11 19:59:39 lstrn09 kernel:  [<ffffffffa15f5d42>] ofd_commitrw+0x512/0xac0 [ofd]
Mar 11 19:59:39 lstrn09 kernel:  [<ffffffffa0ef3a9b>] obd_commitrw.constprop.39+0x2f8/0x33b [ptlrpc]
Mar 11 19:59:39 lstrn09 kernel:  [<ffffffffa0ee0530>] tgt_brw_write+0x1010/0x1720 [ptlrpc]
Mar 11 19:59:39 lstrn09 kernel:  [<ffffffffa0edc3db>] tgt_request_handle+0x8fb/0x11f0 [ptlrpc]
Mar 11 19:59:39 lstrn09 kernel:  [<ffffffffa0e7edab>] ptlrpc_server_handle_request+0x21b/0xa90 [ptlrpc]
Mar 11 19:59:39 lstrn09 kernel:  [<ffffffffa0e826db>] ptlrpc_main+0xc0b/0x2060 [ptlrpc]
Mar 11 19:59:39 lstrn09 kernel:  [<ffffffff810a5b8f>] kthread+0xcf/0xe0
Mar 11 19:59:39 lstrn09 kernel:  [<ffffffff81644fd8>] ret_from_fork+0x58/0x90
 

Write and read requests couldn't be finished after disk failure.

At the same time MDT000 and MDT0001 started reporting hung tasks traces like below:

Mar 11 20:11:36 lstrn02 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 11 20:11:36 lstrn02 kernel: mdt00_001       D ffff880f765211d8     0 24438      2 0x00000000
Mar 11 20:11:36 lstrn02 kernel:  ffff880f77b635a8 0000000000000046 ffff88102b0d6780 ffff880f77b63fd8
Mar 11 20:11:36 lstrn02 kernel:  ffff880f77b63fd8 ffff880f77b63fd8 ffff88102b0d6780 ffff88102b0d6780
Mar 11 20:11:36 lstrn02 kernel:  ffff880f765211c8 ffff880f765211d0 ffffffff00000000 ffff880f765211d8
Mar 11 20:11:36 lstrn02 kernel: Call Trace:
Mar 11 20:11:36 lstrn02 kernel:  [<ffffffff8163a089>] schedule+0x29/0x70
Mar 11 20:11:36 lstrn02 kernel:  [<ffffffff8163b845>] rwsem_down_write_failed+0x115/0x220
Mar 11 20:11:36 lstrn02 kernel:  [<ffffffff813019f3>] call_rwsem_down_write_failed+0x13/0x20
Mar 11 20:11:36 lstrn02 kernel:  [<ffffffffa15d9d1f>] lod_qos_prep_create+0x111f/0x1fbc [lod]
Mar 11 20:11:36 lstrn02 kernel:  [<ffffffffa15d35dc>] lod_declare_striped_object+0x1ec/0x790 [lod]
Mar 11 20:11:36 lstrn02 kernel:  [<ffffffffa15d4bf1>] lod_declare_object_create+0x231/0x4b0 [lod]
Mar 11 20:11:36 lstrn02 kernel:  [<ffffffffa13372ff>] mdd_declare_object_create_internal+0xdf/0x2f0 [mdd]
Mar 11 20:11:36 lstrn02 kernel:  [<ffffffffa132b328>] mdd_declare_create+0x48/0xef0 [mdd]
Mar 11 20:11:36 lstrn02 kernel:  [<ffffffffa132c959>] mdd_create+0x789/0x12a0 [mdd]
Mar 11 20:11:36 lstrn02 kernel:  [<ffffffffa13b1dd2>] mdt_reint_open+0x1f92/0x2e00 [mdt]
Mar 11 20:11:36 lstrn02 kernel:  [<ffffffffa13a4e60>] mdt_reint_rec+0x80/0x210 [mdt]
Mar 11 20:11:36 lstrn02 kernel:  [<ffffffffa1385281>] mdt_reint_internal+0x5e1/0xb40 [mdt]
Mar 11 20:11:36 lstrn02 kernel:  [<ffffffffa1385942>] mdt_intent_reint+0x162/0x430 [mdt]
Mar 11 20:11:36 lstrn02 kernel:  [<ffffffffa1389455>] mdt_intent_opc+0x215/0xb50 [mdt]
Mar 11 20:11:36 lstrn02 kernel:  [<ffffffffa13911a8>] mdt_intent_policy+0x138/0x320 [mdt]
Mar 11 20:11:36 lstrn02 kernel:  [<ffffffffa0df955a>] ldlm_lock_enqueue+0x35a/0x9c0 [ptlrpc]
Mar 11 20:11:36 lstrn02 kernel:  [<ffffffffa0e241b3>] ldlm_handle_enqueue0+0x9c3/0x1830 [ptlrpc]
Mar 11 20:11:36 lstrn02 kernel:  [<ffffffffa0eb1712>] tgt_enqueue+0x62/0x210 [ptlrpc]
Mar 11 20:11:36 lstrn02 kernel:  [<ffffffffa0eb63db>] tgt_request_handle+0x8fb/0x11f0 [ptlrpc]
Mar 11 20:11:36 lstrn02 kernel:  [<ffffffffa0e58dab>] ptlrpc_server_handle_request+0x21b/0xa90 [ptlrpc]
Mar 11 20:11:36 lstrn02 kernel:  [<ffffffffa0e5c6db>] ptlrpc_main+0xc0b/0x2060 [ptlrpc]
Mar 11 20:11:36 lstrn02 kernel:  [<ffffffff810a5b8f>] kthread+0xcf/0xe0
Mar 11 20:11:36 lstrn02 kernel:  [<ffffffff81644fd8>] ret_from_fork+0x58/0x90

It is the result of inability to create new objects on OST000a(from lstrn09).

Hung task's messages were supposed to disappear after lstrn09 failover. But continued to present in messages until lstrn02 crash due to a bug in osp around opd_new_connection. 

[Gerrit Updater]

Sergey Cheremencev ([e-mail]) uploaded a new patch: https://review.whamcloud.com/35078
Subject: LU-12397 osp: always set opd_new_connection
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 83381358299125d4b125c017d74172fe3a130d85

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/35078/
Subject: LU-12397 osp: always set opd_new_connection
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 1b5abf625462a2b66820b2d07e25619afba504c6

[Peter Jones]

Landed for 2.14

[Gerrit Updater]

"Yang Sheng <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/47805
Subject: LU-12397 osp: remove opd_new_connection
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 2d06a167ac46715b45c8677a1a53b64d95e3e5ff