Details

    • Bug
    • Resolution: Fixed
    • Major
    • Lustre 2.14.0
    • None
    • 3
    • 9223372036854775807

    Description

      There is a race around opd_new_connection causing osp_precreate_thread to infinite wait :

                              l_wait_event(d->opd_pre_waitq,
                                           !osp_precreate_running(d) ||
                                           d->opd_new_connection,
                                           &lwi); 
      

      Attachments

        Issue Links

          Activity

            [LU-12397] osp: race around opd_new_connection

            "Yang Sheng <ys@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/47805
            Subject: LU-12397 osp: remove opd_new_connection
            Project: fs/lustre-release
            Branch: master
            Current Patch Set: 1
            Commit: 2d06a167ac46715b45c8677a1a53b64d95e3e5ff

            gerrit Gerrit Updater added a comment - "Yang Sheng <ys@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/47805 Subject: LU-12397 osp: remove opd_new_connection Project: fs/lustre-release Branch: master Current Patch Set: 1 Commit: 2d06a167ac46715b45c8677a1a53b64d95e3e5ff
            pjones Peter Jones added a comment -

            Landed for 2.14

            pjones Peter Jones added a comment - Landed for 2.14

            Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/35078/
            Subject: LU-12397 osp: always set opd_new_connection
            Project: fs/lustre-release
            Branch: master
            Current Patch Set:
            Commit: 1b5abf625462a2b66820b2d07e25619afba504c6

            gerrit Gerrit Updater added a comment - Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/35078/ Subject: LU-12397 osp: always set opd_new_connection Project: fs/lustre-release Branch: master Current Patch Set: Commit: 1b5abf625462a2b66820b2d07e25619afba504c6

            Sergey Cheremencev (c17829@cray.com) uploaded a new patch: https://review.whamcloud.com/35078
            Subject: LU-12397 osp: always set opd_new_connection
            Project: fs/lustre-release
            Branch: master
            Current Patch Set: 1
            Commit: 83381358299125d4b125c017d74172fe3a130d85

            gerrit Gerrit Updater added a comment - Sergey Cheremencev (c17829@cray.com) uploaded a new patch: https://review.whamcloud.com/35078 Subject: LU-12397 osp: always set opd_new_connection Project: fs/lustre-release Branch: master Current Patch Set: 1 Commit: 83381358299125d4b125c017d74172fe3a130d85

            Below is scenario how this race can happen.

            Disk error caused to hung one of OSTs:

            Mar 11 19:59:39 lstrn09 kernel: Call Trace:
            Mar 11 19:59:39 lstrn09 kernel:  [<ffffffff8163a089>] schedule+0x29/0x70
            Mar 11 19:59:39 lstrn09 kernel:  [<ffffffff814ba2a5>] md_make_request+0xb5/0x370
            Mar 11 19:59:39 lstrn09 kernel:  [<ffffffff812c7342>] generic_make_request+0xe2/0x130
            Mar 11 19:59:39 lstrn09 kernel:  [<ffffffff812c7401>] submit_bio+0x71/0x150
            Mar 11 19:59:39 lstrn09 kernel:  [<ffffffffa157612c>] osd_submit_bio+0x1c/0x60 [osd_ldiskfs]
            Mar 11 19:59:39 lstrn09 kernel:  [<ffffffffa1578577>] osd_do_bio.isra.26+0x3d7/0x830 [osd_ldiskfs]
            Mar 11 19:59:39 lstrn09 kernel:  [<ffffffffa15790f2>] osd_write_commit+0x322/0x900 [osd_ldiskfs]
            Mar 11 19:59:39 lstrn09 kernel:  [<ffffffffa15f1ec0>] ofd_commitrw_write.isra.32+0xc60/0x1c20 [ofd]
            Mar 11 19:59:39 lstrn09 kernel:  [<ffffffffa15f5d42>] ofd_commitrw+0x512/0xac0 [ofd]
            Mar 11 19:59:39 lstrn09 kernel:  [<ffffffffa0ef3a9b>] obd_commitrw.constprop.39+0x2f8/0x33b [ptlrpc]
            Mar 11 19:59:39 lstrn09 kernel:  [<ffffffffa0ee0530>] tgt_brw_write+0x1010/0x1720 [ptlrpc]
            Mar 11 19:59:39 lstrn09 kernel:  [<ffffffffa0edc3db>] tgt_request_handle+0x8fb/0x11f0 [ptlrpc]
            Mar 11 19:59:39 lstrn09 kernel:  [<ffffffffa0e7edab>] ptlrpc_server_handle_request+0x21b/0xa90 [ptlrpc]
            Mar 11 19:59:39 lstrn09 kernel:  [<ffffffffa0e826db>] ptlrpc_main+0xc0b/0x2060 [ptlrpc]
            Mar 11 19:59:39 lstrn09 kernel:  [<ffffffff810a5b8f>] kthread+0xcf/0xe0
            Mar 11 19:59:39 lstrn09 kernel:  [<ffffffff81644fd8>] ret_from_fork+0x58/0x90
             

            Write and read requests couldn't be finished after disk failure.

            At the same time MDT000 and MDT0001 started reporting hung tasks traces like below:

            Mar 11 20:11:36 lstrn02 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
            Mar 11 20:11:36 lstrn02 kernel: mdt00_001       D ffff880f765211d8     0 24438      2 0x00000000
            Mar 11 20:11:36 lstrn02 kernel:  ffff880f77b635a8 0000000000000046 ffff88102b0d6780 ffff880f77b63fd8
            Mar 11 20:11:36 lstrn02 kernel:  ffff880f77b63fd8 ffff880f77b63fd8 ffff88102b0d6780 ffff88102b0d6780
            Mar 11 20:11:36 lstrn02 kernel:  ffff880f765211c8 ffff880f765211d0 ffffffff00000000 ffff880f765211d8
            Mar 11 20:11:36 lstrn02 kernel: Call Trace:
            Mar 11 20:11:36 lstrn02 kernel:  [<ffffffff8163a089>] schedule+0x29/0x70
            Mar 11 20:11:36 lstrn02 kernel:  [<ffffffff8163b845>] rwsem_down_write_failed+0x115/0x220
            Mar 11 20:11:36 lstrn02 kernel:  [<ffffffff813019f3>] call_rwsem_down_write_failed+0x13/0x20
            Mar 11 20:11:36 lstrn02 kernel:  [<ffffffffa15d9d1f>] lod_qos_prep_create+0x111f/0x1fbc [lod]
            Mar 11 20:11:36 lstrn02 kernel:  [<ffffffffa15d35dc>] lod_declare_striped_object+0x1ec/0x790 [lod]
            Mar 11 20:11:36 lstrn02 kernel:  [<ffffffffa15d4bf1>] lod_declare_object_create+0x231/0x4b0 [lod]
            Mar 11 20:11:36 lstrn02 kernel:  [<ffffffffa13372ff>] mdd_declare_object_create_internal+0xdf/0x2f0 [mdd]
            Mar 11 20:11:36 lstrn02 kernel:  [<ffffffffa132b328>] mdd_declare_create+0x48/0xef0 [mdd]
            Mar 11 20:11:36 lstrn02 kernel:  [<ffffffffa132c959>] mdd_create+0x789/0x12a0 [mdd]
            Mar 11 20:11:36 lstrn02 kernel:  [<ffffffffa13b1dd2>] mdt_reint_open+0x1f92/0x2e00 [mdt]
            Mar 11 20:11:36 lstrn02 kernel:  [<ffffffffa13a4e60>] mdt_reint_rec+0x80/0x210 [mdt]
            Mar 11 20:11:36 lstrn02 kernel:  [<ffffffffa1385281>] mdt_reint_internal+0x5e1/0xb40 [mdt]
            Mar 11 20:11:36 lstrn02 kernel:  [<ffffffffa1385942>] mdt_intent_reint+0x162/0x430 [mdt]
            Mar 11 20:11:36 lstrn02 kernel:  [<ffffffffa1389455>] mdt_intent_opc+0x215/0xb50 [mdt]
            Mar 11 20:11:36 lstrn02 kernel:  [<ffffffffa13911a8>] mdt_intent_policy+0x138/0x320 [mdt]
            Mar 11 20:11:36 lstrn02 kernel:  [<ffffffffa0df955a>] ldlm_lock_enqueue+0x35a/0x9c0 [ptlrpc]
            Mar 11 20:11:36 lstrn02 kernel:  [<ffffffffa0e241b3>] ldlm_handle_enqueue0+0x9c3/0x1830 [ptlrpc]
            Mar 11 20:11:36 lstrn02 kernel:  [<ffffffffa0eb1712>] tgt_enqueue+0x62/0x210 [ptlrpc]
            Mar 11 20:11:36 lstrn02 kernel:  [<ffffffffa0eb63db>] tgt_request_handle+0x8fb/0x11f0 [ptlrpc]
            Mar 11 20:11:36 lstrn02 kernel:  [<ffffffffa0e58dab>] ptlrpc_server_handle_request+0x21b/0xa90 [ptlrpc]
            Mar 11 20:11:36 lstrn02 kernel:  [<ffffffffa0e5c6db>] ptlrpc_main+0xc0b/0x2060 [ptlrpc]
            Mar 11 20:11:36 lstrn02 kernel:  [<ffffffff810a5b8f>] kthread+0xcf/0xe0
            Mar 11 20:11:36 lstrn02 kernel:  [<ffffffff81644fd8>] ret_from_fork+0x58/0x90
            

            It is the result of inability to create new objects on OST000a(from lstrn09).

            Hung task's messages were supposed to disappear after lstrn09 failover. But continued to present in messages until lstrn02 crash due to a bug in osp around opd_new_connection. 

            scherementsev Sergey Cheremencev added a comment - Below is scenario how this race can happen. Disk error caused to hung one of OSTs: Mar 11 19:59:39 lstrn09 kernel: Call Trace: Mar 11 19:59:39 lstrn09 kernel: [<ffffffff8163a089>] schedule+0x29/0x70 Mar 11 19:59:39 lstrn09 kernel: [<ffffffff814ba2a5>] md_make_request+0xb5/0x370 Mar 11 19:59:39 lstrn09 kernel: [<ffffffff812c7342>] generic_make_request+0xe2/0x130 Mar 11 19:59:39 lstrn09 kernel: [<ffffffff812c7401>] submit_bio+0x71/0x150 Mar 11 19:59:39 lstrn09 kernel: [<ffffffffa157612c>] osd_submit_bio+0x1c/0x60 [osd_ldiskfs] Mar 11 19:59:39 lstrn09 kernel: [<ffffffffa1578577>] osd_do_bio.isra.26+0x3d7/0x830 [osd_ldiskfs] Mar 11 19:59:39 lstrn09 kernel: [<ffffffffa15790f2>] osd_write_commit+0x322/0x900 [osd_ldiskfs] Mar 11 19:59:39 lstrn09 kernel: [<ffffffffa15f1ec0>] ofd_commitrw_write.isra.32+0xc60/0x1c20 [ofd] Mar 11 19:59:39 lstrn09 kernel: [<ffffffffa15f5d42>] ofd_commitrw+0x512/0xac0 [ofd] Mar 11 19:59:39 lstrn09 kernel: [<ffffffffa0ef3a9b>] obd_commitrw.constprop.39+0x2f8/0x33b [ptlrpc] Mar 11 19:59:39 lstrn09 kernel: [<ffffffffa0ee0530>] tgt_brw_write+0x1010/0x1720 [ptlrpc] Mar 11 19:59:39 lstrn09 kernel: [<ffffffffa0edc3db>] tgt_request_handle+0x8fb/0x11f0 [ptlrpc] Mar 11 19:59:39 lstrn09 kernel: [<ffffffffa0e7edab>] ptlrpc_server_handle_request+0x21b/0xa90 [ptlrpc] Mar 11 19:59:39 lstrn09 kernel: [<ffffffffa0e826db>] ptlrpc_main+0xc0b/0x2060 [ptlrpc] Mar 11 19:59:39 lstrn09 kernel: [<ffffffff810a5b8f>] kthread+0xcf/0xe0 Mar 11 19:59:39 lstrn09 kernel: [<ffffffff81644fd8>] ret_from_fork+0x58/0x90 Write and read requests couldn't be finished after disk failure. At the same time MDT000 and MDT0001 started reporting hung tasks traces like below: Mar 11 20:11:36 lstrn02 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Mar 11 20:11:36 lstrn02 kernel: mdt00_001 D ffff880f765211d8 0 24438 2 0x00000000 Mar 11 20:11:36 lstrn02 kernel: ffff880f77b635a8 0000000000000046 ffff88102b0d6780 ffff880f77b63fd8 Mar 11 20:11:36 lstrn02 kernel: ffff880f77b63fd8 ffff880f77b63fd8 ffff88102b0d6780 ffff88102b0d6780 Mar 11 20:11:36 lstrn02 kernel: ffff880f765211c8 ffff880f765211d0 ffffffff00000000 ffff880f765211d8 Mar 11 20:11:36 lstrn02 kernel: Call Trace: Mar 11 20:11:36 lstrn02 kernel: [<ffffffff8163a089>] schedule+0x29/0x70 Mar 11 20:11:36 lstrn02 kernel: [<ffffffff8163b845>] rwsem_down_write_failed+0x115/0x220 Mar 11 20:11:36 lstrn02 kernel: [<ffffffff813019f3>] call_rwsem_down_write_failed+0x13/0x20 Mar 11 20:11:36 lstrn02 kernel: [<ffffffffa15d9d1f>] lod_qos_prep_create+0x111f/0x1fbc [lod] Mar 11 20:11:36 lstrn02 kernel: [<ffffffffa15d35dc>] lod_declare_striped_object+0x1ec/0x790 [lod] Mar 11 20:11:36 lstrn02 kernel: [<ffffffffa15d4bf1>] lod_declare_object_create+0x231/0x4b0 [lod] Mar 11 20:11:36 lstrn02 kernel: [<ffffffffa13372ff>] mdd_declare_object_create_internal+0xdf/0x2f0 [mdd] Mar 11 20:11:36 lstrn02 kernel: [<ffffffffa132b328>] mdd_declare_create+0x48/0xef0 [mdd] Mar 11 20:11:36 lstrn02 kernel: [<ffffffffa132c959>] mdd_create+0x789/0x12a0 [mdd] Mar 11 20:11:36 lstrn02 kernel: [<ffffffffa13b1dd2>] mdt_reint_open+0x1f92/0x2e00 [mdt] Mar 11 20:11:36 lstrn02 kernel: [<ffffffffa13a4e60>] mdt_reint_rec+0x80/0x210 [mdt] Mar 11 20:11:36 lstrn02 kernel: [<ffffffffa1385281>] mdt_reint_internal+0x5e1/0xb40 [mdt] Mar 11 20:11:36 lstrn02 kernel: [<ffffffffa1385942>] mdt_intent_reint+0x162/0x430 [mdt] Mar 11 20:11:36 lstrn02 kernel: [<ffffffffa1389455>] mdt_intent_opc+0x215/0xb50 [mdt] Mar 11 20:11:36 lstrn02 kernel: [<ffffffffa13911a8>] mdt_intent_policy+0x138/0x320 [mdt] Mar 11 20:11:36 lstrn02 kernel: [<ffffffffa0df955a>] ldlm_lock_enqueue+0x35a/0x9c0 [ptlrpc] Mar 11 20:11:36 lstrn02 kernel: [<ffffffffa0e241b3>] ldlm_handle_enqueue0+0x9c3/0x1830 [ptlrpc] Mar 11 20:11:36 lstrn02 kernel: [<ffffffffa0eb1712>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 11 20:11:36 lstrn02 kernel: [<ffffffffa0eb63db>] tgt_request_handle+0x8fb/0x11f0 [ptlrpc] Mar 11 20:11:36 lstrn02 kernel: [<ffffffffa0e58dab>] ptlrpc_server_handle_request+0x21b/0xa90 [ptlrpc] Mar 11 20:11:36 lstrn02 kernel: [<ffffffffa0e5c6db>] ptlrpc_main+0xc0b/0x2060 [ptlrpc] Mar 11 20:11:36 lstrn02 kernel: [<ffffffff810a5b8f>] kthread+0xcf/0xe0 Mar 11 20:11:36 lstrn02 kernel: [<ffffffff81644fd8>] ret_from_fork+0x58/0x90 It is the result of inability to create new objects on OST000a(from lstrn09). Hung task's messages were supposed to disappear after lstrn09 failover. But continued to present in messages until lstrn02 crash due to a bug in osp around opd_new_connection. 

            People

              scherementsev Sergey Cheremencev
              scherementsev Sergey Cheremencev
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: