Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-20640

PCC: deadlock between pcc_file_read_iter() and pcc_mmap_mapping_set()

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Medium Medium
    • None
    • None
    • None
    • 3
    • 9223372036854775807

      They were able to collect a vmcore from the latest occurrence and analyzed it. According to the analysis, there appears to be a deadlock involving two threads, PID 926158 and PID 926526.
      The customer also found that PID 926158 is the owner of the semaphore for which PID 926526 is waiting. In addition, PID 926158 appears to own another mutex for which the Lustre ldlm_bl_52 thread (PID 251087) is waiting.
      The relevant threads are:

      PID 926158 TASK ff2171d21d7e0000 COMMAND "VLLM::EngineCor"
      PID 926526 TASK ff2171d700958000 COMMAND "VLLM::EngineCor"
      PID 251087 TASK ff21718bdc51c000 COMMAND "ldlm_bl_52"
      The stack traces show:

      crash-7.3.1> bt 926158
      PID: 926158 TASK: ff2171d21d7e0000 CPU: 90 COMMAND: "VLLM::EngineCor"
      #0 [ff3b818e8bc27c18] __schedule at ffffffff83bfb5d1
      #1 [ff3b818e8bc27ca8] schedule at ffffffff83bfbbc5
      #2 [ff3b818e8bc27cc0] pcc_mmap_mapping_set at ffffffffc2afa38f [lustre]
      #3 [ff3b818e8bc27d20] pcc_file_mmap at ffffffffc2b02948 [lustre]
      #4 [ff3b818e8bc27d60] ll_file_mmap at ffffffffc2ada394 [lustre]
      #5 [ff3b818e8bc27da0] mmap_region at ffffffff834e7cab
      #6 [ff3b818e8bc27e08] do_mmap at ffffffff834e826b
      #7 [ff3b818e8bc27e60] vm_mmap_pgoff at ffffffff834c0034
      #8 [ff3b818e8bc27ee0] ksys_mmap_pgoff at ffffffff834e585c
      #9 [ff3b818e8bc27f38] do_syscall_64 at ffffffff832052fb
      #10 [ff3b818e8bc27f50] entry_SYSCALL_64_after_hwframe at ffffffff83e000a9
        RIP: 00001490db7a990d RSP: 00007ffcf019d618 RFLAGS: 00000216
        RAX: ffffffffffffffda RBX: 00007ffcf019d680 RCX: 00001490db7a990d
        RDX: 0000000000000001 RSI: 0000000016e15268 RDI: 0000000000000000
        RBP: 0000000000000001 R8: 000000000000006e R9: 0000000000000000
        R10: 0000000000000002 R11: 0000000000000216 R12: 0000148ba0080510
        R13: 00007ffcf019d680 R14: 000000000000006e R15: 0000000200000001
        ORIG_RAX: 0000000000000009 CS: 0033 SS: 002b
      PID 926158 is sleeping in pcc_mmap_mapping_set().

      crash-7.3.1> bt 926526
      PID: 926526 TASK: ff2171d700958000 CPU: 93 COMMAND: "VLLM::EngineCor"
      #0 [ff3b818e76627948] __schedule at ffffffff83bfb5d1
      #1 [ff3b818e766279d8] schedule at ffffffff83bfbbc5
      #2 [ff3b818e766279f0] schedule_preempt_disabled at ffffffff83bfbf1a
      #3 [ff3b818e766279f8] rwsem_down_read_slowpath at ffffffff83bfe73e
      #4 [ff3b818e76627a90] down_read at ffffffff83bfe955
      #5 [ff3b818e76627aa0] __do_page_fault at ffffffff8327f788
      #6 [ff3b818e76627af0] do_page_fault at ffffffff8327f8a7
      #7 [ff3b818e76627b20] page_fault at ffffffff83e0116e
        [exception RIP: copy_user_enhanced_fast_string+14]
        RIP: ffffffff83bea7ae RSP: ff3b818e76627bd0 RFLAGS: 00050206
        RAX: 0000147ed8401c00 RBX: ff2171d5ce84d000 RCX: 0000000000000c00
        RDX: 0000000000001000 RSI: ff2171d5ce84d400 RDI: 0000147ed8401000
        RBP: 0000000000001000 R8: 000000000084e224 R9: 000000000084e224
        R10: ff21710b800026c0 R11: ff2171cb07f49e00 R12: ff3b818e76627e58
        R13: 0000000000001000 R14: ff3b818e76627e48 R15: 0000000000000000
        ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
      #8 [ff3b818e76627bd0] copyout at ffffffff836deb32
      #9 [ff3b818e76627bd8] copy_page_to_iter at ffffffff836df37e
      #10 [ff3b818e76627c20] generic_file_buffered_read at ffffffff8349d985
      #11 [ff3b818e76627d10] xfs_file_buffered_read at ffffffffc0996c3e [xfs]
      #12 [ff3b818e76627d38] xfs_file_read_iter at ffffffffc0996d1e [xfs]
      #13 [ff3b818e76627d48] pcc_file_read_iter at ffffffffc2b01864 [lustre]
      #14 [ff3b818e76627dc0] ll_file_read_iter at ffffffffc2a99fe8 [lustre]
      #15 [ff3b818e76627e40] new_sync_read at ffffffff835624ef
      #16 [ff3b818e76627ec8] vfs_read at ffffffff83564ff1
      #17 [ff3b818e76627f00] ksys_read at ffffffff8356545f
      #18 [ff3b818e76627f38] do_syscall_64 at ffffffff832052fb
      #19 [ff3b818e76627f50] entry_SYSCALL_64_after_hwframe at ffffffff83e000a9
        RIP: 00001490db79f89c RSP: 00001488ee6948a0 RFLAGS: 00000246
        RAX: ffffffffffffffda RBX: 00001488ee6955b0 RCX: 00001490db79f89c
        RDX: 0000000001000000 RSI: 0000147ed8400c00 RDI: 000000000000006d
        RBP: 0000000001000000 R8: 0000000000000000 R9: 0000000000b2d0e0
        R10: 0000000000000000 R11: 0000000000000246 R12: 00001488ee694a28
        R13: 0000147ed8400c00 R14: 000000000000006d R15: 0000147ee0002000
        ORIG_RAX: 0000000000000000 CS: 0033 SS: 002b
      PID 926526 is therefore in the PCC read path and is blocked in rwsem_down_read_slowpath() following a page fault.
      The Lustre blocking AST thread is also blocked in the PCC path:

      crash-7.3.1> bt 251087
      PID: 251087 TASK: ff21718bdc51c000 CPU: 72 COMMAND: "ldlm_bl_52"
      #0 [ff3b818e5faf7ab8] __schedule at ffffffff83bfb5d1
      #1 [ff3b818e5faf7b48] schedule at ffffffff83bfbbc5
      #2 [ff3b818e5faf7b60] schedule_preempt_disabled at ffffffff83bfbf1a
      #3 [ff3b818e5faf7b68] __mutex_lock at ffffffff83bfdbb9
      #4 [ff3b818e5faf7bd0] pcc_layout_invalidate at ffffffffc2b000eb [lustre]
      #5 [ff3b818e5faf7c40] vvp_conf_set at ffffffffc2af4adc [lustre]
      #6 [ff3b818e5faf7c60] cl_conf_set at ffffffffc2473319 [obdclass]
      #7 [ff3b818e5faf7c88] ll_layout_conf at ffffffffc2aa2e52 [lustre]
      #8 [ff3b818e5faf7d08] ll_lock_cancel_bits at ffffffffc2acdafb [lustre]
      #9 [ff3b818e5faf7d78] ll_md_blocking_ast at ffffffffc2ace0a9 [lustre]
      #10 [ff3b818e5faf7da8] ldlm_cancel_callback at ffffffffc2763aeb [ptlrpc]
      #11 [ff3b818e5faf7e00] ldlm_cli_cancel_local at ffffffffc277041b [ptlrpc]
      #12 [ff3b818e5faf7e20] ldlm_cli_cancel_list_local at ffffffffc2774543 [ptlrpc]
      #13 [ff3b818e5faf7e90] ldlm_bl_thread_main at ffffffffc277b0df [ptlrpc]
      #14 [ff3b818e5faf7f10] kthread at ffffffff8331d6a4
      #15 [ff3b818e5faf7f50] ret_from_fork at ffffffff83e0024f
      crash-7.3.1>
      The customer therefore suspects that the issue originates from PID 926158 and requested that we investigate it from the Lustre/PCC perspective.

      The preceding commands establish the following dependency chain:

      PID 926526 (PCC read)
      has executed pcc_io_init()
      pcci_active_ios is therefore 1
      has not yet executed pcc_io_fini()
      waits for a READ lock on mmap_sem ff2171f5700dc270

      v
      PID 926158 (PCC mmap)
      owns mmap_sem ff2171f5700dc270
      sleeps in pcc_mmap_mapping_set()
      waits for pcci_active_ios to become 0

      v
      PID 926526 must finish its PCC I/O and execute pcc_io_fini()

      In compact form:

      PID 926526
      -> waits for mmap_sem
      -> mmap_sem is owned by PID 926158
      -> PID 926158 waits for pcci_active_ios == 0
      -> the outstanding PCC read is PID 926526, after pcc_io_init() and before
      pcc_io_fini()
      -> PID 926526 cannot reach pcc_io_fini() because it is waiting for the
      mmap_sem held by PID 926158
      -> back to PID 926526

      Therefore the vmcore evidence is consistent with this circular wait:

      PID 926526 -> mmap_sem -> PID 926158 -> pcci_active_ios -> PID 926526.

      More precisely, PID 926526 waits for PID 926158 to release mmap_sem, while
      PID 926158 waits for PID 926526's PCC I/O to complete so that
      pcci_active_ios changes from 1 to 0. This conclusion is limited to the facts
      demonstrated by the crash output and disassembly reproduced above.

            wc-triage WC Triage
            qian_wc Qian Yingjin
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated: