Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-21101

lockdep reports in osd-ldiskfs, LNet, lnet_selftest and llite on master

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Medium Medium
    • None
    • None
    • None
    • 3
    • 9223372036854775807

      Problem

      A lockdep kernel shows 5 lock order reports on master. Each report needs investigation. Some reports can be real deadlocks. Others can need only a lockdep annotation, for example a nested lock class.

      Lockdep stops after its first report on a node. S1 occurs at each MGS mount. Thus S1 stops lockdep on every server at mount time. Lockdep cannot check server code under Lustre load until a change fixes or annotates S1.

      I found these reports while I tested LU-21085. All 5 reports are in master code, and LU-21085 does not change that code. The same reports occurred on master without LU-21085.

      Environment

      • master at commit 40b80bc048 (LU-20867).
      • Rocky Linux 9.7 kernel 5.14.0-611.55.1.el9_7, aarch64, built with: CONFIG_LOCKDEP=y, CONFIG_DEBUG_LOCK_ALLOC=y, CONFIG_PROVE_LOCKING=y, CONFIG_DEBUG_SPINLOCK=y, CONFIG_DEBUG_ATOMIC_SLEEP=y and CONFIG_PROVE_RCU=y.
      • VMs with 2 vCPUs and 3 GB each: one server (MGS, MDS and OSS, ldiskfs) and two clients, on tcp.

      Reports

      ID Node When Type Lock classes
      S1 Server Each MGS mount Circular i_mutex_dir_key, jbd2_handle, oo_dt.dd_sem
      S2 Server Target mount after a hard failover (recovery-small test_136) Circular i_mutex_key, jbd2_handle, oo_dt.dd_sem
      S3 Client sanity-lnet test_205 to test_219, peer merge Recursive lp_lock
      S4 Client lnet_selftest unload, with 2 or more CPTs Recursive scd_lock
      S5 Client umount, recovery-small test_102 Recursive lli_lsm_sem

      S1: circular locking at MGS mount

      mount.lustre_tg is trying to acquire &type->i_mutex_dir_key#7 at osd_obj_add_entry+0x1c0
      but task is already holding &mo->oo_dt.dd_sem at dt_write_lock+0x188
      #0 i_mutex_dir_key: down_write <- osd_obj_add_entry <- osd_obj_spec_insert <- osd_oi_insert
         <- __osd_oi_insert <- osd_create <- dt_create <- local_oid_storage_init <- mgs_fs_setup
         <- mgs_init0 <- mgs_device_alloc <- obd_setup <- class_setup <- lustre_start_simple
         <- server_start_mgs <- server_fill_super <- lustre_tgt_fill_super <- vfs_get_tree <- mount
      #1 jbd2_handle: start_this_handle <- jbd2__journal_start <- __ldiskfs_new_inode
         <- ldiskfs_create <- lookup_open <- filp_open <- osd_scrub_setup <- osd_device_init0
         <- osd_device_alloc <- ... <- osd_start <- server_fill_super
      #2 dd_sem: down_write_nested <- dt_write_lock <- local_oid_storage_init <- mgs_fs_setup <- ...
      held: s_umount, ls_los_mutex, sb_writers, oo_ext_idx_sem, jbd2_handle, oo_dt.dd_sem

      Lockdep uses one class for every dd_sem and one class for every directory i_mutex. Thus the chain can link different objects. Investigate whether two paths can take these locks on the same objects in opposite orders.

      S2: circular locking at target mount after a failover

      mount.lustre_tg is trying to acquire &sb->s_type->i_mutex_key#17 at osd_getnblk+0x34c
      but task is already holding &mo->oo_dt.dd_sem at dt_write_lock+0x188
      #0 i_mutex_key: down_write <- osd_getnblk <- osd_ldiskfs_write_fast <- osd_write
         <- dt_record_write <- llog_osd_write_rec <- llog_write_rec <- llog_write_cookie
         <- llog_copy_handler <- llog_process_thread <- llog_process_or_fork <- llog_backup
         <- mgc_llog_local_copy <- mgc_process_server_cfg_log <- mgc_process_cfg_log
         <- mgc_process_log <- mgc_process_config <- lustre_process_log <- server_start_targets
      #1 jbd2_handle: start_this_handle <- __ldiskfs_journal_start_sb <- ldiskfs_truncate
         <- ldiskfs_process_orphan <- ldiskfs_orphan_cleanup <- __ldiskfs_fill_super
         <- vfs_kern_mount <- osd_mount <- osd_device_init0 <- ... <- osd_start
      #2 dd_sem: down_write_nested <- dt_write_lock <- llog_osd_create <- llog_create
         <- llog_open_create <- llog_backup <- mgc_llog_local_copy <- ...

      S2 has the same pattern as S1. The orphan cleanup at mount gives the jbd2_handle link.

      S3: recursive lp_lock in lnet_peer_data_present()

      lnet_discovery is trying to acquire &lp->lp_lock at lnet_peer_data_present+0x1f4
      but task is already holding &lp->lp_lock at lnet_peer_data_present+0x1ec
      held: the_lnet.ln_api_mutex, lp_lock
      _raw_spin_lock <- lnet_peer_data_present <- lnet_peer_discovery <- kthread

      When lnet_peer_data_present() merges lp into new_lp, it locks lp->lp_lock and then new_lp->lp_lock. LU-13894 added this code. The two locks are in one class, and the code does not use spin_lock_nested(). Investigate whether two threads can lock the same two peers in opposite orders. If they cannot, add a lockdep subclass.

      S4: recursive scd_lock in srpc_shutdown_service()

      rmmod is trying to acquire &scd->scd_lock at srpc_shutdown_service+0x50
      but task is already holding &scd->scd_lock at srpc_shutdown_service+0x50
      held: console_session.ses_mutex, scd_lock
      _raw_spin_lock <- srpc_shutdown_service <- lstcon_console_fini <- lnet_selftest_exit
      <- __do_sys_delete_module

      srpc_shutdown_service() locks the scd_lock of each CPT in a loop, and holds all of them at the same time. The loop always uses the same order. Lockdep reports S4 only when libcfs has 2 or more CPTs.

      S5: recursive lli_lsm_sem at umount

      umount is trying to acquire &lli->lli_lsm_sem at ll_dir_clear_lsm_md+0x34
      but task is already holding &lli->lli_lsm_sem at ll_dir_clear_lsm_md+0x34
      held: s_umount, lli_lsm_sem
      down_write <- ll_dir_clear_lsm_md <- ll_clear_inode <- ll_delete_inode <- evict <- iput_final
      <- iput <- lmv_stripe_object_free <- lmv_stripe_object_put <- ll_dir_clear_lsm_md
      <- ll_clear_inode <- ll_delete_inode <- evict <- dispose_list <- evict_inodes
      <- generic_shutdown_super <- kill_anon_super <- lustre_kill_super

      A striped directory inode releases the last reference to a stripe inode while it holds its own lli_lsm_sem. The stripe inode then takes its own lli_lsm_sem. Investigate whether a stripe inode can also be a striped directory with a reference back to its parent.

      Requested work

      For each report:

      1. Find whether a real deadlock is possible.
      2. Fix the lock order, or add a lockdep annotation with a comment that says why the order is safe.

      Start with S1. S1 stops lockdep on every server, and it hides all other server reports.

            wc-triage WC Triage
            hornc Chris Horn
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: