Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-21202

(dt_object.c:533:dt_record_write()) ASSERTION( dt->do_body_ops->dbo_write )

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Medium Medium
    • None
    • None
    • None
    • 3
    • 9223372036854775807

      Summary:
      MDT LBUG in dt_record_write() on an update catalog deleted while in use

      Component: lod / DNE
      Affects: master (2.17.61-84-g7c3c9f8814)

      Description:

      In a DNE filesystem, an MDT crashes when another MDT writes to its
      update-log catalog on it.  On a production system, MDT0032 crashed in
      the same second that its recovery finished:

      Nov 10 20:44:38 Lustre: lustre-MDT0032: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-900
      Nov 10 20:44:38 Lustre: lustre-MDT0032: in recovery but waiting for the first client to connect
      Nov 10 20:44:39 Lustre: lustre-MDT0032: Will be in recovery for at least 2:30, or until 1876 clients reconnect
      Nov 10 20:46:46 Lustre: lustre-MDT0032: Recovery over after 2:07, of 1876 clients 1876 recovered and 0 were evicted.
      Nov 10 20:46:46 LustreError: 36806:0:(dt_object.c:533:dt_record_write()) ASSERTION( dt->do_body_ops->dbo_write ) failed: [0x1180000441:0x1:0x0]
      Nov 10 20:46:46 LustreError: 36806:0:(dt_object.c:533:dt_record_write()) LBUG
      Nov 10 20:46:46 CPU: 17 PID: 36806 Comm: ll_ost_out04_00 Kdump: loaded Tainted: G           OE     -------- -  - 4.18.0-553.158.1.el8_10.x86_64 #1
      Nov 10 20:46:46 Call Trace:
      Nov 10 20:46:46  dump_stack+0x41/0x60
      Nov 10 20:46:46  lbug_with_loc.cold.8+0x5/0x43 [libcfs]
      Nov 10 20:46:46  dt_record_write+0x11c/0x120 [obdclass]
      Nov 10 20:46:46  out_tx_write_exec+0x92/0x390 [ptlrpc]
      Nov 10 20:46:46  out_tx_end+0x166/0x5d0 [ptlrpc]
      Nov 10 20:46:46  out_handle+0x16c0/0x20b0 [ptlrpc]
      Nov 10 20:46:46  ? lustre_msg_get_opc+0x7b/0xf0 [ptlrpc]
      Nov 10 20:46:46  tgt_request_handle+0xc09/0x1b90 [ptlrpc]
      Nov 10 20:46:46  ? ptlrpc_update_export_timer+0x3d/0x500 [ptlrpc]
      Nov 10 20:46:46  ptlrpc_server_handle_request+0x27f/0x1200 [ptlrpc]
      Nov 10 20:46:46  ? lprocfs_counter_add+0x10e/0x180 [obdclass]
      Nov 10 20:46:46  ptlrpc_main+0xb6f/0x13d0 [ptlrpc]
      Nov 10 20:46:46  ? __schedule+0x2d9/0x870
      Nov 10 20:46:46  ? ptlrpc_register_service+0xf20/0xf20 [ptlrpc]
      Nov 10 20:46:46  kthread+0x134/0x150
      Nov 10 20:46:46  ? set_kthread_struct+0x50/0x50
      Nov 10 20:46:46  ret_from_fork+0x1f/0x40
      Nov 10 20:46:46 Kernel panic - not syncing: LBUG
      

      [0x1180000441:0x1:0x0] is MDT0000's update catalog, which is stored
      on MDT0032.  After recovery, MDT0000 resends the update, so MDT0032
      crashes again each time it restarts, until MDT0000 is restarted.

      Master hits the same assertion (dt_object.c:554, in mdt_out00_000)
      with the reproducer below.

            diwa Di Wang
            diwa Di Wang
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: