-
Bug
-
Resolution: Unresolved
-
Medium
-
None
-
None
-
None
-
3
-
9223372036854775807
Summary:
MDT LBUG in dt_record_write() on an update catalog deleted while in use
Component: lod / DNE
Affects: master (2.17.61-84-g7c3c9f8814)
Description:
In a DNE filesystem, an MDT crashes when another MDT writes to its
update-log catalog on it. On a production system, MDT0032 crashed in
the same second that its recovery finished:
Nov 10 20:44:38 Lustre: lustre-MDT0032: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-900 Nov 10 20:44:38 Lustre: lustre-MDT0032: in recovery but waiting for the first client to connect Nov 10 20:44:39 Lustre: lustre-MDT0032: Will be in recovery for at least 2:30, or until 1876 clients reconnect Nov 10 20:46:46 Lustre: lustre-MDT0032: Recovery over after 2:07, of 1876 clients 1876 recovered and 0 were evicted. Nov 10 20:46:46 LustreError: 36806:0:(dt_object.c:533:dt_record_write()) ASSERTION( dt->do_body_ops->dbo_write ) failed: [0x1180000441:0x1:0x0] Nov 10 20:46:46 LustreError: 36806:0:(dt_object.c:533:dt_record_write()) LBUG Nov 10 20:46:46 CPU: 17 PID: 36806 Comm: ll_ost_out04_00 Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.158.1.el8_10.x86_64 #1 Nov 10 20:46:46 Call Trace: Nov 10 20:46:46 dump_stack+0x41/0x60 Nov 10 20:46:46 lbug_with_loc.cold.8+0x5/0x43 [libcfs] Nov 10 20:46:46 dt_record_write+0x11c/0x120 [obdclass] Nov 10 20:46:46 out_tx_write_exec+0x92/0x390 [ptlrpc] Nov 10 20:46:46 out_tx_end+0x166/0x5d0 [ptlrpc] Nov 10 20:46:46 out_handle+0x16c0/0x20b0 [ptlrpc] Nov 10 20:46:46 ? lustre_msg_get_opc+0x7b/0xf0 [ptlrpc] Nov 10 20:46:46 tgt_request_handle+0xc09/0x1b90 [ptlrpc] Nov 10 20:46:46 ? ptlrpc_update_export_timer+0x3d/0x500 [ptlrpc] Nov 10 20:46:46 ptlrpc_server_handle_request+0x27f/0x1200 [ptlrpc] Nov 10 20:46:46 ? lprocfs_counter_add+0x10e/0x180 [obdclass] Nov 10 20:46:46 ptlrpc_main+0xb6f/0x13d0 [ptlrpc] Nov 10 20:46:46 ? __schedule+0x2d9/0x870 Nov 10 20:46:46 ? ptlrpc_register_service+0xf20/0xf20 [ptlrpc] Nov 10 20:46:46 kthread+0x134/0x150 Nov 10 20:46:46 ? set_kthread_struct+0x50/0x50 Nov 10 20:46:46 ret_from_fork+0x1f/0x40 Nov 10 20:46:46 Kernel panic - not syncing: LBUG
[0x1180000441:0x1:0x0] is MDT0000's update catalog, which is stored
on MDT0032. After recovery, MDT0000 resends the update, so MDT0032
crashes again each time it restarts, until MDT0000 is restarted.
Master hits the same assertion (dt_object.c:554, in mdt_out00_000)
with the reproducer below.