[LU-10143] LBUG dt_object.h:2166:dt_declare_record_write

• Type: Bug
• Resolution: Duplicate
• Priority: Minor
• Fix Version/s: Lustre 2.13.0, Lustre 2.10.7, Lustre 2.12.1
• Affects Version/s: Lustre 2.11.0, Lustre 2.12.0, Lustre 2.10.5, Lustre 2.10.6
• Environment: trevis, full DNE; servers: CentOS7.4, zfs, branch master, v2.10.54, b3652; clients: CentOS7.4, branch master, v2.10.54, b3652
• Severity: 3
• Assignee: Alex Zhuravlev
• Reporter: James Casper

Description

https://testing.hpdd.intel.com/test_sessions/2f6027c4-8e9a-4949-af3d-2f8d04940c9c

replay-single, test_80d: Timeout occurred after 155 mins, last suite running was replay-single, restarting cluster to continue tests LBUG

From mds console:

[ 5748.826345] LustreError: 28616:0:(dt_object.h:2166:dt_declare_record_write()) ASSERTION( dt->do_body_ops ) failed: 
[ 5748.828884] LustreError: 28616:0:(dt_object.h:2166:dt_declare_record_write()) LBUG
[ 5748.831194] Pid: 28616, comm: mdt_out00_003
[ 5748.833233] 
[ 5748.833233] Call Trace:
[ 5748.836889]  [<ffffffffc06917ae>] libcfs_call_trace+0x4e/0x60 [libcfs]
[ 5748.839015]  [<ffffffffc069183c>] lbug_with_loc+0x4c/0xb0 [libcfs]
[ 5748.841118]  [<ffffffffc0ef1743>] out_write_add_exec+0x133/0x1b0 [ptlrpc]
[ 5748.843217]  [<ffffffffc0ee84a3>] out_write+0x333/0x370 [ptlrpc]
[ 5748.845224]  [<ffffffffc0eeb1c4>] out_handle+0x1304/0x1920 [ptlrpc]
[ 5748.847209]  [<ffffffffc0e7d4a2>] ? lustre_msg_get_opc+0x22/0xf0 [ptlrpc]
[ 5748.849234]  [<ffffffffc0ee0d49>] ? tgt_request_preprocess.isra.26+0x299/0x7a0 [ptlrpc]
[ 5748.851305]  [<ffffffffc0ee2475>] tgt_request_handle+0x925/0x1370 [ptlrpc]
[ 5748.853263]  [<ffffffffc0e8b37e>] ptlrpc_server_handle_request+0x24e/0xab0 [ptlrpc]
[ 5748.855236]  [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
[ 5748.857056]  [<ffffffffc0e8eb22>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
[ 5748.858924]  [<ffffffffc0e8e090>] ? ptlrpc_main+0x0/0x1e40 [ptlrpc]
[ 5748.860737]  [<ffffffff810b098f>] kthread+0xcf/0xe0
[ 5748.862471]  [<ffffffff810b08c0>] ? kthread+0x0/0xe0
[ 5748.864195]  [<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[ 5748.865926]  [<ffffffff810b08c0>] ? kthread+0x0/0xe0
[ 5748.867590] 
[ 5748.868971] Kernel panic - not syncing: LBUG
[ 5748.869962] CPU: 1 PID: 28616 Comm: mdt_out00_003 Tainted: P           OE  ------------   3.10.0-693.2.2.el7_lustre.x86_64 #1
[ 5748.869962] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007
[ 5748.869962]  ffff88007c7daf00 000000006e5a06dd ffff88005feefa88 ffffffff816a3d8d
[ 5748.869962]  ffff88005feefb08 ffffffff8169dc74 ffffffff00000008 ffff88005feefb18
[ 5748.869962]  ffff88005feefab8 000000006e5a06dd 000000006e5a06dd ffff88007fd0f8b8
[ 5748.869962] Call Trace:
[ 5748.869962]  [<ffffffff816a3d8d>] dump_stack+0x19/0x1b
[ 5748.869962]  [<ffffffff8169dc74>] panic+0xe8/0x20d
[ 5748.869962]  [<ffffffffc0691854>] lbug_with_loc+0x64/0xb0 [libcfs]
[ 5748.869962]  [<ffffffffc0ef1743>] out_write_add_exec+0x133/0x1b0 [ptlrpc]
[ 5748.869962]  [<ffffffffc0ee84a3>] out_write+0x333/0x370 [ptlrpc]
[ 5748.869962]  [<ffffffffc0eeb1c4>] out_handle+0x1304/0x1920 [ptlrpc]
[ 5748.869962]  [<ffffffffc0e7d4a2>] ? lustre_msg_get_opc+0x22/0xf0 [ptlrpc]
[ 5748.869962]  [<ffffffffc0ee0d49>] ? tgt_request_preprocess.isra.26+0x299/0x7a0 [ptlrpc]
[ 5748.893323] Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1
[ 5748.893798]  [<ffffffffc0ee2475>] tgt_request_handle+0x925/0x1370 [ptlrpc]
[ 5748.893798]  [<ffffffffc0e8b37e>] ptlrpc_server_handle_request+0x24e/0xab0 [ptlrpc]
[ 5748.893798]  [<ffffffff810ba588>] ? __wake_up_common+0x58/0x90
[ 5748.893798]  [<ffffffffc0e8eb22>] ptlrpc_main+0xa92/0x1e40 [ptlrpc]
[ 5748.893798]  [<ffffffffc0e8e090>] ? ptlrpc_register_service+0xe80/0xe80 [ptlrpc]
[ 5748.893798]  [<ffffffff810b098f>] kthread+0xcf/0xe0
[ 5748.893798]  [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[ 5748.893798]  [<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[ 5748.893798]  [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40

Issue Links

• is duplicated by: LU-9157 replay-single test_80c: rmdir failed
• is duplicated by: LU-10740 replay-single test_2d: FAIL: checkstat -v /mnt/lustre/d2d.replay-single check failed
• is duplicated by: LU-11538 replay-single test 80g fails with '/usr/bin/lfs getstripe -m /mnt/lustre/d80g.replay-single/remote_dir failed'
• is duplicated by: LU-13195 replay-single test_118: dt_declare_record_write() ASSERTION( dt->do_body_ops ) failed
• is related to: LU-7298 replay-single test_70b: ASSERTION( dt->do_body_ops->dbo_write ) failed
• is related to: LU-9924 LBUG dt_object.c:513:dt_record_write
• is related to: LU-11366 replay-single timeout test 80f: rm: cannot remove '/mnt/lustre/d80f.replay-single/remote_dir': Input/output error