[LU-18409] sanity test_136: (llog_osd.c:630:llog_osd_write_rec()) LBUG

• Type: Bug
• Resolution: Unresolved
• Priority: Minor
• Fix Version/s: Lustre 2.17.0
• Affects Version/s: Lustre 2.16.0
• Severity: 3
• Assignee: WC Triage
• Reporter: Maloo

Description

This issue was created by maloo for jianyu <[e-mail]>

This issue relates to the following test suite run: https://testing.whamcloud.com/test_sets/bc09d757-d8f6-4fe4-be89-74facbb80e27

test_136 failed with the following error:

onyx-140vm12 crashed during sanity test_136

Test session details:
clients: https://build.whamcloud.com/job/lustre-master/4588 - 4.18.0-553.16.1.el8_10.x86_64
servers: https://build.whamcloud.com/job/lustre-master/4588 - 4.18.0-553.16.1.el8_lustre.x86_64

Lustre: DEBUG MARKER: == sanity test 136: Race catalog processing 2 ============ 06:08:28 (1730009308)
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x131a fail_val=1
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n debug
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n debug=0
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n debug=trace+inode+super+iotrace+malloc+cache+info+ioctl+neterror+net+warning+buffs+other+dentry+nettrace+page+dlmtrace+error+emerg+ha+rpctrace+vfstrace+reada+mmap+config+console+quota+sec+lfsck+hsm+snapshot+layout
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n debug
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n debug=0
Lustre: 342757:0:(llog_cat.c:97:llog_cat_new_log()) lustre-OST0000-osc-MDT0000: there are no more free slots in catalog [0x1:0x2:0x0]
Lustre: 424765:0:(llog_cat.c:97:llog_cat_new_log()) lustre-OST0000-osc-MDT0000: there are no more free slots in catalog [0x1:0x2:0x0]
Lustre: 424765:0:(llog_cat.c:97:llog_cat_new_log()) Skipped 520 previous similar messages
Lustre: 344084:0:(llog_cat.c:97:llog_cat_new_log()) lustre-OST0000-osc-MDT0000: there are no more free slots in catalog [0x1:0x2:0x0]
Lustre: 344084:0:(llog_cat.c:97:llog_cat_new_log()) Skipped 1073 previous similar messages
Lustre: 344084:0:(llog_cat.c:97:llog_cat_new_log()) lustre-OST0000-osc-MDT0000: there are no more free slots in catalog [0x1:0x2:0x0]
Lustre: 344084:0:(llog_cat.c:97:llog_cat_new_log()) Skipped 1705 previous similar messages
LustreError: 424765:0:(llog_osd.c:628:llog_osd_write_rec()) lustre-MDT0000-osd: index 62096 already set in llog bitmap [0x1:0x2:0x0]
LustreError: 424765:0:(llog_osd.c:630:llog_osd_write_rec()) LBUG
CPU: 1 PID: 424765 Comm: mdt00_004 Kdump: loaded Tainted: G        W  OE     -------- -  - 4.18.0-553.16.1.el8_lustre.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
 dump_stack+0x41/0x60
 lbug_with_loc.cold.8+0x5/0x58 [libcfs]
 llog_osd_write_rec+0x1be4/0x1d80 [obdclass]
 llog_write_rec+0x41e/0x560 [obdclass]
 llog_cat_new_log+0x229/0xea0 [obdclass]
 llog_cat_add_rec+0x112/0x9f0 [obdclass]
 llog_add+0x19d/0x1f0 [obdclass]
 osp_sync_add+0x2c2/0x730 [osp]
 ? lod_fld_lookup+0x437/0x500 [lod]
 osp_destroy+0x16e/0x2d0 [osp]
 lod_sub_destroy+0x226/0x4c0 [lod]
 ? osd_destroy+0x138/0x510 [osd_ldiskfs]
 lod_obj_for_each_stripe+0x126/0x2e0 [lod]
 lod_destroy+0x6c7/0x9f0 [lod]
 ? lod_attr_get+0x1a0/0x1a0 [lod]
 mdd_finish_unlink+0x8b/0x2b0 [mdd]
 mdd_unlink+0x7f8/0xe20 [mdd]
 mdt_reint_unlink+0xbe7/0x1140 [mdt]
 mdt_reint_rec+0x123/0x270 [mdt]
 mdt_reint_internal+0x4b9/0x810 [mdt]
 mdt_reint+0x5d/0x110 [mdt]
 tgt_request_handle+0x3f4/0x1a30 [ptlrpc]
 ? ptlrpc_update_export_timer+0x3d/0x500 [ptlrpc]
 ptlrpc_server_handle_request+0x2aa/0xcf0 [ptlrpc]
 ? lprocfs_counter_add+0x10e/0x180 [obdclass]
 ptlrpc_main+0xc9e/0x15c0 [ptlrpc]
 ? __schedule+0x2d9/0x870
 ? ptlrpc_wait_event+0x5b0/0x5b0 [ptlrpc]
 kthread+0x134/0x150
 ? set_kthread_struct+0x50/0x50
 ret_from_fork+0x35/0x40
Kernel panic - not syncing: LBUG
CPU: 1 PID: 424765 Comm: mdt00_004 Kdump: loaded Tainted: G        W  OE     -------- -  - 4.18.0-553.16.1.el8_lustre.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
 dump_stack+0x41/0x60
 panic+0xe7/0x2ac
 lbug_with_loc.cold.8+0x2f/0x58 [libcfs]
 llog_osd_write_rec+0x1be4/0x1d80 [obdclass]
 llog_write_rec+0x41e/0x560 [obdclass]
 llog_cat_new_log+0x229/0xea0 [obdclass]
 llog_cat_add_rec+0x112/0x9f0 [obdclass]
 llog_add+0x19d/0x1f0 [obdclass]
 osp_sync_add+0x2c2/0x730 [osp]
 ? lod_fld_lookup+0x437/0x500 [lod]
 osp_destroy+0x16e/0x2d0 [osp]
 lod_sub_destroy+0x226/0x4c0 [lod]
 ? osd_destroy+0x138/0x510 [osd_ldiskfs]
 lod_obj_for_each_stripe+0x126/0x2e0 [lod]
 lod_destroy+0x6c7/0x9f0 [lod]
 ? lod_attr_get+0x1a0/0x1a0 [lod]
 mdd_finish_unlink+0x8b/0x2b0 [mdd]
 mdd_unlink+0x7f8/0xe20 [mdd]
 mdt_reint_unlink+0xbe7/0x1140 [mdt]
 mdt_reint_rec+0x123/0x270 [mdt]
 mdt_reint_internal+0x4b9/0x810 [mdt]
 mdt_reint+0x5d/0x110 [mdt]
 tgt_request_handle+0x3f4/0x1a30 [ptlrpc]
 ? ptlrpc_update_export_timer+0x3d/0x500 [ptlrpc]
 ptlrpc_server_handle_request+0x2aa/0xcf0 [ptlrpc]
 ? lprocfs_counter_add+0x10e/0x180 [obdclass]
 ptlrpc_main+0xc9e/0x15c0 [ptlrpc]
 ? __schedule+0x2d9/0x870
 ? ptlrpc_wait_event+0x5b0/0x5b0 [ptlrpc]
 kthread+0x134/0x150
 ? set_kthread_struct+0x50/0x50
 ret_from_fork+0x35/0x40

VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV
sanity test_136 - onyx-140vm12 crashed during sanity test_136