[LU-19647] sanityn test_80b: osd_destroy() - ASSERTION( !lu_object_is_dying(dt->do_lu.lo_header) ) failed

• Type: Bug
• Resolution: Unresolved
• Priority: Medium
• Severity: 3
• Assignee: WC Triage
• Reporter: Maloo

Description

This issue was created by maloo for Marc Vef <[e-mail]>

This issue relates to the following test suite run: https://testing.whamcloud.com/test_sets/0c86883e-0cad-4c95-9d46-c4eb4bf52341

test_80b failed with the following error:

trevis-156vm23 crashed during sanityn test_80b

Test session details:
clients: https://build.whamcloud.com/job/lustre-reviews/119044 - 5.14.0-503.40.1.el9_5.x86_64
servers: https://build.whamcloud.com/job/lustre-reviews/119044 - 5.14.0-503.40.1_lustre.el9.x86_64

[12105.410124] Lustre: DEBUG MARKER: == sanityn test 80b: Accessing directory during migration ========================================================== 18:02:04 (1764266524)
[12105.457049] Lustre: 332378:0:(mdd_dir.c:4823:mdd_migrate_object()) lustre-MDD0001: [0x2400032e0:0x13:0x0]/file3 is open, migrate only dentry
[12105.559668] LustreError: 332378:0:(mdt_reint.c:2562:mdt_reint_migrate()) lustre-MDT0003: migrate [0x2c0000bd0:0x9:0x0]/source_file failed: rc = -2
[12105.946043] LustreError: 332378:0:(mdd_dir.c:4740:mdd_migrate_cmd_check()) lustre-MDD0003: 'migrate_dir' migration was interrupted, run 'lfs migrate -m 0 -c 1 -H crush migrate_dir' to finish migration: rc = -1
[12106.423203] LustreError: 332378:0:(mdt_reint.c:2562:mdt_reint_migrate()) lustre-MDT0003: migrate [0x200016b61:0x51e:0x0]/migrate_dir failed: rc = -1
[12106.423303] LustreError: 332378:0:(mdt_reint.c:2562:mdt_reint_migrate()) Skipped 6 previous similar messages
[12106.449297] LustreError: 332378:0:(mdd_dir.c:4740:mdd_migrate_cmd_check()) lustre-MDD0001: 'migrate_dir' migration was interrupted, run 'lfs migrate -m 2 -c 1 -H crush migrate_dir' to finish migration: rc = -1
[12106.449533] LustreError: 332378:0:(mdd_dir.c:4740:mdd_migrate_cmd_check()) Skipped 6 previous similar messages
[12108.104750] LustreError: 332377:0:(mdd_dir.c:4740:mdd_migrate_cmd_check()) lustre-MDD0001: 'migrate_dir' migration was interrupted, run 'lfs migrate -m 3 -c 1 -H crush migrate_dir' to finish migration: rc = -1
[12108.104835] LustreError: 332377:0:(mdd_dir.c:4740:mdd_migrate_cmd_check()) Skipped 3 previous similar messages
[12108.107979] LustreError: 332377:0:(mdt_reint.c:2562:mdt_reint_migrate()) lustre-MDT0001: migrate [0x200016b61:0x51e:0x0]/migrate_dir failed: rc = -1
[12108.108133] LustreError: 332377:0:(mdt_reint.c:2562:mdt_reint_migrate()) Skipped 5 previous similar messages
[12110.369763] LustreError: 340569:0:(mdd_dir.c:4740:mdd_migrate_cmd_check()) lustre-MDD0003: 'migrate_dir' migration was interrupted, run 'lfs migrate -m 0 -c 1 -H crush migrate_dir' to finish migration: rc = -1
[12110.369964] LustreError: 340569:0:(mdd_dir.c:4740:mdd_migrate_cmd_check()) Skipped 13 previous similar messages
[12110.370198] LustreError: 340569:0:(mdt_reint.c:2562:mdt_reint_migrate()) lustre-MDT0003: migrate [0x200016b61:0x51e:0x0]/migrate_dir failed: rc = -1
[12110.370365] LustreError: 340569:0:(mdt_reint.c:2562:mdt_reint_migrate()) Skipped 13 previous similar messages
[12114.079265] Lustre: 332377:0:(mdd_dir.c:4823:mdd_migrate_object()) lustre-MDD0001: [0x2400032e1:0x25:0x0]/file4 is open, migrate only dentry
[12114.079345] Lustre: 332377:0:(mdd_dir.c:4823:mdd_migrate_object()) Skipped 2 previous similar messages
[12116.132443] LustreError: 332377:0:(mdd_dir.c:4740:mdd_migrate_cmd_check()) lustre-MDD0001: 'migrate_dir' migration was interrupted, run 'lfs migrate -m 3 -c 1 -H crush migrate_dir' to finish migration: rc = -1
[12116.132544] LustreError: 332377:0:(mdd_dir.c:4740:mdd_migrate_cmd_check()) Skipped 29 previous similar messages
[12116.133444] LustreError: 332377:0:(mdt_reint.c:2562:mdt_reint_migrate()) lustre-MDT0001: migrate [0x200016b61:0x51e:0x0]/migrate_dir failed: rc = -1
[12116.133567] LustreError: 332377:0:(mdt_reint.c:2562:mdt_reint_migrate()) Skipped 29 previous similar messages
[12116.222251] LustreError: 332378:0:(osd_handler.c:4776:osd_ref_del()) lustre-MDT0003: nlink == 0 on [0x2c0000bd5:0xe9:0x0], maybe an upgraded file? (LU-3915)
[12119.307656] LustreError: 340569:0:(osd_handler.c:4776:osd_ref_del()) lustre-MDT0003: nlink == 0 on [0x2c0000bd5:0x125:0x0], maybe an upgraded file? (LU-3915)
[12124.260966] Lustre: 340569:0:(mdd_dir.c:4823:mdd_migrate_object()) lustre-MDD0003: [0x2c0000bd1:0x44:0x0]/file4 is open, migrate only dentry
[12125.200734] LustreError: 340569:0:(mdd_dir.c:4740:mdd_migrate_cmd_check()) lustre-MDD0001: 'migrate_dir' migration was interrupted, run 'lfs migrate -m 3 -c 1 -H crush migrate_dir' to finish migration: rc = -1
[12125.200946] LustreError: 340569:0:(mdd_dir.c:4740:mdd_migrate_cmd_check()) Skipped 58 previous similar messages
[12125.201474] LustreError: 340569:0:(mdt_reint.c:2562:mdt_reint_migrate()) lustre-MDT0001: migrate [0x200016b61:0x51e:0x0]/migrate_dir failed: rc = -1
[12125.201629] LustreError: 340569:0:(mdt_reint.c:2562:mdt_reint_migrate()) Skipped 61 previous similar messages
[12132.327688] Lustre: 340572:0:(mdd_dir.c:4823:mdd_migrate_object()) lustre-MDD0001: [0x200016b61:0x51e:0x0]/migrate_dir is open, migrate only dentry
[12136.362572] Lustre: 340569:0:(mdd_dir.c:4823:mdd_migrate_object()) lustre-MDD0001: [0x200016b61:0x51e:0x0]/migrate_dir is open, migrate only dentry
[12141.373867] LustreError: 332377:0:(mdd_dir.c:4740:mdd_migrate_cmd_check()) lustre-MDD0001: 'migrate_dir' migration was interrupted, run 'lfs migrate -m 2 -c 1 -H crush migrate_dir' to finish migration: rc = -1
[12141.374123] LustreError: 332377:0:(mdd_dir.c:4740:mdd_migrate_cmd_check()) Skipped 75 previous similar messages
[12141.374594] LustreError: 332377:0:(mdt_reint.c:2562:mdt_reint_migrate()) lustre-MDT0001: migrate [0x200016b61:0x51e:0x0]/migrate_dir failed: rc = -1
[12141.374739] LustreError: 332377:0:(mdt_reint.c:2562:mdt_reint_migrate()) Skipped 85 previous similar messages
[12160.332059] LustreError: 340572:0:(osd_handler.c:4776:osd_ref_del()) lustre-MDT0003: nlink == 0 on [0x2c0000bd5:0x504:0x0], maybe an upgraded file? (LU-3915)
[12160.332268] LustreError: 340572:0:(osd_handler.c:4158:osd_destroy()) ASSERTION( !lu_object_is_dying(dt->do_lu.lo_header) ) failed: 
[12160.332461] LustreError: 340572:0:(osd_handler.c:4158:osd_destroy()) LBUG
[12160.332613] CPU: 1 PID: 340572 Comm: mdt_io00_004 Kdump: loaded Tainted: G           OE     -------  ---  5.14.0-503.40.1_lustre.el9.x86_64 #1
[12160.332749] Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-2.el9_5.1 04/01/2014
[12160.332910] Call Trace:
[12160.333078]  <TASK>
[12160.333229]  dump_stack_lvl+0x34/0x48
[12160.333391]  lbug_with_loc.cold+0x5/0x43 [libcfs]
[12160.333566]  osd_destroy+0x4b8/0x630 [osd_ldiskfs]
[12160.333764]  lod_sub_destroy+0x1de/0x4b0 [lod]
[12160.333942]  lod_destroy+0x26c/0xb60 [lod]
[12160.334134]  ? srso_alias_return_thunk+0x5/0xfbef5
[12160.334293]  ? lod_sub_ref_del+0x1de/0x4b0 [lod]
[12160.334487]  mdd_migrate_create+0x3f9/0x8f0 [mdd]
[12160.334672]  mdd_migrate_object+0xef5/0x1170 [mdd]
[12160.334843]  ? srso_alias_return_thunk+0x5/0xfbef5
[12160.334988]  ? lprocfs_counter_add+0x117/0x180 [obdclass]
[12160.335216]  ? mdd_migrate+0x14/0x20 [mdd]
[12160.335396]  mdd_migrate+0x14/0x20 [mdd]
[12160.335549]  mdt_reint_migrate+0x1381/0x1e00 [mdt]
[12160.335748]  ? __entry_text_end+0x102749/0x10274d
[12160.335909]  mdt_reint_rec+0x119/0x270 [mdt]
[12160.336091]  mdt_reint_internal+0x4ea/0x9b0 [mdt]
[12160.336278]  mdt_reint+0x59/0x110 [mdt]
[12160.336432]  tgt_handle_request0+0x147/0x770 [ptlrpc]
[12160.336703]  tgt_request_handle+0x3fd/0xd00 [ptlrpc]
[12160.336960]  ptlrpc_server_handle_request.isra.0+0x2e5/0xd80 [ptlrpc]
[12160.337192]  ? srso_alias_return_thunk+0x5/0xfbef5
[12160.337364]  ptlrpc_main+0x9bf/0xea0 [ptlrpc]
[12160.337616]  ? __pfx_ptlrpc_main+0x10/0x10 [ptlrpc]
[12160.337845]  kthread+0xdd/0x100
[12160.338007]  ? __pfx_kthread+0x10/0x10
[12160.338156]  ret_from_fork+0x29/0x50
[12160.338292]  </TASK>
[12160.338472] Kernel panic - not syncing: LBUG
[12160.359974] CPU: 1 PID: 340572 Comm: mdt_io00_004 Kdump: loaded Tainted: G           OE     -------  ---  5.14.0-503.40.1_lustre.el9.x86_64 #1
[12160.361772] Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-2.el9_5.1 04/01/2014
[12160.362766] Call Trace:
[12160.363110]  <TASK>
[12160.363426]  dump_stack_lvl+0x34/0x48
[12160.363940]  panic+0x107/0x2bb
[12160.364369]  lbug_with_loc.cold+0x1a/0x43 [libcfs]
[12160.365058]  osd_destroy+0x4b8/0x630 [osd_ldiskfs]
[12160.365759]  lod_sub_destroy+0x1de/0x4b0 [lod]
[12160.366399]  lod_destroy+0x26c/0xb60 [lod]
[12160.366997]  ? srso_alias_return_thunk+0x5/0xfbef5
[12160.367678]  ? lod_sub_ref_del+0x1de/0x4b0 [lod]
[12160.368337]  mdd_migrate_create+0x3f9/0x8f0 [mdd]
[12160.369019]  mdd_migrate_object+0xef5/0x1170 [mdd]
[12160.369713]  ? srso_alias_return_thunk+0x5/0xfbef5
[12160.370375]  ? lprocfs_counter_add+0x117/0x180 [obdclass]
[12160.371199]  ? mdd_migrate+0x14/0x20 [mdd]
[12160.371788]  mdd_migrate+0x14/0x20 [mdd]
[12160.372356]  mdt_reint_migrate+0x1381/0x1e00 [mdt]
[12160.373062]  ? __entry_text_end+0x102749/0x10274d
[12160.373725]  mdt_reint_rec+0x119/0x270 [mdt]
[12160.374367]  mdt_reint_internal+0x4ea/0x9b0 [mdt]
[12160.375061]  mdt_reint+0x59/0x110 [mdt]
[12160.375628]  tgt_handle_request0+0x147/0x770 [ptlrpc]
[12160.376437]  tgt_request_handle+0x3fd/0xd00 [ptlrpc]
[12160.377224]  ptlrpc_server_handle_request.isra.0+0x2e5/0xd80 [ptlrpc]
[12160.378206]  ? srso_alias_return_thunk+0x5/0xfbef5
[12160.378883]  ptlrpc_main+0x9bf/0xea0 [ptlrpc]
[12160.379593]  ? __pfx_ptlrpc_main+0x10/0x10 [ptlrpc]
[12160.380357]  kthread+0xdd/0x100
[12160.380805]  ? __pfx_kthread+0x10/0x10
[12160.381332]  ret_from_fork+0x29/0x50
[12160.381841]  </TASK>
 

VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV
sanityn test_80b - trevis-156vm23 crashed during sanityn test_80b