[LU-617] LBUG: (mdt_recovery.c:787:mdt_last_rcvd_update()) ASSERTION(req_is_replay(req)) failed

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.2.0, Lustre 2.1.2
• Affects Version/s: Lustre 2.1.0
• Environment: Lustre Tag: v2_1_0_0_RC0; Lustre Build: http://newbuild.whamcloud.com/job/lustre-master/267/; Distro/Arch: RHEL6/x86_64 (kernel version: 2.6.32-131.6.1.el6)
• Severity: 3
• Assignee: Niu Yawei (Inactive)
• Reporter: Jian Yu

Description

While running racer test, the MDS node (fat-amd-3-ib) hit LBUG as follows:

LustreError: 10364:0:(mdt_recovery.c:787:mdt_last_rcvd_update()) ASSERTION(req_is_replay(req)) failed
LustreError: 10364:0:(mdt_recovery.c:787:mdt_last_rcvd_update()) LBUG
Pid: 10364, comm: mdt_11

Call Trace:
[<ffffffffa03ad855>] libcfs_debug_dumpstack+0x55/0x80 [libcfs]
[<ffffffffa03ade95>] lbug_with_loc+0x75/0xe0 [libcfs]
[<ffffffffa03b8d96>] libcfs_assertion_failed+0x66/0x70 [libcfs]
[<ffffffffa09ca90d>] mdt_txn_stop_cb+0xbbd/0xbd0 [mdt]
[<ffffffffa04bf2a2>] dt_txn_hook_stop+0x62/0x80 [obdclass]
[<ffffffffa09598d8>] osd_trans_stop+0x98/0x290 [osd_ldiskfs]
[<ffffffffa0958470>] ? osd_object_write_unlock+0x60/0xd0 [osd_ldiskfs]
[<ffffffffa0922e3a>] mdd_trans_stop+0x1a/0x20 [mdd]
[<ffffffffa0919cbe>] mdd_rename+0x31e/0x21b0 [mdd]
[<ffffffffa03bcbef>] ? cfs_hash_bd_from_key+0x3f/0xc0 [libcfs]
[<ffffffffa0986239>] ? cmm_mode_get+0x109/0x320 [cmm]
[<ffffffffa0986d5a>] cml_rename+0x33a/0xbb0 [cmm]
[<ffffffffa03bd007>] ? cfs_hash_bd_get+0x37/0x90 [libcfs]
[<ffffffffa09864bd>] ? cmm_is_subdir+0x6d/0x2f0 [cmm]
[<ffffffffa04bcef2>] ? lu_object_put+0x92/0x210 [obdclass]
[<ffffffffa09c5196>] mdt_reint_rename+0x1f96/0x23e0 [mdt]
[<ffffffffa03c394b>] ? upcall_cache_get_entry+0x28b/0xa14 [libcfs]
[<ffffffffa09bd74f>] ? mdt_rename_unpack+0x44f/0x6a0 [mdt]
[<ffffffffa091de56>] ? md_ucred+0x26/0x60 [mdd]
[<ffffffffa09bd9df>] mdt_reint_rec+0x3f/0x100 [mdt]
[<ffffffffa05b1984>] ? lustre_msg_get_flags+0x34/0xa0 [ptlrpc]
[<ffffffffa09b5e84>] mdt_reint_internal+0x6d4/0x9f0 [mdt]
[<ffffffffa09ab946>] ? mdt_reint_opcode+0x96/0x160 [mdt]
[<ffffffffa09b61ec>] mdt_reint+0x4c/0x120 [mdt]
[<ffffffffa05b1458>] ? lustre_msg_check_version+0xc8/0xe0 [ptlrpc]
[<ffffffffa09a9bb5>] mdt_handle_common+0x8d5/0x1810 [mdt]
[<ffffffffa05af104>] ? lustre_msg_get_opc+0x94/0x100 [ptlrpc]
[<ffffffffa09aabc5>] mdt_regular_handle+0x15/0x20 [mdt]
[<ffffffffa05bfc7e>] ptlrpc_main+0xb8e/0x1900 [ptlrpc]
[<ffffffffa05bf0f0>] ? ptlrpc_main+0x0/0x1900 [ptlrpc]
[<ffffffff8100c1ca>] child_rip+0xa/0x20
[<ffffffffa05bf0f0>] ? ptlrpc_main+0x0/0x1900 [ptlrpc]
[<ffffffffa05bf0f0>] ? ptlrpc_main+0x0/0x1900 [ptlrpc]
[<ffffffff8100c1c0>] ? child_rip+0x0/0x20

Kernel panic - not syncing: LBUG

Maloo report: https://maloo.whamcloud.com/test_sets/10c0e904-ccc9-11e0-8d02-52540025f9af

Issue Links

• is related to: LU-5283 LU-617 reappears in 2.4.2
• Trackbacks: Lustre 2.1.0 release testing tracker Lustre 2.1....
• Trackbacks: Changelog 2.1 Changes from version 2.1.1 to version 2.1.2 Server support for kernels: 2.6.18308.4.1.el5 (RHEL5) 2.6.32220.17.1.el6 (RHEL6) Client support for unpatched kernels: 2.6.18308.4.1.el5 (RHEL5) 2.6.32220.17.1....
• Trackbacks: Changelog 2.2 version 2.2.0 Support for networks: o2iblnd OFED 1.5.4 Server support for kernels: 2.6.32220.4.2.el6 (RHEL6) Client support for unpatched kernels: 2.6.18274.18.1.el5 (RHEL5) 2.6.32220.4.2.el6 (RHEL6) 2.6.32.360....

Comments

[Build Master (Inactive)]

Integrated in lustre-master » i686,client,el5,ofa #401
LU-617 recovery: setattr from open breaks recovery (Revision 39308b9999b42bf91aff860a71669ca13686c475)

Result = SUCCESS
Oleg Drokin : 39308b9999b42bf91aff860a71669ca13686c475
Files :

• lustre/mdc/mdc_request.c
• lustre/mdt/mdt_lib.c
• lustre/include/obd.h
• lustre/mdt/mdt_handler.c
• lustre/mdt/mdt_reint.c
• lustre/mdt/mdt_internal.h
• lustre/mdc/mdc_reint.c

[Build Master (Inactive)]

Integrated in lustre-master » i686,client,el5,inkernel #401
LU-617 recovery: setattr from open breaks recovery (Revision 39308b9999b42bf91aff860a71669ca13686c475)

Result = SUCCESS
Oleg Drokin : 39308b9999b42bf91aff860a71669ca13686c475
Files :

• lustre/mdt/mdt_lib.c
• lustre/mdc/mdc_request.c
• lustre/mdt/mdt_internal.h
• lustre/mdc/mdc_reint.c
• lustre/mdt/mdt_handler.c
• lustre/include/obd.h
• lustre/mdt/mdt_reint.c

[Build Master (Inactive)]

Integrated in lustre-master » i686,server,el5,inkernel #401
LU-617 recovery: setattr from open breaks recovery (Revision 39308b9999b42bf91aff860a71669ca13686c475)

Result = SUCCESS
Oleg Drokin : 39308b9999b42bf91aff860a71669ca13686c475
Files :

• lustre/mdc/mdc_request.c
• lustre/mdt/mdt_lib.c
• lustre/mdt/mdt_handler.c
• lustre/mdc/mdc_reint.c
• lustre/include/obd.h
• lustre/mdt/mdt_internal.h
• lustre/mdt/mdt_reint.c

[Build Master (Inactive)]

Integrated in lustre-master » i686,server,el5,ofa #401
LU-617 recovery: setattr from open breaks recovery (Revision 39308b9999b42bf91aff860a71669ca13686c475)

Result = SUCCESS
Oleg Drokin : 39308b9999b42bf91aff860a71669ca13686c475
Files :

• lustre/mdt/mdt_reint.c
• lustre/mdt/mdt_handler.c
• lustre/include/obd.h
• lustre/mdt/mdt_lib.c
• lustre/mdc/mdc_request.c
• lustre/mdc/mdc_reint.c
• lustre/mdt/mdt_internal.h

[Build Master (Inactive)]

Integrated in lustre-master » i686,client,el6,inkernel #401
LU-617 recovery: setattr from open breaks recovery (Revision 39308b9999b42bf91aff860a71669ca13686c475)

Result = SUCCESS
Oleg Drokin : 39308b9999b42bf91aff860a71669ca13686c475
Files :

• lustre/mdt/mdt_handler.c
• lustre/mdc/mdc_request.c
• lustre/mdt/mdt_lib.c
• lustre/mdt/mdt_internal.h
• lustre/mdt/mdt_reint.c
• lustre/include/obd.h
• lustre/mdc/mdc_reint.c

[Peter Jones]

Landed for 2.2

[Build Master (Inactive)]

Integrated in lustre-master » x86_64,server,el6,inkernel #401
LU-617 recovery: setattr from open breaks recovery (Revision 39308b9999b42bf91aff860a71669ca13686c475)

Result = SUCCESS
Oleg Drokin : 39308b9999b42bf91aff860a71669ca13686c475
Files :

• lustre/mdt/mdt_internal.h
• lustre/mdt/mdt_reint.c
• lustre/mdt/mdt_handler.c
• lustre/include/obd.h
• lustre/mdc/mdc_request.c
• lustre/mdt/mdt_lib.c
• lustre/mdc/mdc_reint.c

[Build Master (Inactive)]

Integrated in lustre-master » x86_64,server,el5,inkernel #401
LU-617 recovery: setattr from open breaks recovery (Revision 39308b9999b42bf91aff860a71669ca13686c475)

Result = SUCCESS
Oleg Drokin : 39308b9999b42bf91aff860a71669ca13686c475
Files :

• lustre/mdt/mdt_reint.c
• lustre/mdc/mdc_reint.c
• lustre/mdt/mdt_lib.c
• lustre/mdt/mdt_internal.h
• lustre/mdc/mdc_request.c
• lustre/mdt/mdt_handler.c
• lustre/include/obd.h

[Build Master (Inactive)]

Integrated in lustre-master » x86_64,client,ubuntu1004,inkernel #401
LU-617 recovery: setattr from open breaks recovery (Revision 39308b9999b42bf91aff860a71669ca13686c475)

Result = SUCCESS
Oleg Drokin : 39308b9999b42bf91aff860a71669ca13686c475
Files :

• lustre/mdc/mdc_reint.c
• lustre/mdt/mdt_handler.c
• lustre/mdt/mdt_reint.c
• lustre/mdc/mdc_request.c
• lustre/mdt/mdt_lib.c
• lustre/include/obd.h
• lustre/mdt/mdt_internal.h

[Build Master (Inactive)]

Integrated in lustre-master » x86_64,client,el5,ofa #401
LU-617 recovery: setattr from open breaks recovery (Revision 39308b9999b42bf91aff860a71669ca13686c475)

Result = SUCCESS
Oleg Drokin : 39308b9999b42bf91aff860a71669ca13686c475
Files :

• lustre/mdt/mdt_handler.c
• lustre/mdt/mdt_internal.h
• lustre/mdt/mdt_reint.c
• lustre/mdt/mdt_lib.c
• lustre/include/obd.h
• lustre/mdc/mdc_request.c
• lustre/mdc/mdc_reint.c