[LU-10595] Use after free in mgc_process_cfg_log

Related — Wed, 10 Oct 2018 02:15:41 +0000

Just had two or three instances of this crash on current master:

[ 9714.914126] Lustre: DEBUG MARKER: == conf-sanity test 80: mgc import reconnect race ==================================================== 18:53:56 (1517442836)
[ 9726.765708] Lustre: MGS: Connection restored to MGC192.168.123.219@tcp_0 (at 0@lo)
[ 9726.769718] Lustre: Skipped 5 previous similar messages
[ 9728.389908] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000
[ 9728.392105] Lustre: Skipped 4 previous similar messages
[ 9728.916729] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space
[ 9728.950481] Lustre: lustre-MDT0000: new disk, initializing
[ 9731.840889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 9732.428932] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
[ 9763.590945] Lustre: lustre-OST0000: new disk, initializing
[ 9763.610198] Lustre: srv-lustre-OST0000: No data found on store. Initialize space
[ 9763.611457] Lustre: Skipped 1 previous similar message
[ 9764.792295] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 9767.273735] Lustre: 25479:0:(genops.c:1759:obd_export_evict_by_uuid()) MGS: evicting e9e169f7-485c-0d53-9888-f0d9fa325f2a at adminstrative request
[ 9769.188319] LustreError: 166-1: MGC192.168.123.219@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 9769.190799] Lustre: *** cfs_fail_loc=906, val=2147483648***
[ 9775.191600] Lustre: MGS: Received new LWP connection from 0@lo, removing former export from same NID
[ 9775.194433] Lustre: Evicted from MGS (at 192.168.123.219@tcp) after server handle changed from 0xe819d599f6959e7b to 0xe819d599f6959ff5
[ 9810.655763] Lustre: lustre-OST0001: new disk, initializing
[ 9810.680854] Lustre: srv-lustre-OST0001: No data found on store. Initialize space
[ 9811.278451] Lustre: lustre-MDT0000: Connection restored to 192.168.123.219@tcp (at 0@lo)
[ 9811.279949] Lustre: Skipped 6 previous similar messages
[ 9811.492792] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
[ 9815.865193] Lustre: server umount lustre-MDT0000 complete
[ 9823.266665] Lustre: 31033:0:(client.c:2100:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1517442938/real 1517442938]  req@ffff8802f4f8ec00 x1591153820441296/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1517442945 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
[ 9823.270325] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 9860.012641] Lustre: 21310:0:(client.c:2100:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1517442938/real 1517442938]  req@ffff8802ecd8bc00 x1591153820441264/t0(0) o101->MGC192.168.123.219@tcp@0@lo:26/25 lens 328/344 e 0 to 1 dl 1517442982 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
[ 9860.061544] Lustre: 21310:0:(client.c:2100:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
[ 9860.062498] LustreError: 166-1: MGC192.168.123.219@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 9860.064815] LustreError: 21310:0:(mgc_request.c:603:do_requeue()) failed processing log: -5
[ 9860.319901] Lustre: server umount lustre-OST0000 complete
[ 9880.065686] BUG: unable to handle kernel paging request at ffff8800a25b1e20
[ 9880.066512] IP: [<ffffffffa01b863d>] mgc_process_cfg_log+0x5d/0x1030 [mgc]
[ 9880.067345] PGD 2e75067 PUD 33fa01067 PMD 33f8ee067 PTE 80000000a25b1060
[ 9880.068112] Oops: 0000 [#1] SMP DEBUG_PAGEALLOC
[ 9880.068885] Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) lov(OE) mdc(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) zlib_deflate jbd2 syscopyarea ata_generic sysfillrect sysimgblt pata_acpi ttm drm_kms_helper ata_piix i2c_piix4 drm libata serio_raw virtio_balloon virtio_console pcspkr i2c_core virtio_blk floppy nfsd ip_tables rpcsec_gss_krb5 [last unloaded: libcfs]
[ 9880.075609] CPU: 13 PID: 21310 Comm: ll_cfg_requeue Tainted: P        W  OE  ------------   3.10.0-debug #2
[ 9880.076999] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
[ 9880.077705] task: ffff8800aa5a8200 ti: ffff8802d2dc0000 task.ti: ffff8802d2dc0000
[ 9880.079168] RIP: 0010:[<ffffffffa01b863d>]  [<ffffffffa01b863d>] mgc_process_cfg_log+0x5d/0x1030 [mgc]
[ 9880.080517] RSP: 0018:ffff8802d2dc3ca8  EFLAGS: 00010286
[ 9880.081028] RAX: ffff8800a25b1800 RBX: 0000000000000000 RCX: 0000000000000000
[ 9880.081505] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8802d1a7e9c0
[ 9880.083252] RBP: ffff8802d2dc3d08 R08: 0000000000000000 R09: 0000000000000000
[ 9880.083807] R10: ffff8802a4dbc2ca R11: 0000000000000000 R12: ffff8802d6659e00
[ 9880.084362] R13: ffff8802a17cd000 R14: ffff88032397a1c0 R15: ffff8800aa5a8200
[ 9880.084884] FS:  0000000000000000(0000) GS:ffff88033e5a0000(0000) knlGS:0000000000000000
[ 9880.085839] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 9880.086372] CR2: ffff8800a25b1e20 CR3: 00000000bb3ff000 CR4: 00000000000006e0
[ 9880.086984] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 9880.087508] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 9880.088054] Stack:
[ 9880.088518]  0000000000000246 00000001d6659e00 ffffffffa01c4538 0000000000000246
[ 9880.090260]  ffffffffa01c4520 ffff8800aa5a8200 ffff880200000001 0000000000000000
[ 9880.091773]  ffff8800aa5a8200 ffff8802a17cd000 ffff8802d6659e00 ffff8800aa5a8200
[ 9880.092920] Call Trace:
[ 9880.093433]  [<ffffffffa01bc673>] mgc_process_log+0x3d3/0x8a0 [mgc]
[ 9880.093947]  [<ffffffff810b7cc0>] ? wake_up_state+0x20/0x20
[ 9880.094467]  [<ffffffffa01bea70>] mgc_requeue_thread+0x2c0/0x870 [mgc]
[ 9880.095022]  [<ffffffff810b7cc0>] ? wake_up_state+0x20/0x20
[ 9880.095795]  [<ffffffffa01be7b0>] ? mgc_process_config+0x1400/0x1400 [mgc]
[ 9880.096607]  [<ffffffff810a2eba>] kthread+0xea/0xf0
[ 9880.097352]  [<ffffffff810a2dd0>] ? kthread_create_on_node+0x140/0x140
[ 9880.098189]  [<ffffffff8170fb98>] ret_from_fork+0x58/0x90
[ 9880.099016]  [<ffffffff810a2dd0>] ? kthread_create_on_node+0x140/0x140
[ 9880.099840] Code: 04 00 00 4d 85 e4 0f 84 5a 0d 00 00 41 8b 84 24 c0 00 00 00 83 f8 01 0f 84 17 0d 00 00 49 8b 44 24 30 48 85 c0 0f 84 6b 04 00 00 <4c> 8b b8 20 06 00 00 48 8b 3d c5 8d ae e2 ba 38 00 00 00 be 50 
[ 9880.103060] RIP  [<ffffffffa01b863d>] mgc_process_cfg_log+0x5d/0x1030 [mgc]

(gdb) l *(mgc_process_cfg_log+0x5d)
0x363d is in mgc_process_cfg_log (/home/green/git/lustre-release/lustre/mgc/mgc_request.c:1913).
1908	
1909		LASSERT(cld);
1910		LASSERT(mutex_is_locked(&cld->cld_lock));
1911	
1912		if (cld->cld_cfg.cfg_sb)
1913			lsi = s2lsi(cld->cld_cfg.cfg_sb);
1914	
1915		OBD_ALLOC_PTR(env);
1916		if (env == NULL)
1917			RETURN(-ENOMEM);
(gdb)

Whamcloud Community JIRA

[LU-10595] Use after free in mgc_process_cfg_log