[LU-3729] uncovered page in osc_req_attr_set

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.5.0
• Type: Bug
• Priority: Major
• Reporter: Oleg Drokin
• Assignee: WC Triage
• Resolution: Cannot Reproduce
• Severity: 3

Description

This is potentially another hit of LU-1650?

Hit this during racer on tip of master:

<4>[597951.119031] Lustre: 19544:0:(vvp_io.c:866:vvp_io_read_page()) page@ffff880098850df0[2 ffff880041514f90:0 ^(null)_ffff8800692f7df0 1 0 1 ffff88005f85ceb8 (null) 0x0]
<4>[597951.119877] Lustre: 19544:0:(vvp_io.c:866:vvp_io_read_page()) without a lock: -61
<3>[597951.120398] LustreError: 19544:0:(osc_io.c:795:osc_req_attr_set()) lock@ffff880070e24ee0[1 2 0 1 1 00000000] R(1):[0, 18446744073709551615]@[0x100000000:0x2bbb:0x0] {
<3>[597951.121234] LustreError: 19544:0:(osc_io.c:795:osc_req_attr_set())     lovsub@ffff8800378214b0: [0 ffff880060813230 P(0):[0, 18446744073709551615]@[0x200000403:0xb287:0x0]] 
<3>[597951.122095] LustreError: 19544:0:(osc_io.c:795:osc_req_attr_set())     osc@ffff880027c4ff38: ffff8800546fcdb8    0x20000001001 0x59b8e65062a2d012 3 (null) size: 0 mtime: 0 atime: 0 ctime: 0 blocks: 0
<3>[597951.122945] LustreError: 19544:0:(osc_io.c:795:osc_req_attr_set()) } lock@ffff880070e24ee0
<3>[597951.123453] LustreError: 19544:0:(osc_io.c:795:osc_req_attr_set()) no cover page!
<3>[597951.123949] LustreError: 19544:0:(osc_io.c:797:osc_req_attr_set()) page@ffff880098850df0[4 ffff880041514f90:0 ^(null)_ffff8800692f7df0 3 0 1 (null) ffff8800383022a0 0x0]
<3>[597951.124752] LustreError: 19544:0:(osc_io.c:797:osc_req_attr_set()) page@ffff8800692f7df0[2 ffff880054617fa8:0 ^ffff880098850df0_(null) 3 0 1 (null) (null) 0x0]
<3>[597951.125578] LustreError: 19544:0:(osc_io.c:797:osc_req_attr_set()) vvp-page@ffff880098850ec8(0:0:0) vm@ffffea0001de5878 20000000000801 4:0 ffff880098850df0 0 lru
<3>[597951.126487] LustreError: 19544:0:(osc_io.c:797:osc_req_attr_set()) lov-page@ffff880098850f18
<3>[597951.127006] LustreError: 19544:0:(osc_io.c:797:osc_req_attr_set()) osc-page@ffff8800692f7ef0: 1< 0x845fed 257 0 + + > 2< 0 0 4096 0x7 0x108 | (null) ffff8800b75cab18 ffff880047ff4ec0 > 3< + ffff880028c963c0 1 1 0 > 4< 0 0 8 0 - | - - - + > 5< - - - + | 0 - | 0 - ->
<3>[597951.128148] LustreError: 19544:0:(osc_io.c:797:osc_req_attr_set()) end page@ffff8800692f7df0
<3>[597951.128663] LustreError: 19544:0:(osc_io.c:797:osc_req_attr_set()) dump uncover page!
<4>[597951.129258] Pid: 19544, comm: cat
<4>[597951.130485] 
<4>[597951.130486] Call Trace:
<4>[597951.130950]  [<ffffffffa0d108a5>] libcfs_debug_dumpstack+0x55/0x80 [libcfs]
<4>[597951.131267]  [<ffffffffa04be51f>] osc_req_attr_set+0x55f/0x5b0 [osc]
<4>[597951.131599]  [<ffffffffa0fbdd79>] cl_req_attr_set+0xd9/0x230 [obdclass]
<4>[597951.131904]  [<ffffffffa04afc14>] osc_build_rpc+0x484/0x17f0 [osc]
<4>[597951.132204]  [<ffffffffa0d21041>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
<4>[597951.132510]  [<ffffffffa04c9e2c>] osc_io_unplug0+0x12ac/0x1f30 [osc]
<4>[597951.132819]  [<ffffffffa0d20a63>] ? libcfs_debug_vmsg2+0x513/0xab0 [libcfs]
<4>[597951.133149]  [<ffffffffa04cc641>] osc_io_unplug+0x11/0x20 [osc]
<4>[597951.133442]  [<ffffffffa04cc822>] osc_queue_sync_pages+0x1d2/0x360 [osc]
<4>[597951.133754]  [<ffffffffa04bea8b>] osc_io_submit+0x1cb/0x490 [osc]
<4>[597951.134068]  [<ffffffffa0fbd6dc>] cl_io_submit_rw+0x6c/0x160 [obdclass]
<4>[597951.134387]  [<ffffffffa053a889>] lov_io_submit+0x209/0xb70 [lov]
<4>[597951.134710]  [<ffffffffa0fbd6dc>] cl_io_submit_rw+0x6c/0x160 [obdclass]
<4>[597951.135063]  [<ffffffffa0fbfbfe>] cl_io_read_page+0xae/0x170 [obdclass]
<4>[597951.135410]  [<ffffffffa0fb3a77>] ? cl_page_assume+0xf7/0x220 [obdclass]
<4>[597951.135749]  [<ffffffffa0a501c6>] ll_readpage+0x96/0x1a0 [lustre]
<4>[597951.136043]  [<ffffffff81119955>] generic_file_aio_read+0x215/0x740
<4>[597951.136349]  [<ffffffffa0a763d4>] ? cl_glimpse_lock+0x2a4/0x490 [lustre]
<4>[597951.136693]  [<ffffffffa0a7fef9>] vvp_io_read_start+0x259/0x470 [lustre]
<4>[597951.137130]  [<ffffffffa0fbd83a>] cl_io_start+0x6a/0x140 [obdclass]
<4>[597951.137479]  [<ffffffffa0fc1e34>] cl_io_loop+0xb4/0x1b0 [obdclass]
<4>[597951.137795]  [<ffffffffa0a20f5f>] ll_file_io_generic+0x33f/0x610 [lustre]
<4>[597951.138109]  [<ffffffffa0a2136f>] ll_file_aio_read+0x13f/0x2c0 [lustre]
<4>[597951.138434]  [<ffffffffa0a21c0c>] ll_file_read+0x16c/0x2a0 [lustre]
<4>[597951.138759]  [<ffffffff81181bd5>] vfs_read+0xb5/0x1a0
<4>[597951.139028]  [<ffffffff81181d11>] sys_read+0x51/0x90
<4>[597951.139294]  [<ffffffff8100b0b2>] system_call_fastpath+0x16/0x1b
<4>[597951.139669] 
<0>[597951.139904] LustreError: 19544:0:(osc_io.c:799:osc_req_attr_set()) LBUG
<4>[597951.140366] Pid: 19544, comm: cat
<4>[597951.140657] 
<4>[597951.140657] Call Trace:
<4>[597951.141149]  [<ffffffffa0d108a5>] libcfs_debug_dumpstack+0x55/0x80 [libcfs]
<4>[597951.141466]  [<ffffffffa0d10ea7>] lbug_with_loc+0x47/0xb0 [libcfs]
<4>[597951.141773]  [<ffffffffa04be535>] osc_req_attr_set+0x575/0x5b0 [osc]
<4>[597951.142101]  [<ffffffffa0fbdd79>] cl_req_attr_set+0xd9/0x230 [obdclass]
<4>[597951.142457]  [<ffffffffa04afc14>] osc_build_rpc+0x484/0x17f0 [osc]
<4>[597951.142847]  [<ffffffffa0d21041>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
<4>[597951.143164]  [<ffffffffa04c9e2c>] osc_io_unplug0+0x12ac/0x1f30 [osc]
<4>[597951.143468]  [<ffffffffa0d20a63>] ? libcfs_debug_vmsg2+0x513/0xab0 [libcfs]
<4>[597951.143786]  [<ffffffffa04cc641>] osc_io_unplug+0x11/0x20 [osc]
<4>[597951.144079]  [<ffffffffa04cc822>] osc_queue_sync_pages+0x1d2/0x360 [osc]
<4>[597951.144385]  [<ffffffffa04bea8b>] osc_io_submit+0x1cb/0x490 [osc]
<4>[597951.144710]  [<ffffffffa0fbd6dc>] cl_io_submit_rw+0x6c/0x160 [obdclass]
<4>[597951.145040]  [<ffffffffa053a889>] lov_io_submit+0x209/0xb70 [lov]
<4>[597951.145356]  [<ffffffffa0fbd6dc>] cl_io_submit_rw+0x6c/0x160 [obdclass]
<4>[597951.145680]  [<ffffffffa0fbfbfe>] cl_io_read_page+0xae/0x170 [obdclass]
<4>[597951.146001]  [<ffffffffa0fb3a77>] ? cl_page_assume+0xf7/0x220 [obdclass]
<4>[597951.146324]  [<ffffffffa0a501c6>] ll_readpage+0x96/0x1a0 [lustre]
<4>[597951.146624]  [<ffffffff81119955>] generic_file_aio_read+0x215/0x740
<4>[597951.146928]  [<ffffffffa0a763d4>] ? cl_glimpse_lock+0x2a4/0x490 [lustre]
<4>[597951.147241]  [<ffffffffa0a7fef9>] vvp_io_read_start+0x259/0x470 [lustre]
<4>[597951.147566]  [<ffffffffa0fbd83a>] cl_io_start+0x6a/0x140 [obdclass]
<4>[597951.147885]  [<ffffffffa0fc1e34>] cl_io_loop+0xb4/0x1b0 [obdclass]
<4>[597951.148188]  [<ffffffffa0a20f5f>] ll_file_io_generic+0x33f/0x610 [lustre]
<4>[597951.148497]  [<ffffffffa0a2136f>] ll_file_aio_read+0x13f/0x2c0 [lustre]
<4>[597951.148809]  [<ffffffffa0a21c0c>] ll_file_read+0x16c/0x2a0 [lustre]
<4>[597951.149112]  [<ffffffff81181bd5>] vfs_read+0xb5/0x1a0
<4>[597951.149386]  [<ffffffff81181d11>] sys_read+0x51/0x90
<4>[597951.149681]  [<ffffffff8100b0b2>] system_call_fastpath+0x16/0x1b
<4>[597951.149973] 
<0>[597951.217252] Kernel panic - not syncing: LBUG

Comments

[Oleg Drokin]

I just hit this again on master while running master.

Crashdump in /exports/crashdumps/192.168.10.210-2015-08-17-01\:29\:24/
tag in my tree master-20150816

<3>[52056.764891] LustreError: 7366:0:(osc_io.c:976:osc_req_attr_set()) page@ffff88006d92cdf0[2 ffff8800b4adcfa0 3 0 1 (null) ffff880067f70508]
<3>[52056.764892] 
<3>[52056.765524] LustreError: 7366:0:(osc_io.c:976:osc_req_attr_set()) vvp-page@ffff88006d92ce58(1:0) vm@ffffea000135f6e8 20000000000821 2:0 ffff88006d92cdf0 256 lru
<3>[52056.765525] 
<3>[52056.766311] LustreError: 7366:0:(osc_io.c:976:osc_req_attr_set()) lov-page@ffff88006d92ce98, raid0
<3>[52056.766311] 
<3>[52056.766905] LustreError: 7366:0:(osc_io.c:976:osc_req_attr_set()) osc-page@ffff88006d92cf00 0: 1< 0x845fed 257 0 + + > 2< 0 0 4096 0x7 0x108 | (null) ffff88002d9f2ac8 ffff880095990ea0 > 3< 1 9 0 > 4< 0 0 8 36995072 - | - - + + > 5< - - - + | 256 + | 0 - ->
<3>[52056.766906] 
<3>[52056.768140] LustreError: 7366:0:(osc_io.c:976:osc_req_attr_set()) end page@ffff88006d92cdf0
<3>[52056.768141] 
<3>[52056.768833] LustreError: 7366:0:(osc_io.c:976:osc_req_attr_set()) uncovered page!
<3>[52056.769315] LustreError: 7366:0:(ldlm_resource.c:1448:ldlm_resource_dump()) --- Resource: [0xe05:0x0:0x0].0 (ffff88006d854ef8) refcount = 2
<4>[52056.770013] Pid: 7366, comm: ptlrpcd_01_00
<4>[52056.770260] 
<4>[52056.770260] Call Trace:
<4>[52056.770679]  [<ffffffffa0896885>] libcfs_debug_dumpstack+0x55/0x80 [libcfs]
<4>[52056.771025]  [<ffffffffa04d5e6a>] osc_req_attr_set+0x55a/0x720 [osc]
<4>[52056.771339]  [<ffffffffa0da0ab9>] cl_req_attr_set+0xc9/0x220 [obdclass]
<4>[52056.771634]  [<ffffffffa04c6ecf>] osc_build_rpc+0x84f/0x1280 [osc]
<4>[52056.772018]  [<ffffffffa04e1d1e>] osc_io_unplug0+0xe3e/0x1ae0 [osc]
<4>[52056.772314]  [<ffffffff8151e3ee>] ? thread_return+0x4e/0x7e0
<4>[52056.772584]  [<ffffffffa08a2c31>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
<4>[52056.772926]  [<ffffffffa04e4f40>] osc_io_unplug+0x10/0x20 [osc]
<4>[52056.773363]  [<ffffffffa04bf65b>] brw_queue_work+0x3b/0xf0 [osc]
<4>[52056.773752]  [<ffffffffa1ad03f0>] work_interpreter+0x30/0x100 [ptlrpc]
<4>[52056.774203]  [<ffffffffa1ad9693>] ptlrpc_check_set+0x613/0x1bf0 [ptlrpc]
<4>[52056.774527]  [<ffffffff81522574>] ? _spin_lock_irqsave+0x24/0x30
<4>[52056.774866]  [<ffffffffa1b076e3>] ptlrpcd_check+0x3e3/0x630 [ptlrpc]
<4>[52056.775281]  [<ffffffffa1b07c4b>] ptlrpcd+0x31b/0x500 [ptlrpc]
<4>[52056.775563]  [<ffffffff81061630>] ? default_wake_function+0x0/0x20
<4>[52056.775875]  [<ffffffffa1b07930>] ? ptlrpcd+0x0/0x500 [ptlrpc]
<4>[52056.776255]  [<ffffffff8109ce4e>] kthread+0x9e/0xc0
<4>[52056.776510]  [<ffffffff8100c24a>] child_rip+0xa/0x20
<4>[52056.776772]  [<ffffffff8109cdb0>] ? kthread+0x0/0xc0
<4>[52056.777040]  [<ffffffff8100c240>] ? child_rip+0x0/0x20
<4>[52056.777372] 
<0>[52056.777565] LustreError: 7366:0:(osc_io.c:986:osc_req_attr_set()) LBUG
<4>[52056.777875] Pid: 7366, comm: ptlrpcd_01_00
<4>[52056.778198] 
<4>[52056.778199] Call Trace:
<4>[52056.778601]  [<ffffffffa0896885>] libcfs_debug_dumpstack+0x55/0x80 [libcfs]
<4>[52056.778917]  [<ffffffffa0896e87>] lbug_with_loc+0x47/0xb0 [libcfs]
<4>[52056.779281]  [<ffffffffa04d5e80>] osc_req_attr_set+0x570/0x720 [osc]
<4>[52056.779583]  [<ffffffffa0da0ab9>] cl_req_attr_set+0xc9/0x220 [obdclass]
<4>[52056.779888]  [<ffffffffa04c6ecf>] osc_build_rpc+0x84f/0x1280 [osc]
<4>[52056.780258]  [<ffffffffa04e1d1e>] osc_io_unplug0+0xe3e/0x1ae0 [osc]
<4>[52056.781567]  [<ffffffff8151e3ee>] ? thread_return+0x4e/0x7e0
<4>[52056.781857]  [<ffffffffa08a2c31>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
<4>[52056.782246]  [<ffffffffa04e4f40>] osc_io_unplug+0x10/0x20 [osc]
<4>[52056.782521]  [<ffffffffa04bf65b>] brw_queue_work+0x3b/0xf0 [osc]
<4>[52056.782817]  [<ffffffffa1ad03f0>] work_interpreter+0x30/0x100 [ptlrpc]
<4>[52056.783231]  [<ffffffffa1ad9693>] ptlrpc_check_set+0x613/0x1bf0 [ptlrpc]
<4>[52056.783520]  [<ffffffff81522574>] ? _spin_lock_irqsave+0x24/0x30
<4>[52056.783816]  [<ffffffffa1b076e3>] ptlrpcd_check+0x3e3/0x630 [ptlrpc]
<4>[52056.784154]  [<ffffffffa1b07c4b>] ptlrpcd+0x31b/0x500 [ptlrpc]
<4>[52056.784498]  [<ffffffff81061630>] ? default_wake_function+0x0/0x20
<4>[52056.784807]  [<ffffffffa1b07930>] ? ptlrpcd+0x0/0x500 [ptlrpc]
<4>[52056.785108]  [<ffffffff8109ce4e>] kthread+0x9e/0xc0
<4>[52056.785457]  [<ffffffff8100c24a>] child_rip+0xa/0x20
<4>[52056.785710]  [<ffffffff8109cdb0>] ? kthread+0x0/0xc0
<4>[52056.785992]  [<ffffffff8100c240>] ? child_rip+0x0/0x20
<4>[52056.786301] 
<0>[52057.013098] Kernel panic - not syncing: LBUG

[Oleg Drokin]

Just hit once again.
dump is in /exports/crashdumps/192.168.10.223-2015-09-17-10\:31\:51/

[James A Simmons]

On master or 2.5?