[LU-5459] (cl_page.c:694:cl_page_assume()) ASSERTION( 0 ) failed

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.6.0, Lustre 2.7.0
• Fix Version/s: Lustre 2.7.0
• Type: Bug
• Priority: Critical
• Reporter: John Hammond
• Assignee: Jinshan Xiong (Inactive)
• Resolution: Fixed
• Labels: hsm
• Severity: 3

Description

To reproduce perform repeated HSM release and explicit restore of file:

t:lustre-release# cd /mnt/lustre
t:lustre# echo 'Uludag Gazoz!' > f0
t:lustre# lfs hsm_archive f0
t:lustre# while true; do
> lfs hsm_release f0
> cat f0^C
t:lustre# while true; do
> lfs hsm_release f0
> cat f0
> done
Uludag Gazoz!
Uludag Gazoz!
Uludag Gazoz!
Uludag Gazoz!
Uludag Gazoz!
Uludag Gazoz!
...
Uludag Gazoz!

Message from syslogd@t at Aug  6 15:21:37 ...
 kernel:[84351.829259] LustreError: 3111:0:(cl_page.c:694:cl_page_assume()) ASSERTION( 0 ) failed: 

Message from syslogd@t at Aug  6 15:21:37 ...
 kernel:[84351.830735] LustreError: 3111:0:(cl_page.c:694:cl_page_assume()) LBUG
Write failed: Broken pipe

[84351.808432] LustreError: 3111:0:(cl_page.c:694:cl_page_assume()) page@ffff8801dccfd878[2 ffff8801df332f68 0 0 1 d84156c5635688c0 ffff8801cdc6a820]
[84351.808436] 
[84351.812902] LustreError: 3111:0:(cl_page.c:694:cl_page_assume()) vvp-page@ffff8801dccfd8e0(0:0:0) vm@ffffea000553c768 40000000000801 4:0 ffff8801dccfd878 0 lru
[84351.812906] 
[84351.817618] LustreError: 3111:0:(cl_page.c:694:cl_page_assume()) lov-page@ffff8801dccfd930, raid0
[84351.817620] 
[84351.820793] LustreError: 3111:0:(cl_page.c:694:cl_page_assume()) osc-page@ffff8801dcd0d788 0: 1< 0x845fed 0 0 - - > 2< 0 0 0 0x0 0x100 | (null) ffff8802157630f0 ffff8801d20d5818 > 3< - (null) 0 0 0 > 4< 0 0 8 10481664 - | - - - - > 5< - - - - | 0 - | 0 - ->
[84351.820798] 
[84351.826036] LustreError: 3111:0:(cl_page.c:694:cl_page_assume()) end page@ffff8801dccfd878
[84351.826037] 
[84351.827838] LustreError: 3111:0:(cl_page.c:694:cl_page_assume()) pg->cp_owner == NULL
[84351.829259] LustreError: 3111:0:(cl_page.c:694:cl_page_assume()) ASSERTION( 0 ) failed: 
[84351.830735] LustreError: 3111:0:(cl_page.c:694:cl_page_assume()) LBUG
[84351.831958] Pid: 3111, comm: cat
[84351.832583] 
[84351.832584] Call Trace:
[84351.833370]  [<ffffffffa0eae8c5>] libcfs_debug_dumpstack+0x55/0x80 [libcfs]
[84351.834657]  [<ffffffffa0eaeec7>] lbug_with_loc+0x47/0xb0 [libcfs]
[84351.835855]  [<ffffffffa10077e2>] cl_page_assume+0x202/0x210 [obdclass]
[84351.837103]  [<ffffffffa09a5d4d>] ll_readpage+0xbd/0x280 [lustre]
[84351.838241]  [<ffffffff81133335>] generic_file_aio_read+0x215/0x740
[84351.839416]  [<ffffffffa09d864e>] vvp_io_read_start+0x22e/0x450 [lustre]
[84351.840677]  [<ffffffffa10101ca>] cl_io_start+0x6a/0x140 [obdclass]
[84351.841857]  [<ffffffffa10141f4>] cl_io_loop+0xb4/0x1b0 [obdclass]
[84351.843023]  [<ffffffffa0976872>] ll_file_io_generic+0x412/0x8f0 [lustre]
[84351.844298]  [<ffffffffa0976e80>] ll_file_aio_read+0x130/0x2b0 [lustre]
[84351.845516]  [<ffffffffa0977329>] ll_file_read+0x159/0x290 [lustre]
[84351.846674]  [<ffffffff811a3185>] vfs_read+0xb5/0x1a0
[84351.847626]  [<ffffffff81554298>] ? lockdep_sys_exit_thunk+0x35/0x67
[84351.848791]  [<ffffffff811a32c1>] sys_read+0x51/0x90
[84351.849724]  [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
[84351.850837] 
[84351.851538] Kernel panic - not syncing: LBUG

Comments

[Frank Zago (Inactive)]

Hi John,

I've seen the same thing, but it's much harder to reproduce. With your test, the copytool hangs within seconds
This bug is also present in Lustre 2.5

Could you test again with this patch: http://review.whamcloud.com/#/c/11386/

[Jinshan Xiong (Inactive)]

Thanks Zago, there is a similar patch at http://review.whamcloud.com/11394

[Frank Zago (Inactive)]

Thanks. That looks better than my patch.

[Peter Jones]

Landed for 2.7

[James Nunez (Inactive)]

Patch for b2_5 at http://review.whamcloud.com/#/c/12393/