[LU-2954] Interop failure on test suite sanityn test_31b: (ofd_lvb.c:287:ofd_lvbo_fill()) ASSERTION( lvb_len <= res->lr_lvb_len ) failed

• Status: Closed
• Project: Lustre
• Affects Version/s: Lustre 2.4.0
• Type: Bug
• Priority: Minor
• Reporter: Maloo
• Assignee: WC Triage
• Resolution: Duplicate

Environment

server: lustre-master build #1295
client: 2.1.4

• Severity: 3

Description

This issue was created by maloo for sarah <[e-mail]>

This issue relates to the following test suite run: https://maloo.whamcloud.com/test_sets/668f0ec0-8905-11e2-b643-52540035b04c.

The sub-test test_31b failed with the following error:

test failed to respond and timed out

OST console:

18:50:32:Lustre: DEBUG MARKER: == sanityn test 31b: voluntary OST cancel / blocking ast race================ 18:50:30 (1362797430)
18:50:32:Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n obdfilter.*.mds_sync
18:50:44:ll_ost00_040: page allocation failure. order:5, mode:0x50
18:50:44:Pid: 3689, comm: ll_ost00_040 Not tainted 2.6.32-279.19.1.el6_lustre.x86_64 #1
18:50:44:Call Trace:
18:50:44: [<ffffffff811231ff>] ? __alloc_pages_nodemask+0x77f/0x940
18:50:44: [<ffffffff8115d1a2>] ? kmem_getpages+0x62/0x170
18:50:44: [<ffffffff8115ddba>] ? fallback_alloc+0x1ba/0x270
18:50:44: [<ffffffff8115d80f>] ? cache_grow+0x2cf/0x320
18:50:44: [<ffffffff8115db39>] ? ____cache_alloc_node+0x99/0x160
18:50:44: [<ffffffffa04d8b60>] ? cfs_alloc+0x30/0x60 [libcfs]
18:50:44: [<ffffffff8115e909>] ? __kmalloc+0x189/0x220
18:50:44: [<ffffffffa04d8b60>] ? cfs_alloc+0x30/0x60 [libcfs]
18:50:44: [<ffffffffa0d3a81e>] ? osd_key_init+0x1e/0x670 [osd_ldiskfs]
18:50:44: [<ffffffffa066399f>] ? keys_fill+0x6f/0x190 [obdclass]
18:50:44: [<ffffffffa06675db>] ? lu_context_init+0xab/0x260 [obdclass]
18:50:44: [<ffffffff8115da20>] ? cache_alloc_refill+0x1c0/0x240
18:50:44: [<ffffffffa06677ae>] ? lu_env_init+0x1e/0x30 [obdclass]
18:50:44: [<ffffffffa0e2b687>] ? ofd_lvbo_init+0x137/0x8e0 [ofd]
18:50:44: [<ffffffffa07a493b>] ? ldlm_resource_get+0x36b/0x730 [ptlrpc]
18:50:44: [<ffffffffa079e915>] ? ldlm_lock_create+0x55/0xa30 [ptlrpc]
18:50:44: [<ffffffffa07c4076>] ? ldlm_handle_enqueue0+0x156/0x1080 [ptlrpc]
18:50:44: [<ffffffffa07c5006>] ? ldlm_handle_enqueue+0x66/0x70 [ptlrpc]
18:50:44: [<ffffffffa07c5010>] ? ldlm_server_completion_ast+0x0/0x630 [ptlrpc]
18:50:44: [<ffffffffa0de0130>] ? ost_blocking_ast+0x0/0xe40 [ost]
18:50:44: [<ffffffffa07c19b0>] ? ldlm_server_glimpse_ast+0x0/0x3b0 [ptlrpc]
18:50:44: [<ffffffffa0de8cc8>] ? ost_handle+0x1e28/0x46f0 [ost]
18:50:44: [<ffffffffa04e4154>] ? libcfs_id2str+0x74/0xb0 [libcfs]
18:50:44: [<ffffffffa07f604c>] ? ptlrpc_server_handle_request+0x41c/0xdf0 [ptlrpc]
18:50:44: [<ffffffffa04d85de>] ? cfs_timer_arm+0xe/0x10 [libcfs]
18:50:44: [<ffffffffa07ed799>] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc]
18:50:44: [<ffffffff81052223>] ? __wake_up+0x53/0x70
18:50:44: [<ffffffffa07f7596>] ? ptlrpc_main+0xb76/0x1870 [ptlrpc]
18:50:44: [<ffffffffa07f6a20>] ? ptlrpc_main+0x0/0x1870 [ptlrpc]
18:50:44: [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
18:50:44: [<ffffffffa07f6a20>] ? ptlrpc_main+0x0/0x1870 [ptlrpc]
18:50:44: [<ffffffffa07f6a20>] ? ptlrpc_main+0x0/0x1870 [ptlrpc]
18:50:44: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
18:50:44:Mem-Info:
18:50:44:Node 0 DMA per-cpu:
18:50:44:CPU    0: hi:    0, btch:   1 usd:   0
18:50:44:Node 0 DMA32 per-cpu:
18:50:44:CPU    0: hi:  186, btch:  31 usd:   0
18:50:44:active_anon:4152 inactive_anon:1541 isolated_anon:0
18:50:44: active_file:143715 inactive_file:178755 isolated_file:0
18:50:44: unevictable:0 dirty:60 writeback:0 unstable:0
18:50:44: free:85893 slab_reclaimable:12500 slab_unreclaimable:16827
18:50:44: mapped:2440 shmem:48 pagetables:750 bounce:0
18:50:44:Node 0 DMA free:8432kB min:332kB low:412kB high:496kB active_anon:0kB inactive_anon:0kB active_file:6468kB inactive_file:84kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15324kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:200kB slab_unreclaimable:548kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
18:50:44:lowmem_reserve[]: 0 2003 2003 2003
18:50:44:Node 0 DMA32 free:335140kB min:44720kB low:55900kB high:67080kB active_anon:16608kB inactive_anon:6164kB active_file:568392kB inactive_file:714936kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:2052064kB mlocked:0kB dirty:240kB writeback:0kB mapped:9760kB shmem:192kB slab_reclaimable:49800kB slab_unreclaimable:66760kB kernel_stack:2912kB pagetables:3000kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
18:50:44:lowmem_reserve[]: 0 0 0 0
18:50:44:Node 0 DMA: 4*4kB 2*8kB 3*16kB 1*32kB 4*64kB 1*128kB 3*256kB 2*512kB 2*1024kB 2*2048kB 0*4096kB = 8432kB
18:50:44:Node 0 DMA32: 7225*4kB 26578*8kB 5503*16kB 54*32kB 4*64kB 2*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB = 335140kB
18:50:44:322317 total pagecache pages
18:50:44:0 pages in swap cache
18:50:44:Swap cache stats: add 0, delete 0, find 0/0
18:50:44:Free swap  = 4128760kB
18:50:44:Total swap = 4128760kB
18:50:44:524284 pages RAM
18:50:44:43608 pages reserved
18:50:44:318178 pages shared
18:50:44:71883 pages non-shared
18:50:44:LustreError: 3689:0:(ldlm_resource.c:1161:ldlm_resource_get()) lvbo_init failed for resource 37852: rc -12
18:50:44:LustreError: 3689:0:(ldlm_resource.c:1161:ldlm_resource_get()) Skipped 45 previous similar messages
18:50:44:LustreError: 3689:0:(ofd_lvb.c:287:ofd_lvbo_fill()) ASSERTION( lvb_len <= res->lr_lvb_len ) failed: 
18:50:44:LustreError: 3689:0:(ofd_lvb.c:287:ofd_lvbo_fill()) LBUG
18:50:44:Pid: 3689, comm: ll_ost00_040
18:50:44:
18:50:44:Call Trace:
18:50:44: [<ffffffffa04d7895>] libcfs_debug_dumpstack+0x55/0x80 [libcfs]
18:50:44: [<ffffffffa04d7e97>] lbug_with_loc+0x47/0xb0 [libcfs]
18:50:45: [<ffffffffa0e2bf0c>] ofd_lvbo_fill+0xac/0xb0 [ofd]
18:50:45: [<ffffffffa0e2be60>] ? ofd_lvbo_fill+0x0/0xb0 [ofd]
18:50:45: [<ffffffffa07c4561>] ldlm_handle_enqueue0+0x641/0x1080 [ptlrpc]
18:50:45: [<ffffffffa07c5006>] ldlm_handle_enqueue+0x66/0x70 [ptlrpc]
18:50:45: [<ffffffffa07c5010>] ? ldlm_server_completion_ast+0x0/0x630 [ptlrpc]
18:50:45: [<ffffffffa0de0130>] ? ost_blocking_ast+0x0/0xe40 [ost]
18:50:45: [<ffffffffa07c19b0>] ? ldlm_server_glimpse_ast+0x0/0x3b0 [ptlrpc]
18:50:45: [<ffffffffa0de8cc8>] ost_handle+0x1e28/0x46f0 [ost]
18:50:45: [<ffffffffa04e4154>] ? libcfs_id2str+0x74/0xb0 [libcfs]
18:50:45: [<ffffffffa07f604c>] ptlrpc_server_handle_request+0x41c/0xdf0 [ptlrpc]
18:50:45: [<ffffffffa04d85de>] ? cfs_timer_arm+0xe/0x10 [libcfs]
18:50:45: [<ffffffffa07ed799>] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc]
18:50:45: [<ffffffff81052223>] ? __wake_up+0x53/0x70
18:50:45: [<ffffffffa07f7596>] ptlrpc_main+0xb76/0x1870 [ptlrpc]
18:50:45: [<ffffffffa07f6a20>] ? ptlrpc_main+0x0/0x1870 [ptlrpc]
18:50:45: [<ffffffff8100c0ca>] child_rip+0xa/0x20
18:50:45: [<ffffffffa07f6a20>] ? ptlrpc_main+0x0/0x1870 [ptlrpc]
18:50:45: [<ffffffffa07f6a20>] ? ptlrpc_main+0x0/0x1870 [ptlrpc]
18:50:45: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
18:50:45:
18:50:45:Kernel panic - not syncing: LBUG
18:50:45:Pid: 3689, comm: ll_ost00_040 Not tainted 2.6.32-279.19.1.el6_lustre.x86_64 #1
18:50:45:Call Trace:
18:50:45: [<ffffffff814e9811>] ? panic+0xa0/0x168
18:50:45: [<ffffffffa04d7eeb>] ? lbug_with_loc+0x9b/0xb0 [libcfs]
18:50:45: [<ffffffffa0e2bf0c>] ? ofd_lvbo_fill+0xac/0xb0 [ofd]
18:50:45: [<ffffffffa0e2be60>] ? ofd_lvbo_fill+0x0/0xb0 [ofd]
18:50:45: [<ffffffffa07c4561>] ? ldlm_handle_enqueue0+0x641/0x1080 [ptlrpc]
18:50:45: [<ffffffffa07c5006>] ? ldlm_handle_enqueue+0x66/0x70 [ptlrpc]
18:50:45: [<ffffffffa07c5010>] ? ldlm_server_completion_ast+0x0/0x630 [ptlrpc]
18:50:45: [<ffffffffa0de0130>] ? ost_blocking_ast+0x0/0xe40 [ost]
18:50:45: [<ffffffffa07c19b0>] ? ldlm_server_glimpse_ast+0x0/0x3b0 [ptlrpc]
18:50:45: [<ffffffffa0de8cc8>] ? ost_handle+0x1e28/0x46f0 [ost]
18:50:45: [<ffffffffa04e4154>] ? libcfs_id2str+0x74/0xb0 [libcfs]
18:50:45: [<ffffffffa07f604c>] ? ptlrpc_server_handle_request+0x41c/0xdf0 [ptlrpc]
18:50:45: [<ffffffffa04d85de>] ? cfs_timer_arm+0xe/0x10 [libcfs]
18:50:45: [<ffffffffa07ed799>] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc]
18:50:45: [<ffffffff81052223>] ? __wake_up+0x53/0x70
18:50:45: [<ffffffffa07f7596>] ? ptlrpc_main+0xb76/0x1870 [ptlrpc]
18:50:45: [<ffffffffa07f6a20>] ? ptlrpc_main+0x0/0x1870 [ptlrpc]
18:50:45: [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
18:50:45: [<ffffffffa07f6a20>] ? ptlrpc_main+0x0/0x1870 [ptlrpc]
18:50:45: [<ffffffffa07f6a20>] ? ptlrpc_main+0x0/0x1870 [ptlrpc]
18:50:45: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20

Comments

[nasf (Inactive)]

This have been fixed by the patch:

http://review.whamcloud.com/#change,5634

[Jodi Levi (Inactive)]

Duplicate of LU-2791