[LU-2818] Failure on test suite parallel-scale-nfsv4 test_compilebench: (mdt_lvb.c:126:mdt_lvbo_fill()) ASSERTION( rc == 0 ) failed

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.6.0, Lustre 2.5.1
• Affects Version/s: Lustre 2.4.0, Lustre 2.5.0
• Severity: 3
• Assignee: Hongchao Zhang
• Reporter: Maloo

Description

This issue was created by maloo for sarah <[e-mail]>

This issue relates to the following test suite run: https://maloo.whamcloud.com/test_sets/25d39028-755e-11e2-bf59-52540035b04c.

The sub-test test_compilebench failed with the following error:

compilebench failed: 1

MDS console:

12:44:26:Lustre: DEBUG MARKER: == parallel-scale-nfsv4 test compilebench: compilebench ============================================== 12:44:25 (1360701865)
12:44:27:Lustre: DEBUG MARKER: /usr/sbin/lctl mark .\/compilebench -D \/mnt\/lustre\/d0.compilebench -i 2         -r 2 --makej
12:44:27:Lustre: DEBUG MARKER: ./compilebench -D /mnt/lustre/d0.compilebench -i 2 -r 2 --makej
13:06:42:mdt00_003: page allocation failure. order:5, mode:0x50
13:06:42:Pid: 22716, comm: mdt00_003 Not tainted 2.6.32-279.19.1.el6_lustre.x86_64 #1
13:06:42:Call Trace:
13:06:42: [<ffffffff811231ff>] ? __alloc_pages_nodemask+0x77f/0x940
13:06:42: [<ffffffff8115d1a2>] ? kmem_getpages+0x62/0x170
13:06:42: [<ffffffff8115ddba>] ? fallback_alloc+0x1ba/0x270
13:06:42: [<ffffffff8115d80f>] ? cache_grow+0x2cf/0x320
13:06:42: [<ffffffff8115db39>] ? ____cache_alloc_node+0x99/0x160
13:06:42: [<ffffffffa04d8b60>] ? cfs_alloc+0x30/0x60 [libcfs]
13:06:42: [<ffffffff8115e909>] ? __kmalloc+0x189/0x220
13:06:42: [<ffffffffa04d8b60>] ? cfs_alloc+0x30/0x60 [libcfs]
13:06:42: [<ffffffffa0d3b69e>] ? osd_key_init+0x1e/0x670 [osd_ldiskfs]
13:06:42: [<ffffffffa066622f>] ? keys_fill+0x6f/0x190 [obdclass]
13:06:42: [<ffffffffa0669e6b>] ? lu_context_init+0xab/0x260 [obdclass]
13:06:42: [<ffffffffa066a03e>] ? lu_env_init+0x1e/0x30 [obdclass]
13:06:42: [<ffffffffa0e775ff>] ? mdt_lvbo_fill+0x1af/0x800 [mdt]
13:06:42: [<ffffffffa0e77450>] ? mdt_lvbo_fill+0x0/0x800 [mdt]
13:06:42: [<ffffffffa07c6591>] ? ldlm_handle_enqueue0+0x641/0x1080 [ptlrpc]
13:06:42: [<ffffffffa0e4e376>] ? mdt_enqueue+0x46/0x110 [mdt]
13:06:42: [<ffffffffa0e42fb8>] ? mdt_handle_common+0x628/0x1620 [mdt]
13:06:42: [<ffffffffa0e7a5c5>] ? mds_regular_handle+0x15/0x20 [mdt]
13:06:42: [<ffffffffa07f800c>] ? ptlrpc_server_handle_request+0x41c/0xdf0 [ptlrpc]
13:06:42: [<ffffffffa04d85de>] ? cfs_timer_arm+0xe/0x10 [libcfs]
13:06:42: [<ffffffffa07ef739>] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc]
13:06:42: [<ffffffff81052223>] ? __wake_up+0x53/0x70
13:06:42: [<ffffffffa07f9556>] ? ptlrpc_main+0xb76/0x1870 [ptlrpc]
13:06:42: [<ffffffffa07f89e0>] ? ptlrpc_main+0x0/0x1870 [ptlrpc]
13:06:42: [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
13:06:42: [<ffffffffa07f89e0>] ? ptlrpc_main+0x0/0x1870 [ptlrpc]
13:06:42: [<ffffffffa07f89e0>] ? ptlrpc_main+0x0/0x1870 [ptlrpc]
13:06:42: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
13:06:42:Mem-Info:
13:06:42:Node 0 DMA per-cpu:
13:06:42:CPU    0: hi:    0, btch:   1 usd:   0
13:06:42:Node 0 DMA32 per-cpu:
13:06:42:CPU    0: hi:  186, btch:  31 usd: 152
13:06:42:active_anon:4078 inactive_anon:1377 isolated_anon:0
13:06:42: active_file:109040 inactive_file:162532 isolated_file:0
13:06:42: unevictable:0 dirty:14064 writeback:512 unstable:0
13:06:42: free:14243 slab_reclaimable:32585 slab_unreclaimable:116609
13:06:42: mapped:3106 shmem:41 pagetables:752 bounce:0
13:06:42:Node 0 DMA free:8348kB min:332kB low:412kB high:496kB active_anon:0kB inactive_anon:0kB active_file:184kB inactive_file:3376kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15324kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:556kB slab_unreclaimable:3268kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
13:06:42:lowmem_reserve[]: 0 2003 2003 2003
13:06:42:Node 0 DMA32 free:48624kB min:44720kB low:55900kB high:67080kB active_anon:16312kB inactive_anon:5508kB active_file:435976kB inactive_file:646752kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:2052064kB mlocked:0kB dirty:56256kB writeback:2048kB mapped:12424kB shmem:164kB slab_reclaimable:129784kB slab_unreclaimable:463168kB kernel_stack:1960kB pagetables:3008kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
13:06:42:lowmem_reserve[]: 0 0 0 0
13:06:42:Node 0 DMA: 33*4kB 5*8kB 3*16kB 0*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 3*2048kB 0*4096kB = 8348kB
13:06:42:Node 0 DMA32: 5780*4kB 2228*8kB 4*16kB 28*32kB 3*64kB 3*128kB 2*256kB 1*512kB 1*1024kB 0*2048kB 1*4096kB = 48624kB
13:06:42:195649 total pagecache pages
13:06:42:0 pages in swap cache
13:06:42:Swap cache stats: add 0, delete 0, find 0/0
13:06:42:Free swap  = 4128760kB
13:06:42:Total swap = 4128760kB
13:06:42:524284 pages RAM
13:06:42:43608 pages reserved
13:06:42:207968 pages shared
13:06:42:267301 pages non-shared
13:06:42:LustreError: 22716:0:(mdt_lvb.c:126:mdt_lvbo_fill()) ASSERTION( rc == 0 ) failed: 
13:06:42:LustreError: 22716:0:(mdt_lvb.c:126:mdt_lvbo_fill()) LBUG
13:06:42:Pid: 22716, comm: mdt00_003
13:06:42:
13:06:42:Call Trace:
13:06:42: [<ffffffffa04d7895>] libcfs_debug_dumpstack+0x55/0x80 [libcfs]
13:06:42: [<ffffffffa04d7e97>] lbug_with_loc+0x47/0xb0 [libcfs]
13:06:42: [<ffffffffa0e77bd5>] mdt_lvbo_fill+0x785/0x800 [mdt]
13:06:42: [<ffffffffa0e77450>] ? mdt_lvbo_fill+0x0/0x800 [mdt]
13:06:42: [<ffffffffa07c6591>] ldlm_handle_enqueue0+0x641/0x1080 [ptlrpc]
13:06:42: [<ffffffffa0e4e376>] mdt_enqueue+0x46/0x110 [mdt]
13:06:43: [<ffffffffa0e42fb8>] mdt_handle_common+0x628/0x1620 [mdt]
13:06:43: [<ffffffffa0e7a5c5>] mds_regular_handle+0x15/0x20 [mdt]
13:06:43: [<ffffffffa07f800c>] ptlrpc_server_handle_request+0x41c/0xdf0 [ptlrpc]
13:06:43: [<ffffffffa04d85de>] ? cfs_timer_arm+0xe/0x10 [libcfs]
13:06:43: [<ffffffffa07ef739>] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc]
13:06:43: [<ffffffff81052223>] ? __wake_up+0x53/0x70
13:06:43: [<ffffffffa07f9556>] ptlrpc_main+0xb76/0x1870 [ptlrpc]
13:06:43: [<ffffffffa07f89e0>] ? ptlrpc_main+0x0/0x1870 [ptlrpc]
13:06:43: [<ffffffff8100c0ca>] child_rip+0xa/0x20
13:06:43: [<ffffffffa07f89e0>] ? ptlrpc_main+0x0/0x1870 [ptlrpc]
13:06:43: [<ffffffffa07f89e0>] ? ptlrpc_main+0x0/0x1870 [ptlrpc]
13:06:43: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
13:06:43:
13:06:43:Kernel panic - not syncing: LBUG

Issue Links

• is duplicated by: LU-6472 OI scrub causes MDS reboot
• is related to: LU-2748 OSD uses kmalloc with high order to allocate a keys