Details
-
Bug
-
Resolution: Duplicate
-
Major
-
Lustre 2.5.0
-
OS: CentOS 6.6
Kernel: 2.6.32-358.18.1.el6_lustre.x86_64
Proc: 2 x AMD Opteron(tm) Processor 4334
Mem: 32GB
IB: Mellanox FDR(10) Firmware version: 2.31.5050
lustre: 2.5.0 patchless_client
-
3
-
9223372036854775807
Description
After an RPC triggered OI scrub, the MDS reboots right after a page allocation failure. I noticed this happening a few times now and can fully replicate this from the MDS while running a LFSCK. Each time it reboots, I've been running e2fsck to fix the mdt and it returns fine with no badblocks. I was able to run a LFSCK in dryrun without it failing. I'm posting a couple of sequences of this and the call trace that happens just before it reboots. The only bug that I found that was similar was LU-2818 when the ASSERTION( rc == 0 ) failed happens. Sometimes the I get this though and it continues without a reboot, so I'm not sure if it was the same issue: LustreError: 7542:0:(mdt_lvb.c:157:mdt_lvbo_fill()) panlfs3-MDT0000: expected 56 actual 0.
Currently I have disabled the oi_scrub and haven't had this happen all week. But I don't want to leave it like that since obviously the client has been triggering such an event.
Apr 13 13:48:40 pan3mds kernel: LustreError: 0-0: panlfs3-MDT0000: trigger OI scrub by RPC for [0x200006ec6:0x2212:0x0], rc = 0 [1]
Apr 13 13:54:15 pan3mds kernel: mdt03_026: page allocation failure. order:1, mode:0x40
Apr 13 13:54:15 pan3mds kernel: Pid: 53466, comm: mdt03_026 Not tainted 2.6.32-358.18.1.el6_lustre.x86_64 #1
Apr 13 13:54:15 pan3mds kernel: Call Trace:
Apr 13 13:54:15 pan3mds kernel: [<ffffffff8112c257>] ? __alloc_pages_nodemask+0x757/0x8d0
Apr 13 13:54:15 pan3mds kernel: [<ffffffffa0ce202c>] ? osd_object_init+0xb0c/0x11b0 [osd_ldiskfs]
Apr 13 13:54:15 pan3mds kernel: [<ffffffff81166d92>] ? kmem_getpages+0x62/0x170
Apr 13 13:54:15 pan3mds kernel: [<ffffffff811679aa>] ? fallback_alloc+0x1ba/0x270
Apr 13 13:54:15 pan3mds kernel: [<ffffffff811673ff>] ? cache_grow+0x2cf/0x320
Apr 13 13:54:15 pan3mds kernel: [<ffffffff81167729>] ? ____cache_alloc_node+0x99/0x160
Apr 13 13:54:15 pan3mds kernel: [<ffffffff81167f97>] ? kmem_cache_alloc_trace+0x127/0x1b0
Apr 13 13:54:15 pan3mds kernel: [<ffffffffa0cd8e35>] ? osd_key_init+0x25/0x5a0 [osd_ldiskfs]
Apr 13 13:54:15 pan3mds kernel: [<ffffffffa059246f>] ? keys_fill+0x6f/0x190 [obdclass]
Apr 13 13:54:15 pan3mds kernel: [<ffffffffa0596723>] ? lu_context_init+0xa3/0x240 [obdclass]
Apr 13 13:54:15 pan3mds kernel: [<ffffffffa05968de>] ? lu_env_init+0x1e/0x30 [obdclass]
Apr 13 13:54:15 pan3mds kernel: [<ffffffffa0e1894b>] ? mdt_lvbo_fill+0x1ab/0x840 [mdt]
Apr 13 13:54:15 pan3mds kernel: [<ffffffffa0e187a0>] ? mdt_lvbo_fill+0x0/0x840 [mdt]
Apr 13 13:54:15 pan3mds kernel: [<ffffffffa06fa2b1>] ? ldlm_handle_enqueue0+0x621/0x10a0 [ptlrpc]
Apr 13 13:54:15 pan3mds kernel: [<ffffffffa0ddbd96>] ? mdt_enqueue+0x46/0xe0 [mdt]
Apr 13 13:54:15 pan3mds kernel: [<ffffffffa0de2a8a>] ? mdt_handle_common+0x52a/0x1470 [mdt]
Apr 13 13:54:15 pan3mds kernel: [<ffffffffa0e1cc55>] ? mds_regular_handle+0x15/0x20 [mdt]
Apr 13 13:54:15 pan3mds kernel: [<ffffffffa0729e25>] ? ptlrpc_server_handle_request+0x385/0xc00 [ptlrpc]
Apr 13 13:54:15 pan3mds kernel: [<ffffffffa04444ce>] ? cfs_timer_arm+0xe/0x10 [libcfs]
Apr 13 13:54:15 pan3mds kernel: [<ffffffffa045527f>] ? lc_watchdog_touch+0x6f/0x170 [libcfs]
Apr 13 13:54:15 pan3mds kernel: [<ffffffffa07214c9>] ? ptlrpc_wait_event+0xa9/0x2d0 [ptlrpc]
Apr 13 13:54:15 pan3mds kernel: [<ffffffff81051439>] ? __wake_up_common+0x59/0x90
Apr 13 13:54:15 pan3mds kernel: [<ffffffffa072b18d>] ? ptlrpc_main+0xaed/0x1740 [ptlrpc]
Apr 13 13:54:15 pan3mds kernel: [<ffffffffa072a6a0>] ? ptlrpc_main+0x0/0x1740 [ptlrpc]
Apr 13 13:54:15 pan3mds kernel: [<ffffffff81096a36>] ? kthread+0x96/0xa0
Apr 13 13:54:15 pan3mds kernel: [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Apr 13 13:54:15 pan3mds kernel: [<ffffffff810969a0>] ? kthread+0x0/0xa0
Apr 13 13:54:15 pan3mds kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Apr 13 13:54:15 pan3mds kernel: Mem-Info:
This time right after I fire up LFSCK this happened:
Apr 13 15:19:32 pan3mds kernel: Lustre: panlfs3-MDT0000: Recovery over after 0:59, of 1033 clients 1033 recovered and 0 were evicted.
Apr 13 17:19:38 pan3mds kernel: mdt00_000: page allocation failure. order:1, mode:0x40
Apr 13 17:19:38 pan3mds kernel: Pid: 7636, comm: mdt00_000 Not tainted 2.6.32-358.18.1.el6_lustre.x86_64 #1
Apr 13 17:19:38 pan3mds kernel: Call Trace:
Apr 13 17:19:38 pan3mds kernel: [<ffffffff8112c257>] ? __alloc_pages_nodemask+0x757/0x8d0
Apr 13 17:19:38 pan3mds kernel: [<ffffffffa0ce202c>] ? osd_object_init+0xb0c/0x11b0 [osd_ldiskfs]
Apr 13 17:19:38 pan3mds kernel: [<ffffffff81166d92>] ? kmem_getpages+0x62/0x170
Apr 13 17:19:38 pan3mds kernel: [<ffffffff811679aa>] ? fallback_alloc+0x1ba/0x270
Apr 13 17:19:38 pan3mds kernel: [<ffffffff811673ff>] ? cache_grow+0x2cf/0x320
Apr 13 17:19:38 pan3mds kernel: [<ffffffff81167729>] ? ____cache_alloc_node+0x99/0x160
Apr 13 17:19:38 pan3mds kernel: [<ffffffff81167f97>] ? kmem_cache_alloc_trace+0x127/0x1b0
Apr 13 17:19:38 pan3mds kernel: [<ffffffffa0cd8e35>] ? osd_key_init+0x25/0x5a0 [osd_ldiskfs]
Apr 13 17:19:38 pan3mds kernel: [<ffffffffa059246f>] ? keys_fill+0x6f/0x190 [obdclass]
Apr 13 17:19:38 pan3mds kernel: [<ffffffffa0596723>] ? lu_context_init+0xa3/0x240 [obdclass]
Apr 13 17:19:38 pan3mds kernel: [<ffffffffa05968de>] ? lu_env_init+0x1e/0x30 [obdclass]
Apr 13 17:19:38 pan3mds kernel: [<ffffffffa0e1894b>] ? mdt_lvbo_fill+0x1ab/0x840 [mdt]
Apr 13 17:19:38 pan3mds kernel: [<ffffffffa0e187a0>] ? mdt_lvbo_fill+0x0/0x840 [mdt]
Apr 13 17:19:38 pan3mds kernel: [<ffffffffa06fa2b1>] ? ldlm_handle_enqueue0+0x621/0x10a0 [ptlrpc]
Apr 13 17:19:38 pan3mds kernel: [<ffffffffa0ddbd96>] ? mdt_enqueue+0x46/0xe0 [mdt]
Apr 13 17:19:38 pan3mds kernel: [<ffffffffa0de2a8a>] ? mdt_handle_common+0x52a/0x1470 [mdt]
Apr 13 17:19:38 pan3mds kernel: [<ffffffffa0e1cc55>] ? mds_regular_handle+0x15/0x20 [mdt]
Apr 13 17:19:38 pan3mds kernel: [<ffffffffa0729e25>] ? ptlrpc_server_handle_request+0x385/0xc00 [ptlrpc]
Apr 13 17:19:38 pan3mds kernel: [<ffffffffa04444ce>] ? cfs_timer_arm+0xe/0x10 [libcfs]
Apr 13 17:19:38 pan3mds kernel: [<ffffffffa045527f>] ? lc_watchdog_touch+0x6f/0x170 [libcfs]
Apr 13 17:19:38 pan3mds kernel: [<ffffffffa07214c9>] ? ptlrpc_wait_event+0xa9/0x2d0 [ptlrpc]
Apr 13 17:19:38 pan3mds kernel: [<ffffffff81051439>] ? __wake_up_common+0x59/0x90
Apr 13 17:19:38 pan3mds kernel: [<ffffffffa072b18d>] ? ptlrpc_main+0xaed/0x1740 [ptlrpc]
Apr 13 17:19:38 pan3mds kernel: [<ffffffffa072a6a0>] ? ptlrpc_main+0x0/0x1740 [ptlrpc]
Apr 13 17:19:38 pan3mds kernel: [<ffffffff81096a36>] ? kthread+0x96/0xa0
Apr 13 17:19:38 pan3mds kernel: [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Apr 13 17:19:38 pan3mds kernel: [<ffffffff810969a0>] ? kthread+0x0/0xa0
Apr 13 17:19:38 pan3mds kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Apr 13 17:19:38 pan3mds kernel: Mem-Info:
Apr 13 17:19:38 pan3mds kernel: active_anon:15433 inactive_anon:91815 isolated_anon:0
Apr 13 17:19:38 pan3mds kernel: active_file:2189974 inactive_file:2215127 isolated_file:0
Apr 13 17:19:38 pan3mds kernel: unevictable:0 dirty:313 writeback:0 unstable:0
Apr 13 17:19:38 pan3mds kernel: free:50741 slab_reclaimable:2534646 slab_unreclaimable:849962
Apr 13 17:19:38 pan3mds kernel: mapped:11340 shmem:11 pagetables:2424 bounce:0
Apr 13 17:19:38 pan3mds kernel: Node 0 DMA free:15668kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15280kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Apr 13 17:19:38 pan3mds kernel: lowmem_reserve[]: 0 2990 16120 16120
Apr 13 17:19:38 pan3mds kernel: Node 0 DMA32 free:64320kB min:8348kB low:10432kB high:12520kB active_anon:2052kB inactive_anon:2944kB active_file:176424kB inactive_file:231988kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3062596kB mlocked:0kB dirty:220kB writeback:0kB mapped:80kB shmem:0kB slab_reclaimable:1671828kB slab_unreclaimable:515616kB kernel_stack:32kB pagetables:8kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr 13 17:19:38 pan3mds kernel: lowmem_reserve[]: 0 0 13130 13130
Apr 13 17:19:38 pan3mds kernel: Node 0 Normal free:57464kB min:36652kB low:45812kB high:54976kB active_anon:47808kB inactive_anon:196472kB active_file:4026680kB inactive_file:4071040kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13445120kB mlocked:0kB dirty:252kB writeback:0kB mapped:37744kB shmem:16kB slab_reclaimable:3539496kB slab_unreclaimable:1180808kB kernel_stack:1920kB pagetables:4892kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr 13 17:19:38 pan3mds kernel: lowmem_reserve[]: 0 0 0 0
Apr 13 17:19:38 pan3mds kernel: Node 1 Normal free:64892kB min:45064kB low:56328kB high:67596kB active_anon:11872kB inactive_anon:167844kB active_file:4557296kB inactive_file:4557480kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:16531680kB mlocked:0kB dirty:780kB writeback:0kB mapped:7536kB shmem:28kB slab_reclaimable:4927260kB slab_unreclaimable:1703424kB kernel_stack:4024kB pagetables:4796kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:28 all_unreclaimable? no
Apr 13 17:19:38 pan3mds kernel: lowmem_reserve[]: 0 0 0 0
Apr 13 17:19:38 pan3mds kernel: Node 0 DMA: 1*4kB 0*8kB 1*16kB 1*32kB 2*64kB 1*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15668kB
Apr 13 17:19:38 pan3mds kernel: Node 0 DMA32: 15585*4kB 7*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 64444kB
Apr 13 17:19:38 pan3mds kernel: Node 0 Normal: 13768*4kB 43*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 57464kB
Apr 13 17:19:38 pan3mds kernel: Node 1 Normal: 14132*4kB 810*8kB 2*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 65088kB
Apr 13 17:19:38 pan3mds kernel: 4415403 total pagecache pages
Apr 13 17:19:38 pan3mds kernel: 10318 pages in swap cache
Apr 13 17:19:38 pan3mds kernel: Swap cache stats: add 102491, delete 92173, find 46023/51168
Apr 13 17:19:38 pan3mds kernel: Free swap = 16376952kB
Apr 13 17:19:38 pan3mds kernel: Total swap = 16498680kB
Apr 13 17:19:38 pan3mds kernel: 8384511 pages RAM
Apr 13 17:19:38 pan3mds kernel: 172523 pages reserved
Apr 13 17:19:38 pan3mds kernel: 4102992 pages shared
Apr 13 17:19:38 pan3mds kernel: 4039786 pages non-shared
Apr 13 17:19:38 pan3mds kernel: LustreError: 7636:0:(mdt_lvb.c:125:mdt_lvbo_fill()) ASSERTION( rc == 0 ) failed:
Apr 13 17:19:38 pan3mds kernel: LustreError: 7636:0:(mdt_lvb.c:125:mdt_lvbo_fill()) LBUG
Apr 13 17:19:38 pan3mds kernel: Pid: 7636, comm: mdt00_000
Apr 13 17:19:38 pan3mds kernel:
Apr 13 17:19:38 pan3mds kernel: Call Trace:
Apr 13 17:19:38 pan3mds kernel: [<ffffffffa0443895>] libcfs_debug_dumpstack+0x55/0x80 [libcfs]
Apr 13 17:19:38 pan3mds kernel: [<ffffffffa0443e97>] lbug_with_loc+0x47/0xb0 [libcfs]
Apr 13 17:19:38 pan3mds kernel: [<ffffffffa0e18f04>] mdt_lvbo_fill+0x764/0x840 [mdt]
Apr 13 17:19:38 pan3mds kernel: [<ffffffffa0e187a0>] ? mdt_lvbo_fill+0x0/0x840 [mdt]
Apr 13 17:19:38 pan3mds kernel: [<ffffffffa06fa2b1>] ldlm_handle_enqueue0+0x621/0x10a0 [ptlrpc]
Apr 13 17:19:38 pan3mds kernel: [<ffffffffa0ddbd96>] mdt_enqueue+0x46/0xe0 [mdt]
Apr 13 17:19:38 pan3mds kernel: [<ffffffffa0de2a8a>] mdt_handle_common+0x52a/0x1470 [mdt]
Apr 13 17:19:38 pan3mds kernel: [<ffffffffa0e1cc55>] mds_regular_handle+0x15/0x20 [mdt]
Apr 13 17:19:38 pan3mds kernel: [<ffffffffa0729e25>] ptlrpc_server_handle_request+0x385/0xc00 [ptlrpc]
Apr 13 17:19:38 pan3mds kernel: [<ffffffffa04444ce>] ? cfs_timer_arm+0xe/0x10 [libcfs]
Apr 13 17:19:38 pan3mds kernel: [<ffffffffa045527f>] ? lc_watchdog_touch+0x6f/0x170 [libcfs]
Apr 13 17:19:38 pan3mds kernel: [<ffffffffa07214c9>] ? ptlrpc_wait_event+0xa9/0x2d0 [ptlrpc]
Apr 13 17:19:38 pan3mds kernel: [<ffffffff81051439>] ? __wake_up_common+0x59/0x90
Apr 13 17:19:38 pan3mds kernel: [<ffffffffa072b18d>] ptlrpc_main+0xaed/0x1740 [ptlrpc]
Apr 13 17:19:38 pan3mds kernel: [<ffffffffa072a6a0>] ? ptlrpc_main+0x0/0x1740 [ptlrpc]
Apr 13 17:19:38 pan3mds kernel: [<ffffffff81096a36>] kthread+0x96/0xa0
Apr 13 17:19:38 pan3mds kernel: [<ffffffff8100c0ca>] child_rip+0xa/0x20
Apr 13 17:19:38 pan3mds kernel: [<ffffffff810969a0>] ? kthread+0x0/0xa0
Apr 13 17:19:38 pan3mds kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Apr 13 17:19:38 pan3mds kernel:
Then another attempt with LFSCK -->
Apr 13 22:10:12 pan3mds kernel: mdt03_003: page allocation failure. order:1, mode:0x40
Apr 13 22:10:12 pan3mds kernel: Pid: 7544, comm: mdt03_003 Not tainted 2.6.32-358.18.1.el6_lustre.x86_64 #1
Apr 13 22:10:12 pan3mds kernel: Call Trace:
Apr 13 22:10:12 pan3mds kernel: [<ffffffff8112c257>] ? __alloc_pages_nodemask+0x757/0x8d0
Apr 13 22:10:12 pan3mds kernel: [<ffffffffa0ce202c>] ? osd_object_init+0xb0c/0x11b0 [osd_ldiskfs]
Apr 13 22:10:12 pan3mds kernel: [<ffffffff81166d92>] ? kmem_getpages+0x62/0x170
Apr 13 22:10:12 pan3mds kernel: [<ffffffff811679aa>] ? fallback_alloc+0x1ba/0x270
Apr 13 22:10:12 pan3mds kernel: [<ffffffff811673ff>] ? cache_grow+0x2cf/0x320
Apr 13 22:10:12 pan3mds kernel: [<ffffffff81167729>] ? ____cache_alloc_node+0x99/0x160
Apr 13 22:10:12 pan3mds kernel: [<ffffffff81167f97>] ? kmem_cache_alloc_trace+0x127/0x1b0
Apr 13 22:10:12 pan3mds kernel: [<ffffffffa0cd8e35>] ? osd_key_init+0x25/0x5a0 [osd_ldiskfs]
Apr 13 22:10:12 pan3mds kernel: [<ffffffffa059246f>] ? keys_fill+0x6f/0x190 [obdclass]
Apr 13 22:10:12 pan3mds kernel: [<ffffffffa0596723>] ? lu_context_init+0xa3/0x240 [obdclass]
Apr 13 22:10:12 pan3mds kernel: [<ffffffffa05968de>] ? lu_env_init+0x1e/0x30 [obdclass]
Apr 13 22:10:12 pan3mds kernel: [<ffffffffa0e1894b>] ? mdt_lvbo_fill+0x1ab/0x840 [mdt]
Apr 13 22:10:12 pan3mds kernel: [<ffffffffa0e187a0>] ? mdt_lvbo_fill+0x0/0x840 [mdt]
Apr 13 22:10:12 pan3mds kernel: [<ffffffffa06fa2b1>] ? ldlm_handle_enqueue0+0x621/0x10a0 [ptlrpc]
Apr 13 22:10:12 pan3mds kernel: [<ffffffffa0ddbd96>] ? mdt_enqueue+0x46/0xe0 [mdt]
Apr 13 22:10:12 pan3mds kernel: [<ffffffffa0de2a8a>] ? mdt_handle_common+0x52a/0x1470 [mdt]
Apr 13 22:10:12 pan3mds kernel: [<ffffffffa0e1cc55>] ? mds_regular_handle+0x15/0x20 [mdt]
Apr 13 22:10:12 pan3mds kernel: [<ffffffffa0729e25>] ? ptlrpc_server_handle_request+0x385/0xc00 [ptlrpc]
Apr 13 22:10:12 pan3mds kernel: [<ffffffffa04444ce>] ? cfs_timer_arm+0xe/0x10 [libcfs]
Apr 13 22:10:12 pan3mds kernel: [<ffffffffa045527f>] ? lc_watchdog_touch+0x6f/0x170 [libcfs]
Apr 13 22:10:12 pan3mds kernel: [<ffffffffa07214c9>] ? ptlrpc_wait_event+0xa9/0x2d0 [ptlrpc]
Apr 13 22:10:12 pan3mds kernel: [<ffffffff81051439>] ? __wake_up_common+0x59/0x90
Apr 13 22:10:12 pan3mds kernel: [<ffffffffa072b18d>] ? ptlrpc_main+0xaed/0x1740 [ptlrpc]
Apr 13 22:10:12 pan3mds kernel: [<ffffffffa072a6a0>] ? ptlrpc_main+0x0/0x1740 [ptlrpc]
Apr 13 22:10:12 pan3mds kernel: [<ffffffff81096a36>] ? kthread+0x96/0xa0
Apr 13 22:10:12 pan3mds kernel: [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Apr 13 22:10:12 pan3mds kernel: [<ffffffff810969a0>] ? kthread+0x0/0xa0
Apr 13 22:10:12 pan3mds kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Apr 13 22:10:12 pan3mds kernel: active_anon:23201 inactive_anon:88738 isolated_anon:0
Apr 13 22:10:12 pan3mds kernel: active_file:1738980 inactive_file:1296912 isolated_file:0
Apr 13 22:10:12 pan3mds kernel: unevictable:0 dirty:565 writeback:0 unstable:0
Apr 13 22:10:12 pan3mds kernel: free:64942 slab_reclaimable:3442925 slab_unreclaimable:1290832
Apr 13 22:10:12 pan3mds kernel: mapped:19115 shmem:10 pagetables:2491 bounce:0
Apr 13 22:10:12 pan3mds kernel: Node 0 DMA free:15668kB min:40kB low:48kB high:60kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15280kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Apr 13 22:10:12 pan3mds kernel: lowmem_reserve[]: 0 2990 16120 16120
Apr 13 22:10:12 pan3mds kernel: Node 0 DMA32 free:142368kB min:8348kB low:10432kB high:12520kB active_anon:0kB inactive_anon:4712kB active_file:184032kB inactive_file:148804kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3062596kB mlocked:0kB dirty:24kB writeback:0kB mapped:1364kB shmem:0kB slab_reclaimable:1832404kB slab_unreclaimable:351212kB kernel_stack:16kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:36 all_unreclaimable? no
Apr 13 22:10:12 pan3mds kernel: lowmem_reserve[]: 0 0 13130 13130
Apr 13 22:10:12 pan3mds kernel: Node 0 Normal free:45540kB min:36652kB low:45812kB high:54976kB active_anon:77112kB inactive_anon:214792kB active_file:3930824kB inactive_file:2217444kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13445120kB mlocked:0kB dirty:1656kB writeback:0kB mapped:63768kB shmem:8kB slab_reclaimable:4680096kB slab_unreclaimable:2090816kB kernel_stack:4856kB pagetables:5012kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:99 all_unreclaimable? no
Apr 13 22:10:12 pan3mds kernel: lowmem_reserve[]: 0 0 0 0
Apr 13 22:10:12 pan3mds kernel: Node 1 Normal free:56192kB min:45064kB low:56328kB high:67596kB active_anon:15692kB inactive_anon:135448kB active_file:2841064kB inactive_file:2821400kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:16531680kB mlocked:0kB dirty:580kB writeback:0kB mapped:11328kB shmem:32kB slab_reclaimable:7259200kB slab_unreclaimable:2721300kB kernel_stack:1152kB pagetables:4952kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:9 all_unreclaimable? no
Apr 13 22:10:12 pan3mds kernel: lowmem_reserve[]: 0 0 0 0
Apr 13 22:10:12 pan3mds kernel: Node 0 DMA: 1*4kB 0*8kB 1*16kB 1*32kB 2*64kB 1*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15668kB
Apr 13 22:10:12 pan3mds kernel: Node 0 DMA32: 34956*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 141872kB
Apr 13 22:10:12 pan3mds kernel: Node 0 Normal: 11190*4kB 3*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 45664kB
Apr 13 22:10:12 pan3mds kernel: Node 1 Normal: 14173*4kB 15*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 56812kB
Apr 13 22:10:12 pan3mds kernel: 3038613 total pagecache pages
Apr 13 22:10:12 pan3mds kernel: 2731 pages in swap cache
Apr 13 22:10:12 pan3mds kernel: Swap cache stats: add 114253, delete 111522, find 58808/64778
Apr 13 22:10:12 pan3mds kernel: Free swap = 16415432kB
Apr 13 22:10:12 pan3mds kernel: Total swap = 16498680kB
Apr 13 22:10:12 pan3mds kernel: 8384511 pages RAM
Apr 13 22:10:12 pan3mds kernel: 172523 pages reserved
Apr 13 22:10:12 pan3mds kernel: 2926474 pages shared
Apr 13 22:10:12 pan3mds kernel: 5212158 pages non-shared
Apr 13 22:10:12 pan3mds kernel: LustreError: 7544:0:(mdt_lvb.c:125:mdt_lvbo_fill()) ASSERTION( rc == 0 ) failed:
Apr 13 22:10:12 pan3mds kernel: LustreError: 7544:0:(mdt_lvb.c:125:mdt_lvbo_fill()) LBUG
Apr 13 22:10:12 pan3mds kernel: Pid: 7544, comm: mdt03_003
Apr 13 22:10:12 pan3mds kernel: Call Trace:
Apr 13 22:10:12 pan3mds kernel: [<ffffffffa0443895>] libcfs_debug_dumpstack+0x55/0x80 [libcfs]
Apr 13 22:10:12 pan3mds kernel: [<ffffffffa0443e97>] lbug_with_loc+0x47/0xb0 [libcfs]
Apr 13 22:10:12 pan3mds kernel: [<ffffffffa0e18f04>] mdt_lvbo_fill+0x764/0x840 [mdt]
Apr 13 22:10:12 pan3mds kernel: [<ffffffffa0e187a0>] ? mdt_lvbo_fill+0x0/0x840 [mdt]
Apr 13 22:10:12 pan3mds kernel: [<ffffffffa06fa2b1>] ldlm_handle_enqueue0+0x621/0x10a0 [ptlrpc]
Apr 13 22:10:12 pan3mds kernel: [<ffffffffa0ddbd96>] mdt_enqueue+0x46/0xe0 [mdt]
Apr 13 22:10:12 pan3mds kernel: [<ffffffffa0de2a8a>] mdt_handle_common+0x52a/0x1470 [mdt]
Apr 13 22:10:12 pan3mds kernel: [<ffffffffa0e1cc55>] mds_regular_handle+0x15/0x20 [mdt]
Apr 13 22:10:12 pan3mds kernel: [<ffffffffa0729e25>] ptlrpc_server_handle_request+0x385/0xc00 [ptlrpc]
Apr 13 22:10:12 pan3mds kernel: [<ffffffffa04444ce>] ? cfs_timer_arm+0xe/0x10 [libcfs]
Apr 13 22:10:12 pan3mds kernel: [<ffffffffa045527f>] ? lc_watchdog_touch+0x6f/0x170 [libcfs]
Apr 13 22:10:12 pan3mds kernel: [<ffffffffa07214c9>] ? ptlrpc_wait_event+0xa9/0x2d0 [ptlrpc]
Apr 13 22:10:12 pan3mds kernel: [<ffffffff81051439>] ? __wake_up_common+0x59/0x90
Apr 13 22:10:12 pan3mds kernel: [<ffffffffa072b18d>] ptlrpc_main+0xaed/0x1740 [ptlrpc]
Apr 13 22:10:12 pan3mds kernel: [<ffffffffa072a6a0>] ? ptlrpc_main+0x0/0x1740 [ptlrpc]
Apr 13 22:10:12 pan3mds kernel: [<ffffffff81096a36>] kthread+0x96/0xa0
Apr 13 22:10:12 pan3mds kernel: [<ffffffff8100c0ca>] child_rip+0xa/0x20
Apr 13 22:10:12 pan3mds kernel: [<ffffffff810969a0>] ? kthread+0x0/0xa0
Apr 13 22:10:12 pan3mds kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Apr 13 22:10:12 pan3mds kernel: