[LU-4622] Failure on test suite recovery-small test_19a: Cannot allocate memory

• Type: Bug
• Resolution: Duplicate
• Priority: Minor
• Affects Version/s: Lustre 2.6.0
• Environment: lustre-master build # 1877
• Severity: 3
• Assignee: WC Triage
• Reporter: Maloo

Description

This issue was created by maloo for sarah <[e-mail]>

This issue relates to the following test suite run: http://maloo.whamcloud.com/test_sets/5479b506-9088-11e3-91ee-52540035b04c.

The sub-test test_19a failed with the following error:

failed to mount /mnt/lustre2

Info required for matching: recovery-small 19a

Issue Links

• is duplicated by: LU-4357 page allocation failure. mode:0x40 caused by missing __GFP_WAIT flag

Comments

[Sarah Liu]

client 1 console:

12:55:09:Lustre: DEBUG MARKER: == recovery-small test 19a: test expired_lock_main on mds (2867) == 12:54:52 (1391806492)
12:55:10:Lustre: DEBUG MARKER: mkdir -p /mnt/lustre2
12:55:10:Lustre: DEBUG MARKER: mount -t lustre -o user_xattr,flock client-32vm3:client-32vm7:/lustre /mnt/lustre2
12:55:10:mount.lustre: page allocation failure. order:2, mode:0x40
12:55:10:Pid: 22161, comm: mount.lustre Not tainted 2.6.32-358.23.2.el6.x86_64 #1
12:55:10:Call Trace:
12:55:10: [<ffffffff8112c287>] ? __alloc_pages_nodemask+0x757/0x8d0
12:55:11: [<ffffffff81281436>] ? vsnprintf+0x336/0x5e0
12:55:11: [<ffffffff81281436>] ? vsnprintf+0x336/0x5e0
12:55:11: [<ffffffff81166dc2>] ? kmem_getpages+0x62/0x170
12:55:11: [<ffffffff811679da>] ? fallback_alloc+0x1ba/0x270
12:55:11: [<ffffffff8116742f>] ? cache_grow+0x2cf/0x320
12:55:11: [<ffffffff81167759>] ? ____cache_alloc_node+0x99/0x160
12:55:12: [<ffffffff81167fc7>] ? kmem_cache_alloc_trace+0x127/0x1b0
12:55:13: [<ffffffffa0b5d582>] ? ll_init_sbi+0x52/0x4b0 [lustre]
12:55:13: [<ffffffffa03efa81>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
12:55:13: [<ffffffffa0b66fed>] ? ll_fill_super+0xfd/0x14b0 [lustre]
12:55:13: [<ffffffffa057c9fd>] ? lustre_fill_super+0x34d/0x510 [obdclass]
12:55:13: [<ffffffffa057c6b0>] ? lustre_fill_super+0x0/0x510 [obdclass]
12:55:14: [<ffffffff811845ff>] ? get_sb_nodev+0x5f/0xa0
12:55:14: [<ffffffffa05745a5>] ? lustre_get_sb+0x25/0x30 [obdclass]
12:55:14: [<ffffffff81183c1b>] ? vfs_kern_mount+0x7b/0x1b0
12:55:15: [<ffffffff81183dc2>] ? do_kern_mount+0x52/0x130
12:55:15: [<ffffffff811a3f22>] ? do_mount+0x2d2/0x8d0
12:55:15: [<ffffffff811a45b0>] ? sys_mount+0x90/0xe0
12:55:15: [<ffffffff8100b072>] ? system_call_fastpath+0x16/0x1b
12:55:15:Mem-Info:
12:55:15:Node 0 DMA per-cpu:
12:55:16:CPU    0: hi:    0, btch:   1 usd:   0
12:55:16:CPU    1: hi:    0, btch:   1 usd:   0
12:55:16:Node 0 DMA32 per-cpu:
12:55:16:CPU    0: hi:  186, btch:  31 usd:  52
12:55:17:CPU    1: hi:  186, btch:  31 usd: 170
12:55:17:active_anon:3085 inactive_anon:11754 isolated_anon:0
12:55:17: active_file:158271 inactive_file:156504 isolated_file:0
12:55:18: unevictable:0 dirty:32 writeback:1 unstable:0
12:55:19: free:115643 slab_reclaimable:13212 slab_unreclaimable:7920
12:55:19: mapped:1516 shmem:47 pagetables:1173 bounce:0
12:55:20:Node 0 DMA free:8280kB min:332kB low:412kB high:496kB active_anon:0kB inactive_anon:0kB active_file:7184kB inactive_file:36kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15324kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:200kB slab_unreclaimable:32kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
12:55:20:lowmem_reserve[]: 0 2003 2003 2003
12:55:20:Node 0 DMA32 free:472396kB min:44720kB low:55900kB high:67080kB active_anon:12340kB inactive_anon:47016kB active_file:617068kB inactive_file:617020kB unevictable:0kB isolated(anon):0kB isolated(file):112kB present:2052064kB mlocked:0kB dirty:128kB writeback:4kB mapped:6064kB shmem:188kB slab_reclaimable:52312kB slab_unreclaimable:31648kB kernel_stack:1760kB pagetables:4692kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:32 all_unreclaimable? no
12:55:20:lowmem_reserve[]: 0 0 0 0
12:55:21:Node 0 DMA: 0*4kB 1*8kB 1*16kB 2*32kB 0*64kB 0*128kB 2*256kB 1*512kB 1*1024kB 3*2048kB 0*4096kB = 8280kB
12:55:21:Node 0 DMA32: 74551*4kB 21798*8kB 637*16kB 3*32kB 3*64kB 1*128kB 2*256kB 2*512kB 2*1024kB 0*2048kB 0*4096kB = 486780kB
12:55:21:306123 total pagecache pages
12:55:21:49 pages in swap cache
12:55:21:Swap cache stats: add 49, delete 0, find 0/0
12:55:21:Free swap  = 4128564kB
12:55:22:Total swap = 4128760kB
12:55:22:524284 pages RAM
12:55:22:43709 pages reserved
12:55:22:305583 pages shared
12:55:23:49985 pages non-shared
12:55:23:LustreError: 22161:0:(obd_mount.c:1326:lustre_fill_super()) Unable to mount 10.10.4.198@tcp:10.10.4.202@tcp:/lustre (-12)
12:55:23:Lustre: DEBUG MARKER: /usr/sbin/lctl mark  recovery-small test_19a: @@@@@@ FAIL: failed to mount \/mnt\/lustre2 
12:55:23:Lustre: DEBUG MARKER: recovery-small test_19a: @@@@@@ FAIL: failed to mount /mnt/lustre2
12:55:23:Lustre: DEBUG MARKER: /usr/sbin/lctl dk > /logdir/test_logs/2014-02-05/lustre-master-el6-x86_64--failover--1_9_1__1877__-70046501643780-100806/recovery-small.test_19a.debug_log.$(hostname -s).1391806493.log;
12:55:23:         dmesg > /logdir/test_logs/2014-02-05/lustre-master-el6-x86_6
12:55:23:Lustre: DEBUG MARKER: /usr/sbin/lctl mark == recovery-small test 19b: test expired_lock_main on ost \(2867\) == 12:54:54 \(1391806494\)