[LU-13414] Page allocation failure: order 1

• Type: Bug
• Resolution: Duplicate
• Priority: Critical
• Affects Version/s: Lustre 2.12.3
• Severity: 2
• Assignee: Serguei Smirnov
• Reporter: Mahmoud Hanafi

Description

Client crashed with page allocation

[1585943028.070342] Lustre: 4989:0:(client.c:2133:ptlrpc_expire_one_request()) Skipped 6 previous similar messages^M

[1585943028.082342] Lustre: nbp1-MDT0000-mdc-ffff95b11d94a000: Connection to nbp1-MDT0000 (at 10.151.26.117@o2ib) was lost; in progress operations using this service will wait for recovery to complete^M

[1585943985.970771] ldlm_bl_58: page allocation failure: order:1, mode:0x1604040(GFP_NOFS|__GFP_COMP|__GFP_NOTRACK), nodemask=(null)^M

[1585943985.982770] ldlm_bl_110: ^M

[1585943985.986770] ldlm_bl_09: page allocation failure: order:1, mode:0x1604040(GFP_NOFS|__GFP_COMP|__GFP_NOTRACK), nodemask=(null)^M

[1585943985.998770] CPU: 13 PID: 42680 Comm: ldlm_bl_09 Tainted: G           OE      4.12.14-95.48.1.20200304-nasa #1 SLE12-SP4 (unreleased)^M

[1585943986.010769] Hardware name: SGI.COM C1104-RP7/X9DRW-3LN4F+/X9DRW-3TF+, BIOS 3.00 09/12/2013^M

[1585943986.018769] Call Trace:^M

[1585943986.022769]  dump_stack+0x5a/0x75^M

[1585943986.022769]  warn_alloc+0xf0/0x190^M

[1585943986.026769]  __alloc_pages_slowpath+0x865/0xa0d^M

[1585943986.034768]  __alloc_pages_nodemask+0x1e9/0x210^M

[1585943986.038768]  cache_grow_begin+0x85/0x560^M

[1585943986.042768]  fallback_alloc+0x167/0x1f0^M

[1585943986.046768]  kmem_cache_alloc+0x187/0x1d0^M

[1585943986.050768]  ptlrpc_request_cache_alloc+0x26/0x100 [ptlrpc]^M

[1585943986.058767] ldlm_bl_10: page allocation failure: order:1^M

[1585943986.062767]  ptlrpc_request_alloc_internal+0x1e/0x540 [ptlrpc]^M

[1585943986.070767] , mode:0x1604040(GFP_NOFS|__GFP_COMP|__GFP_NOTRACK), nodemask=^M

[1585943986.078767]  ldlm_cli_cancel_req+0x161/0x6c0 [ptlrpc]^M

[1585943986.082767] (null)^M

[1585943986.082767]  ldlm_cli_cancel_list+0x268/0x3b0 [ptlrpc]^M

[1585943986.090766] / mems_allowed=0-1^M

[1585943986.094766]  ldlm_bl_thread_main+0x3d1/0x8b0 [ptlrpc]^M

[1585943986.098766] ldlm_bl_02: page allocation failure: order:1^M

[1585943986.106766]  ? wake_up_q+0x70/0x70^M

[1585943986.110766] , mode:0x1604040(GFP_NOFS|__GFP_COMP|__GFP_NOTRACK), nodemask=^M

[1585943986.114765]  kthread+0xff/0x140^M

[1585943986.118765] (null)^M

[1585943986.122765]  ? ldlm_handle_bl_callback+0x4d0/0x4d0 [ptlrpc]^M

[1585943986.126765]  ? __kthread_parkme+0x70/0x70^M

[1585943986.130765] /^M

[1585943986.134765]  ret_from_fork+0x35/0x40^M

[1585943986.138764]  mems_allowed=0-1^M

[1585943986.142764] CPU: 2 PID: 46182 Comm: ldlm_bl_10 Tainted: G           OE      4.12.14-95.48.1.20200304-nasa #1 SLE12-SP4 (unreleased)^M

[1585943986.154764] Mem-Info:^M

[1585943986.158764] Hardware name: SGI.COM C1104-RP7/X9DRW-3LN4F+/X9DRW-3TF+, BIOS 3.00 09/12/2013^M

[1585943986.166763] Call Trace:^M

[1585943986.166763] active_anon:5278063 inactive_anon:527834 isolated_anon:0^M

[1585943986.166763]  active_file:829456 inactive_file:220089 isolated_file:0^M

[1585943986.166763]  unevictable:20 dirty:37 writeback:7936 unstable:0^M

[1585943986.166763]  slab_reclaimable:480664 slab_unreclaimable:7863786^M

[1585943986.166763]  mapped:125735 shmem:15500 pagetables:35760 bounce:0^M

[1585943986.166763]  free:63139 free_pcp:4009 free_cma:0^M

[1585943986.206762]  dump_stack+0x5a/0x75^M

[1585943986.210762] Node 0 active_anon:11355416kB inactive_anon:1135584kB active_file:2393776kB inactive_file:573716kB unevictable:80kB isolated(anon):0kB isolated(file):0kB mapped:297864kB dirty:96kB writeback:18944kB shmem:41224kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 991232kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no^M

[1585943986.238761]  warn_alloc+0xf0/0x190^M

[1585943986.242761] Node 1 active_anon:9756836kB inactive_anon:975752kB active_file:924048kB inactive_file:306640kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:205076kB dirty:52kB writeback:12800kB shmem:20776kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 6144kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no^M

[1585943986.270760]  __alloc_pages_slowpath+0x865/0xa0d^M

[1585943986.278759] Node 0 DMA free:4kB min:60kB low:72kB high:84kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15920kB managed:15748kB mlocked:0kB slab_reclaimable:0kB slab_unreclaimable:15744kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB^M