Details
-
Bug
-
Resolution: Duplicate
-
Blocker
-
None
-
Lustre 2.3.0
-
None
-
LLNL/Hyperon
-
3
-
4257
Description
Running miranda-io, OSS dies with oom-killer
Sep 28 07:26:07 hyperion-dit29 kernel: Lustre: 5836:0:(lustre_log.h:474:llog_group_set_export()) Skipped 13 previous similar messages Sep 28 07:26:07 hyperion-dit29 kernel: Lustre: 5836:0:(llog_net.c:162:llog_receptor_accept()) changing the import ffff880254a4c800 - ffff88011c83e000 Sep 28 07:26:07 hyperion-dit29 kernel: Lustre: 5836:0:(llog_net.c:162:llog_receptor_accept()) Skipped 13 previous similar messages Sep 28 07:30:09 hyperion-dit29 cfengine:hyperion-dit29[7360]: stat: No such file or directory Sep 28 07:30:09 hyperion-dit29 cfengine:hyperion-dit29[7360]: stat: No such file or directory Sep 28 07:30:09 hyperion-dit29 cfengine:hyperion-dit29[7360]: stat: No such file or directory Sep 28 07:30:11 hyperion-dit29 syslog-ng[3472]: EOF occurred while idle; fd='10' Sep 28 07:30:14 hyperion-dit29 cfengine:hyperion-dit29[7360]: stat: No such file or directory Sep 28 07:30:14 hyperion-dit29 cfengine:hyperion-dit29[7360]: stat: No such file or directory Sep 28 11:34:02 hyperion-dit29 LDAPOTP-AUTH[8025]: root@ehyperion576 as root: cmd='/usr/bin/pdcp -p -y -z /etc' Sep 28 11:43:08 hyperion-dit29 kernel: ll_ost_io02_036 invoked oom-killer: gfp_mask=0x200d2, order=0, oom_adj=-17, oom_score_adj=0 Sep 28 11:43:08 hyperion-dit29 kernel: ll_ost_io02_036 cpuset=/ mems_allowed=1 Sep 28 11:43:08 hyperion-dit29 kernel: Pid: 6335, comm: ll_ost_io02_036 Tainted: P --------------- 2.6.32-279.5.1.el6_lustre.x86_64 #1 Sep 28 11:43:08 hyperion-dit29 kernel: Call Trace: Sep 28 11:43:08 hyperion-dit29 kernel: [<ffffffff810c4aa1>] ? cpuset_print_task_mems_allowed+0x91/0xb0 Sep 28 11:43:08 hyperion-dit29 kernel: [<ffffffff81117210>] ? dump_header+0x90/0x1b0 Sep 28 11:43:08 hyperion-dit29 kernel: [<ffffffff810c58b1>] ? cpuset_mems_allowed_intersects+0x21/0x30 Sep 28 11:43:08 hyperion-dit29 kernel: [<ffffffff81117692>] ? oom_kill_process+0x82/0x2a0 Sep 28 11:43:08 hyperion-dit29 kernel: [<ffffffff8111758e>] ? select_bad_process+0x9e/0x120 Sep 28 11:43:08 hyperion-dit29 kernel: [<ffffffff81117ad0>] ? out_of_memory+0x220/0x3c0 Sep 28 11:43:08 hyperion-dit29 kernel: [<ffffffff811277ee>] ? __alloc_pages_nodemask+0x89e/0x940 Sep 28 11:43:08 hyperion-dit29 kernel: [<ffffffff8115c30a>] ? alloc_pages_current+0xaa/0x110 Sep 28 11:43:08 hyperion-dit29 kernel: [<ffffffff81114617>] ? __page_cache_alloc+0x87/0x90 Sep 28 11:43:08 hyperion-dit29 kernel: [<ffffffff8111541f>] ? find_or_create_page+0x4f/0xb0 Sep 28 11:43:08 hyperion-dit29 kernel: [<ffffffffa0e361b5>] ? filter_get_page+0x35/0x70 [obdfilter] Sep 28 11:43:08 hyperion-dit29 kernel: [<ffffffffa0e378a8>] ? filter_preprw_write+0x12b8/0x2340 [obdfilter] Sep 28 11:43:08 hyperion-dit29 kernel: [<ffffffffa04d3edb>] ? lnet_ni_send+0x4b/0x110 [lnet] Sep 28 11:43:08 hyperion-dit29 kernel: [<ffffffffa0a1ce6b>] ? null_alloc_rs+0x1ab/0x3b0 [ptlrpc] Sep 28 11:43:08 hyperion-dit29 kernel: [<ffffffffa0a0a024>] ? sptlrpc_svc_alloc_rs+0x74/0x2d0 [ptlrpc] Sep 28 11:43:08 hyperion-dit29 kernel: [<ffffffffa0e39730>] ? filter_preprw+0x80/0xa0 [obdfilter] Sep 28 11:43:08 hyperion-dit29 kernel: [<ffffffffa0b0f81c>] ? obd_preprw+0x12c/0x3d0 [ost] Sep 28 11:43:08 hyperion-dit29 kernel: [<ffffffffa0b1698a>] ? ost_brw_write+0x87a/0x1600 [ost] Sep 28 11:43:08 hyperion-dit29 kernel: [<ffffffffa09d36de>] ? ptlrpc_send_reply+0x28e/0x860 [ptlrpc] Sep 28 11:43:08 hyperion-dit29 kernel: [<ffffffffa09db1dc>] ? lustre_msg_get_version+0x8c/0x100 [ptlrpc] Sep 28 11:43:08 hyperion-dit29 kernel: [<ffffffffa09db338>] ? lustre_msg_check_version+0xe8/0x100 [ptlrpc] Sep 28 11:43:08 hyperion-dit29 kernel: [<ffffffffa0b1d02c>] ? ost_handle+0x360c/0x4850 [ost] Sep 28 11:43:08 hyperion-dit29 kernel: [<ffffffffa09db8fc>] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc] Sep 28 11:43:08 hyperion-dit29 kernel: [<ffffffffa09e26fb>] ? ptlrpc_update_export_timer+0x4b/0x470 [ptlrpc] Sep 28 11:43:08 hyperion-dit29 kernel: [<ffffffffa09eab3c>] ? ptlrpc_server_handle_request+0x41c/0xe00 [ptlrpc] Sep 28 11:43:08 hyperion-dit29 kernel: [<ffffffffa042665e>] ? cfs_timer_arm+0xe/0x10 [libcfs] Sep 28 11:43:08 hyperion-dit29 kernel: [<ffffffffa043813f>] ? lc_watchdog_touch+0x6f/0x180 [libcfs] Sep 28 11:43:08 hyperion-dit29 kernel: [<ffffffffa09e1f37>] ? ptlrpc_wait_event+0xa7/0x2a0 [ptlrpc] Sep 28 11:43:08 hyperion-dit29 kernel: [<ffffffff810533f3>] ? __wake_up+0x53/0x70 Sep 28 11:43:08 hyperion-dit29 kernel: [<ffffffffa09ec111>] ? ptlrpc_main+0xbf1/0x19e0 [ptlrpc] Sep 28 11:43:08 hyperion-dit29 kernel: [<ffffffffa09eb520>] ? ptlrpc_main+0x0/0x19e0 [ptlrpc] Sep 28 11:43:08 hyperion-dit29 kernel: [<ffffffff8100c14a>] ? child_rip+0xa/0x20 Sep 28 11:43:08 hyperion-dit29 kernel: [<ffffffffa09eb520>] ? ptlrpc_main+0x0/0x19e0 [ptlrpc] Sep 28 11:43:08 hyperion-dit29 kernel: [<ffffffffa09eb520>] ? ptlrpc_main+0x0/0x19e0 [ptlrpc] Sep 28 11:43:08 hyperion-dit29 kernel: [<ffffffff8100c140>] ? child_rip+0x0/0x20 Sep 28 11:43:08 hyperion-dit29 kernel: Mem-Info: Sep 28 11:43:08 hyperion-dit29 kernel: Mem-Info: Sep 28 11:43:08 hyperion-dit29 kernel: Node 1 Normal per-cpu: Sep 28 11:43:08 hyperion-dit29 kernel: CPU 0: hi: 186, btch: 31 usd: 7 Sep 28 11:43:08 hyperion-dit29 kernel: CPU 1: hi: 186, btch: 31 usd: 18 Sep 28 11:43:08 hyperion-dit29 kernel: CPU 2: hi: 186, btch: 31 usd: 7 Sep 28 11:43:08 hyperion-dit29 kernel: CPU 3: hi: 186, btch: 31 usd: 29 Sep 28 11:43:08 hyperion-dit29 kernel: CPU 4: hi: 186, btch: 31 usd: 179 Sep 28 11:43:08 hyperion-dit29 kernel: CPU 5: hi: 186, btch: 31 usd: 178 Sep 28 11:43:08 hyperion-dit29 kernel: CPU 6: hi: 186, btch: 31 usd: 23 Sep 28 11:43:08 hyperion-dit29 kernel: CPU 7: hi: 186, btch: 31 usd: 91 Sep 28 11:43:08 hyperion-dit29 kernel: CPU 8: hi: 186, btch: 31 usd: 26 Sep 28 11:43:08 hyperion-dit29 kernel: CPU 9: hi: 186, btch: 31 usd: 11 Sep 28 11:43:08 hyperion-dit29 kernel: CPU 10: hi: 186, btch: 31 usd: 7 Sep 28 11:43:08 hyperion-dit29 kernel: CPU 11: hi: 186, btch: 31 usd: 30 Sep 28 11:43:08 hyperion-dit29 kernel: CPU 12: hi: 186, btch: 31 usd: 0 Sep 28 11:43:08 hyperion-dit29 kernel: CPU 13: hi: 186, btch: 31 usd: 2 Sep 28 11:43:08 hyperion-dit29 kernel: CPU 14: hi: 186, btch: 31 usd: 23 Sep 28 11:43:08 hyperion-dit29 kernel: CPU 15: hi: 186, btch: 31 usd: 11 Sep 28 11:43:08 hyperion-dit29 kernel: active_anon:4402 inactive_anon:7450 isolated_anon:0 Sep 28 11:43:08 hyperion-dit29 kernel: active_file:17703 inactive_file:27090 isolated_file:64 Sep 28 11:43:08 hyperion-dit29 kernel: unevictable:0 dirty:69 writeback:0 unstable:0 Sep 28 11:43:08 hyperion-dit29 kernel: free:38023 slab_reclaimable:5697 slab_unreclaimable:5547440 Sep 28 11:43:08 hyperion-dit29 kernel: mapped:490 shmem:8175 pagetables:439 bounce:0 Sep 28 11:43:08 hyperion-dit29 kernel: Node 1 Normal free:52084kB min:45096kB low:56368kB high:67644kB active_anon:4372kB inactive_anon:4064kB active_file:52884kB inactive_file:86880kB unevictable:0kB isolated(anon):0kB isolated(file):128kB present:12410880kB mlocked:0kB dirty:272kB writeback:0kB mapped:1952kB shmem:1192kB slab_reclaimable:10456kB slab_unreclaimable:11372900kB kernel_stack:3024kB pagetables:912kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:384 all_unreclaimable? no Sep 28 11:43:08 hyperion-dit29 kernel: lowmem_reserve[]: 0 0 0 0 Sep 28 11:43:08 hyperion-dit29 kernel: Node 1 Normal: 2366*4kB 1236*8kB 443*16kB 200*32kB 103*64kB 37*128kB 10*256kB 0*512kB 0*1024kB 1*2048kB 1*4096kB = 52872kB Sep 28 11:43:08 hyperion-dit29 kernel: 53304 total pagecache pages Sep 28 11:43:08 hyperion-dit29 kernel: 0 pages in swap cache Sep 28 11:43:08 hyperion-dit29 kernel: Swap cache stats: add 0, delete 0, find 0/0 Sep 28 11:43:08 hyperion-dit29 kernel: Free swap = 0kB Sep 28 11:43:08 hyperion-dit29 kernel: Total swap = 0kB Sep 28 11:43:08 hyperion-dit29 kernel: 6291440 pages RAM Sep 28 11:43:08 hyperion-dit29 kernel: 174434 pages reserved Sep 28 11:43:08 hyperion-dit29 kernel: 65273 pages shared Sep 28 11:43:08 hyperion-dit29 kernel: 5987243 pages non-shared ep 28 11:43:08 hyperion-dit29 kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name Sep 28 11:43:08 hyperion-dit29 kernel: [ 2308] 0 2308 2791 224 0 -17 -1000 udevd Sep 28 11:43:08 hyperion-dit29 kernel: [ 3471] 0 3471 6600 51 0 0 0 syslog-ng Sep 28 11:43:08 hyperion-dit29 kernel: [ 3472] 0 3472 14198 414 6 0 0 syslog-ng Sep 28 11:43:08 hyperion-dit29 kernel: [ 3473] 0 3473 2821 47 4 0 0 shutdown-hot Sep 28 11:43:08 hyperion-dit29 kernel: [ 3521] 0 3521 2284 121 3 0 0 irqbalance Sep 28 11:43:08 hyperion-dit29 kernel: [ 3535] 32 3535 4739 75 6 0 0 rpcbind Sep 28 11:43:08 hyperion-dit29 kernel: [ 3553] 29 3553 5832 120 0 0 0 rpc.statd Sep 28 11:43:08 hyperion-dit29 kernel: [ 3574] 0 3574 6842 70 10 0 0 rpc.idmapd Sep 28 11:43:08 hyperion-dit29 kernel: [ 3666] 81 3666 5342 105 10 0 0 dbus-daemon Sep 28 11:43:08 hyperion-dit29 kernel: [ 3745] 0 3745 55130 228 0 0 0 munged Sep 28 11:43:08 hyperion-dit29 kernel: [ 3757] 0 3757 29122 639 14 0 0 snmpd Sep 28 11:43:08 hyperion-dit29 kernel: [ 3768] 0 3768 16009 165 13 -17 -1000 sshd Sep 28 11:43:08 hyperion-dit29 kernel: [ 3778] 0 3778 5519 131 14 0 0 xinetd Sep 28 11:43:08 hyperion-dit29 kernel: [ 3797] 38 3797 6481 224 4 0 0 ntpd Sep 28 11:43:08 hyperion-dit29 kernel: [ 3849] 0 3849 5093 255 10 0 0 crond Sep 28 11:43:08 hyperion-dit29 kernel: [ 3858] 0 3858 3532 974 13 0 0 cerebrod Sep 28 11:43:08 hyperion-dit29 kernel: [ 3906] 0 3906 352127 342 12 0 0 opensm Sep 28 11:43:09 hyperion-dit29 kernel: [ 3936] 0 3936 12412 74 3 0 0 srp_daemon Sep 28 11:43:09 hyperion-dit29 kernel: [ 4185] 0 4185 1015 21 7 0 0 agetty Sep 28 11:43:09 hyperion-dit29 kernel: [ 4190] 0 4190 1012 20 9 0 0 mingetty Sep 28 11:43:09 hyperion-dit29 kernel: [ 4192] 0 4192 1012 20 11 0 0 mingetty Sep 28 11:43:09 hyperion-dit29 kernel: [ 4195] 0 4195 1012 20 13 0 0 mingetty Sep 28 11:43:09 hyperion-dit29 kernel: [ 4198] 0 4198 1012 20 14 0 0 mingetty Sep 28 11:43:09 hyperion-dit29 kernel: [ 4201] 0 4201 1012 20 14 0 0 mingetty Sep 28 11:43:09 hyperion-dit29 kernel: [ 4203] 0 4203 1012 20 13 0 0 mingetty Sep 28 11:43:09 hyperion-dit29 kernel: [ 4244] 0 4244 2790 222 8 -17 -1000 udevd Sep 28 11:43:09 hyperion-dit29 kernel: [ 4245] 0 4245 2790 222 1 -17 -1000 udevd Sep 28 11:43:09 hyperion-dit29 kernel: Out of memory: Kill process 3471 (syslog-ng) score 1 or sacrifice child Sep 28 11:43:09 hyperion-dit29 kernel: Killed process 3472, UID 0, (syslog-ng) total-vm:56792kB, anon-rss:888kB, file-rss:768kB