2011-06-23 16:42:22 LustreError: 20342:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-43) req@ffff810fe2d73800 x1372347906469188/t0 o36->0b43766f-9b44-a5b7-b3b7-3a19415c21bf@NET_0x50000c0a87259_UUID:0/0 lens 448/360 e 0 to 0 dl 1308872645 ref 1 fl Interpret:/0/0 rc 0/0 2011-06-23 16:42:22 LustreError: 20342:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 1621 previous similar messages 2011-06-23 16:47:30 LustreError: 20323:0:(ldlm_lockd.c:313:waiting_locks_callback()) ### lock callback timer expired after 101s: evicting client at 192.168.114.61@o2ib ns: mds-lustre-MDT0000_UUID lock: ffff813fd70ae600/0xb5e7eae8a288c677 lrc: 3/0,0 mode: CR/CR res: 133532367/3491738438 bits 0x1 rrc: 2 type: IBT flags: 0x4000020 remote: 0xabba4e35265cde60 expref: 70149 pid: 20349 timeout: 4298120101 2011-06-23 16:47:31 Lustre: 20184:0:(mds_reint.c:238:mds_finish_transno()) commit transaction for disconnected client 2929adef-b5c2-6328-599b-cb358950485c: rc -39 2011-06-23 16:47:31 LustreError: 20337:0:(handler.c:1512:mds_handle()) operation 36 on unconnected MDS from 12345-192.168.114.61@o2ib 2011-06-23 16:47:31 LustreError: 20337:0:(handler.c:1512:mds_handle()) Skipped 26046 previous similar messages 2011-06-23 16:47:42 LustreError: 20411:0:(mds_open.c:1693:mds_close()) @@@ no handle for file close ino 113446029: cookie 0xb5e7eae8a4c7b0fe req@ffff81302ddfd450 x1372347887830596/t0 o35->2929adef-b5c2-6328-599b-cb358950485c@NET_0x50000c0a8723d_UUID:0/0 lens 408/864 e 0 to 0 dl 1308872877 ref 1 fl Interpret:/0/0 rc 0/0 2011-06-23 16:47:42 LustreError: 20411:0:(mds_open.c:1693:mds_close()) Skipped 17660 previous similar messages 2011-06-23 16:48:52 LustreError: 20386:0:(mds_open.c:1693:mds_close()) @@@ no handle for file close ino 143299303: cookie 0xb5e7eae8a4f6b8b1 req@ffff812036f7e800 x1372347887853828/t0 o35->2929adef-b5c2-6328-599b-cb358950485c@NET_0x50000c0a8723d_UUID:0/0 lens 408/864 e 0 to 0 dl 1308872947 ref 1 fl Interpret:/0/0 rc 0/0 2011-06-23 16:48:52 LustreError: 20386:0:(mds_open.c:1693:mds_close()) Skipped 2 previous similar messages 2011-06-23 16:49:31 LustreError: 20364:0:(mds_open.c:1693:mds_close()) @@@ no handle for file close ino 143199242: cookie 0xb5e7eae8a5814708 req@ffff811027b91800 x1372347887857767/t0 o35->2929adef-b5c2-6328-599b-cb358950485c@NET_0x50000c0a8723d_UUID:0/0 lens 408/864 e 0 to 0 dl 1308872986 ref 1 fl Interpret:/0/0 rc 0/0 2011-06-23 16:49:31 LustreError: 20364:0:(mds_open.c:1693:mds_close()) Skipped 242 previous similar messages 2011-06-23 16:51:11 md: rebuild md0 throttled due to IO 2011-06-23 16:51:34 LustreError: 20311:0:(ldlm_lockd.c:313:waiting_locks_callback()) ### lock callback timer expired after 154s: evicting client at 192.168.114.61@o2ib ns: mds-lustre-MDT0000_UUID lock: ffff811f21089a00/0xb5e7eae8af884a5b lrc: 3/0,0 mode: CR/CR res: 143885484/3492121254 bits 0x1 rrc: 2 type: IBT flags: 0x4000020 remote: 0xabba4e3526763f16 expref: 11134 pid: 20312 timeout: 4298364513 2011-06-23 16:51:34 Lustre: 13043:0:(mds_reint.c:238:mds_finish_transno()) commit transaction for disconnected client 2929adef-b5c2-6328-599b-cb358950485c: rc -39 2011-06-23 16:51:34 LustreError: 20322:0:(handler.c:1512:mds_handle()) operation 101 on unconnected MDS from 12345-192.168.114.61@o2ib 2011-06-23 16:51:38 LustreError: 20403:0:(mds_open.c:1693:mds_close()) @@@ no handle for file close ino 125078324: cookie 0xb5e7eae8af8a215a req@ffff813f3796f400 x1372347887859757/t0 o35->2929adef-b5c2-6328-599b-cb358950485c@NET_0x50000c0a8723d_UUID:0/0 lens 408/864 e 0 to 0 dl 1308873113 ref 1 fl Interpret:/0/0 rc 0/0 2011-06-23 16:51:38 LustreError: 20403:0:(mds_open.c:1693:mds_close()) Skipped 9 previous similar messages 2011-06-23 16:51:50 LustreError: 13048:0:(ldlm_lockd.c:313:waiting_locks_callback()) ### lock callback timer expired after 153s: evicting client at 192.168.114.77@o2ib ns: mds-lustre-MDT0000_UUID lock: ffff813ffc2bfc00/0xb5e7eae89e922f23 lrc: 3/0,0 mode: CR/CR res: 141526183/3491241594 bits 0x1 rrc: 2 type: IBT flags: 0x4000020 remote: 0x190522b4cedc66f5 expref: 82859 pid: 20348 timeout: 4298380583 2011-06-23 16:51:51 Lustre: 20307:0:(mds_reint.c:238:mds_finish_transno()) commit transaction for disconnected client 2523a362-52a0-2ea2-a239-33b2b2f768ec: rc -39 2011-06-23 16:52:33 LustreError: 20197:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-116) req@ffff813f018d0000 x1372347898325499/t0 o35->2523a362-52a0-2ea2-a239-33b2b2f768ec@NET_0x50000c0a8724d_UUID:0/0 lens 408/624 e 0 to 0 dl 1308873168 ref 1 fl Interpret:/0/0 rc -116/0 2011-06-23 16:52:33 LustreError: 20197:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 1029 previous similar messages 2011-06-23 16:54:13 LustreError: 20382:0:(mds_open.c:1693:mds_close()) @@@ no handle for file close ino 143628354: cookie 0xb5e7eae8aa34c4af req@ffff81303174e850 x1372347887860164/t0 o35->2929adef-b5c2-6328-599b-cb358950485c@NET_0x50000c0a8723d_UUID:0/0 lens 408/864 e 0 to 0 dl 1308873268 ref 1 fl Interpret:/0/0 rc 0/0 2011-06-23 16:54:13 LustreError: 20382:0:(mds_open.c:1693:mds_close()) Skipped 17 previous similar messages 2011-06-23 16:55:31 LustreError: 20183:0:(ldlm_lockd.c:313:waiting_locks_callback()) ### lock callback timer expired after 232s: evicting client at 192.168.114.61@o2ib ns: mds-lustre-MDT0000_UUID lock: ffff813f04b81200/0xb5e7eae8b1d948d0 lrc: 3/0,0 mode: CR/CR res: 143294901/3491869316 bits 0x3 rrc: 2 type: IBT flags: 0x4000020 remote: 0xabba4e352676601e expref: 670 pid: 15387 timeout: 4298601111 2011-06-23 16:55:31 Lustre: 13030:0:(mds_reint.c:238:mds_finish_transno()) commit transaction for disconnected client 2929adef-b5c2-6328-599b-cb358950485c: rc -39 2011-06-23 16:55:31 LustreError: 20185:0:(handler.c:1512:mds_handle()) operation 101 on unconnected MDS from 12345-192.168.114.61@o2ib 2011-06-23 16:55:31 LustreError: 20185:0:(handler.c:1512:mds_handle()) Skipped 3 previous similar messages 2011-06-23 16:58:36 LustreError: 20355:0:(mds_open.c:1693:mds_close()) @@@ no handle for file close ino 143200392: cookie 0xb5e7eae8af837990 req@ffff813f74b86800 x1372347898414147/t0 o35->2523a362-52a0-2ea2-a239-33b2b2f768ec@NET_0x50000c0a8724d_UUID:0/0 lens 408/864 e 0 to 0 dl 1308873566 ref 1 fl Interpret:/0/0 rc 0/0 2011-06-23 16:58:36 LustreError: 20355:0:(mds_open.c:1693:mds_close()) Skipped 1862 previous similar messages Console [hyperion-mds1] log at 2011-06-23 17:00:00 PDT. 2011-06-23 17:01:19 md: rebuild md0 throttled due to IO 2011-06-23 17:04:19 LustreError: 20277:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-116) req@ffff810fc553ac00 x1372347898533064/t0 o35->2523a362-52a0-2ea2-a239-33b2b2f768ec@NET_0x50000c0a8724d_UUID:0/0 lens 408/624 e 0 to 0 dl 1308873872 ref 1 fl Interpret:/0/0 rc -116/0 2011-06-23 17:04:19 LustreError: 20277:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 2248 previous similar messages 2011-06-23 17:07:47 LustreError: 20374:0:(mds_open.c:1693:mds_close()) @@@ no handle for file close ino 95752047: cookie 0xb5e7eae8aca0f568 req@ffff812fa5924400 x1372347888258337/t0 o35->2929adef-b5c2-6328-599b-cb358950485c@NET_0x50000c0a8723d_UUID:0/0 lens 408/864 e 0 to 0 dl 1308874082 ref 1 fl Interpret:/0/0 rc 0/0 2011-06-23 17:07:47 LustreError: 20374:0:(mds_open.c:1693:mds_close()) Skipped 364 previous similar messages 2011-06-23 17:11:20 md: rebuild md0 throttled due to IO 2011-06-23 17:14:36 LustreError: 20277:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-116) req@ffff812f91def800 x1372347898752728/t0 o35->2523a362-52a0-2ea2-a239-33b2b2f768ec@NET_0x50000c0a8724d_UUID:0/0 lens 408/624 e 0 to 0 dl 1308874492 ref 1 fl Interpret:/0/0 rc -116/0 2011-06-23 17:14:37 LustreError: 20277:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 686 previous similar messages 2011-06-23 17:18:11 list_del corruption. next->prev should be ffff813e6d1a3f40, but was 0000000000000000 2011-06-23 17:18:11 ----------- [cut here ] --------- [please bite here ] --------- 2011-06-23 17:18:11 Kernel BUG at lib/list_debug.c:70 2011-06-23 17:18:11 invalid opcode: 0000 [1] SMP 2011-06-23 17:18:11 last sysfs file: /devices/pci0000:00/0000:00:0d.0/0000:05:00.0/0000:06:03.1/irq 2011-06-23 17:18:11 CPU 11 2011-06-23 17:18:11 Modules linked in: mds(U) fsfilt_ldiskfs(U) mgs(U) mgc(U) ldiskfs(U) jbd2(U) crc16(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) ib_srp(U) ib_ipoib(U) rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ucm(U) ib_uverbs(U) ib_umad(U) mlx4_ib(U) mlx4_core(U) ipoib_helper(U) ib_cm(U) ib_sa(U) ib_mad(U) ib_core(U) ipv6(U) xfrm_nalgo(U) crypto_api(U) dm_mirror(U) dm_log(U) dm_multipath(U) scsi_dh(U) dm_mod(U) raid10(U) video(U) backlight(U) sbs(U) power_meter(U) i2c_ec(U) dell_wmi(U) wmi(U) button(U) battery(U) asus_acpi(U) acpi_memhotplug(U) ac(U) parport_pc(U) lp(U) parport(U) sd_mod(U) sg(U) floppy(U) mptsas(U) i2c_nforce2(U) mptscsih(U) ohci_hcd(U) i2c_core(U) sata_nv(U) pcspkr(U) mptbase(U) libata(U) amd64_edac_mod(U) k10temp(U) edac_mc(U) hwmon(U) scsi_transport_sas(U) tpm_tis(U) shpchp(U) scsi_mod(U) tpm(U) ide_cd(U) tpm_bios(U) serio_raw(U) cdrom(U) nfs(U) nfs_acl(U) lockd(U) fscache(U) sunrpc(U) e1000(U) 2011-06-23 17:18:11 Pid: 12736, comm: kiblnd_sd_04 Tainted: G 2.6.18-238.12.1.el5_lustre.g266a955 #1 2011-06-23 17:18:11 RIP: 0010:[] [] list_del+0x48/0x71 2011-06-23 17:18:11 RSP: 0018:ffff812035d65ac0 EFLAGS: 00010286 2011-06-23 17:18:11 RAX: 0000000000000058 RBX: ffff813e6d1a3f40 RCX: 00000000000003e8 2011-06-23 17:18:11 RDX: ffff81107023c848 RSI: 0000000000000000 RDI: ffffffff8031af20 2011-06-23 17:18:11 RBP: ffff813e6d19d440 R08: ffffffff8031af28 R09: 0000000000000001 2011-06-23 17:18:11 R10: 0000000000000080 R11: 0000000000000000 R12: 0000000000000278 2011-06-23 17:18:11 R13: 00050000c0a87253 R14: 0000000000000001 R15: 0000000000003039 2011-06-23 17:18:11 FS: 00002aaaab01f6e0(0000) GS:ffff8120381ab2c0(0000) knlGS:0000000000000000 2011-06-23 17:18:11 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b 2011-06-23 17:18:11 CR2: 00002aaaaad626e0 CR3: 0000000000201000 CR4: 00000000000006e0 2011-06-23 17:18:11 Process kiblnd_sd_04 (pid: 12736, threadinfo ffff812035d64000, task ffff81102976d7a0) 2011-06-23 17:18:11 Stack: ffff813e6d1a3f40 ffffffff885e952d 0000000000000000 0000000000000002 2011-06-23 17:18:11 0000000000000000 ffffffff8879571f ffff813036e5dac0 0000000000000286 2011-06-23 17:18:11 ffff813e6d19d440 ffffffff885ebbba ffff813033e04a00 0000000000000000 2011-06-23 17:18:11 Call Trace: 2011-06-23 17:18:11 [] :lnet:lnet_me_unlink+0xd/0x120 2011-06-23 17:18:11 [] :ko2iblnd:kiblnd_post_rx+0x2bf/0x2f0 2011-06-23 17:18:11 [] :lnet:lnet_md_unlink+0x3a/0x290 2011-06-23 17:18:11 [] :lnet:lnet_try_match_md+0x378/0x3e0 2011-06-23 17:18:11 [] :lnet:lnet_match_md+0x27c/0x310 2011-06-23 17:18:11 [] :lnet:lnet_parse+0xf57/0x1ae0 2011-06-23 17:18:11 [] :lnet:lnet_complete_msg_locked+0x1bc/0x300 2011-06-23 17:18:11 [] task_rq_lock+0x3d/0x6f 2011-06-23 17:18:11 [] :ko2iblnd:kiblnd_check_sends+0xa6/0x560 2011-06-23 17:18:11 [] :ko2iblnd:kiblnd_unpack_msg+0x365/0x670 2011-06-23 17:18:12 [] :ko2iblnd:kiblnd_handle_rx+0x4db/0x570 2011-06-23 17:18:12 [] :ko2iblnd:kiblnd_rx_complete+0x263/0x3a0 2011-06-23 17:18:12 [] __wake_up+0x38/0x4f 2011-06-23 17:18:12 [] :ko2iblnd:kiblnd_scheduler+0x4d9/0x670 2011-06-23 17:18:12 [] default_wake_function+0x0/0xe 2011-06-23 17:18:12 [] keventd_create_kthread+0x0/0xc4 2011-06-23 17:18:12 [] child_rip+0xa/0x11 2011-06-23 17:18:12 [] :ko2iblnd:kiblnd_scheduler+0x0/0x670 2011-06-23 17:18:12 [] child_rip+0x0/0x11 2011-06-23 17:18:12 2011-06-23 17:18:12 2011-06-23 17:18:12 Code: 0f 0b 68 19 41 2c 80 c2 46 00 48 8b 13 48 8b 43 08 48 89 42 2011-06-23 17:18:12 RIP [] list_del+0x48/0x71 2011-06-23 17:18:12 RSP 2011-06-23 17:18:12 REWRITING MCP55 CFG REG 2011-06-23 17:18:12 CFG = c1 2011-06-23 17:18:12 Linux version 2.6.18-238.12.1.el5_lustre.g266a955 (jenkins@rhel5-64-build.lab.whamcloud.com) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-50)) #1 SMP Fri Jun 10 16:39:27 PDT 2011 2011-06-23 17:18:12 Command line: initrd=initrd-2.6.18-238.12.1.el5_lustre.g266a955 console=ttyS0,115200n8 elevator=deadline swiotlb=65536 selinux=0 BOOT_IMAGE=vmlinuz-2.6.18-238.12.1.el5_lustre.g266a955 BOOTIF=01-00-30-48-5f-28-26 irqpoll maxcpus=1 reset_devices hdb=cdrom memmap=exactmap memmap=640K@0K memmap=6080K@16384K memmap=124336K@23104K elfcorehdr=147440K memmap=56K#3276736K memmap=8K#3276792K memmap=4K$4173824K memmap=1024K$4175872K memmap=9216K$4185088K 2011-06-23 17:18:12 BIOS-provided physical RAM map: 2011-06-23 17:18:12 BIOS-e820: 0000000000010000 - 000000000009b000 (usable) 2011-06-23 17:18:12 BIOS-e820: 000000000009b000 - 00000000000a0000 (reserved) 2011-06-23 17:18:12 BIOS-e820: 0000000000100000 - 00000000c7ff0000 (usable) 2011-06-23 17:18:12 BIOS-e820: 00000000c7ff0000 - 00000000c7ffe000 (ACPI data) 2011-06-23 17:18:12 BIOS-e820: 00000000c7ffe000 - 00000000c8000000 (ACPI NVS) 2011-06-23 17:18:12 BIOS-e820: 00000000fec00000 - 00000000fec01000 (reserved) 2011-06-23 17:18:12 BIOS-e820: 00000000fee00000 - 00000000fef00000 (reserved) 2011-06-23 17:18:12 BIOS-e820: 00000000ff700000 - 0000000100000000 (reserved) 2011-06-23 17:18:12 BIOS-e820: 0000000100000000 - 0000004038000000 (usable) 2011-06-23 17:18:12 user-defined physical RAM map: 2011-06-23 17:18:12 user: 0000000000000000 - 00000000000a0000 (usable) 2011-06-23 17:18:12 user: 0000000001000000 - 00000000015f0000 (usable) 2011-06-23 17:18:12 user: 0000000001690000 - 0000000008ffc000 (usable) 2011-06-23 17:18:12 user: 00000000c7ff0000 - 00000000c8000000 (ACPI data) 2011-06-23 17:18:12 user: 00000000fec00000 - 00000000fec01000 (reserved) 2011-06-23 17:18:12 user: 00000000fee00000 - 00000000fef00000 (reserved) 2011-06-23 17:18:12 user: 00000000ff700000 - 0000000100000000 (reserved) 2011-06-23 17:18:12 DMI present. 2011-06-23 17:18:12 SRAT: PXM 0 -> APIC 0 -> Node 0 2011-06-23 17:18:12 SRAT: PXM 0 -> APIC 1 -> Node 0 2011-06-23 17:18:12 SRAT: PXM 0 -> APIC 2 -> Node 0 2011-06-23 17:18:12 SRAT: PXM 0 -> APIC 3 -> Node 0 2011-06-23 17:18:12 SRAT: PXM 1 -> APIC 4 -> Node 1 2011-06-23 17:18:12 SRAT: PXM 1 -> APIC 5 -> Node 1 2011-06-23 17:18:12 SRAT: PXM 1 -> APIC 6 -> Node 1 2011-06-23 17:18:12 SRAT: PXM 1 -> APIC 7 -> Node 1 2011-06-23 17:18:12 SRAT: PXM 2 -> APIC 8 -> Node 2 2011-06-23 17:18:12 SRAT: PXM 2 -> APIC 9 -> Node 2 2011-06-23 17:18:12 SRAT: PXM 2 -> APIC 10 -> Node 2 2011-06-23 17:18:12 SRAT: PXM 2 -> APIC 11 -> Node 2 2011-06-23 17:18:12 SRAT: PXM 3 -> APIC 12 -> Node 3 2011-06-23 17:18:12 SRAT: PXM 3 -> APIC 13 -> Node 3 2011-06-23 17:18:12 SRAT: PXM 3 -> APIC 14 -> Node 3 2011-06-23 17:18:12 SRAT: PXM 3 -> APIC 15 -> Node 3 2011-06-23 17:18:12 SRAT: Node 0 PXM 0 0-a0000 2011-06-23 17:18:12 SRAT: Node 0 PXM 0 0-c8000000 2011-06-23 17:18:12 SRAT: Node 0 PXM 0 0-1038000000 2011-06-23 17:18:12 SRAT: Node 1 PXM 1 1038000000-2038000000 2011-06-23 17:18:12 SRAT: Node 2 PXM 2 2038000000-3038000000 2011-06-23 17:18:12 SRAT: Node 3 PXM 3 3038000000-4038000000 2011-06-23 17:18:12 Bootmem setup node 0 0000000000000000-0000000008ffc000 2011-06-23 17:18:12 Memory for crash kernel (0x0 to 0x0) notwithin permissible range 2011-06-23 17:18:12 disabling kdump 2011-06-23 17:18:12 ACPI: PM-Timer IO Port: 0x2008 2011-06-23 17:18:12 ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled) 2011-06-23 17:18:12 Processor #0 0:2 APIC version 16 2011-06-23 17:18:12 ACPI: LAPIC (acpi_id[0x02] lapic_id[0x01] enabled) 2011-06-23 17:18:12 Processor #1 0:2 APIC version 16 2011-06-23 17:18:12 ACPI: LAPIC (acpi_id[0x03] lapic_id[0x02] enabled) 2011-06-23 17:18:12 Processor #2 0:2 APIC version 16 2011-06-23 17:18:12 ACPI: LAPIC (acpi_id[0x04] lapic_id[0x03] enabled) 2011-06-23 17:18:12 Processor #3 0:2 APIC version 16 2011-06-23 17:18:12 ACPI: LAPIC (acpi_id[0x05] lapic_id[0x04] enabled) 2011-06-23 17:18:12 Processor #4 0:2 APIC version 16 2011-06-23 17:18:12 ACPI: LAPIC (acpi_id[0x06] lapic_id[0x05] enabled) 2011-06-23 17:18:12 Processor #5 0:2 APIC version 16 2011-06-23 17:18:12 ACPI: LAPIC (acpi_id[0x07] lapic_id[0x06] enabled) 2011-06-23 17:18:12 Processor #6 0:2 APIC version 16 2011-06-23 17:18:12 ACPI: LAPIC (acpi_id[0x08] lapic_id[0x07] enabled) 2011-06-23 17:18:12 Processor #7 0:2 APIC version 16 2011-06-23 17:18:12 ACPI: LAPIC (acpi_id[0x09] lapic_id[0x08] enabled) 2011-06-23 17:18:12 Processor #8 0:2 APIC version 16 2011-06-23 17:18:12 ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x09] enabled) 2011-06-23 17:18:12 Processor #9 0:2 APIC version 16 2011-06-23 17:18:12 ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x0a] enabled) 2011-06-23 17:18:12 Processor #10 0:2 APIC version 16 2011-06-23 17:18:12 ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x0b] enabled) 2011-06-23 17:18:12 Processor #11 0:2 APIC version 16 2011-06-23 17:18:12 ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x0c] enabled) 2011-06-23 17:18:12 Processor #12 0:2 APIC version 16 2011-06-23 17:18:12 ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x0d] enabled) 2011-06-23 17:18:12 Processor #13 0:2 APIC version 16 2011-06-23 17:18:12 ACPI: LAPIC (acpi_id[0x0f] lapic_id[0x0e] enabled) 2011-06-23 17:18:12 Processor #14 0:2 APIC version 16 2011-06-23 17:18:12 ACPI: LAPIC (acpi_id[0x10] lapic_id[0x0f] enabled) 2011-06-23 17:18:12 Processor #15 0:2 APIC version 16 2011-06-23 17:18:12 ACPI: IOAPIC (id[0x10] address[0xfec00000] gsi_base[0]) 2011-06-23 17:18:12 IOAPIC[0]: apic_id 16, version 17, address 0xfec00000, GSI 0-23 2011-06-23 17:18:12 ACPI: IOAPIC (id[0x11] address[0xcfdff000] gsi_base[24]) 2011-06-23 17:18:12 IOAPIC[1]: apic_id 17, version 17, address 0xcfdff000, GSI 24-47 2011-06-23 17:18:12 ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) 2011-06-23 17:18:12 ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) 2011-06-23 17:18:12 ACPI: INT_SRC_OVR (bus 0 bus_irq 14 global_irq 14 high edge) 2011-06-23 17:18:12 ACPI: INT_SRC_OVR (bus 0 bus_irq 15 global_irq 15 high edge) 2011-06-23 17:18:12 Setting APIC routing to physical flat 2011-06-23 17:18:12 ACPI: HPET id: 0x10de8201 base: 0xfed00000 2011-06-23 17:18:12 Using ACPI (MADT) for SMP configuration information 2011-06-23 17:18:12 Nosave address range: 00000000000a0000 - 0000000001000000 2011-06-23 17:18:12 Nosave address range: 00000000015f0000 - 0000000001690000 2011-06-23 17:18:12 Allocating PCI resources starting at 10000000 (gap: 8ffc000:beff4000) 2011-06-23 17:18:12 SMP: Allowing 16 CPUs, 0 hotplug CPUs 2011-06-23 17:18:12 Built 1 zonelists. Total pages: 32251 2011-06-23 17:18:12 Kernel command line: initrd=initrd-2.6.18-238.12.1.el5_lustre.g266a955 console=ttyS0,115200n8 elevator=deadline swiotlb=65536 selinux=0 BOOT_IMAGE=vmlinuz-2.6.18-238.12.1.el5_lustre.g266a955 BOOTIF=01-00-30-48-5f-28-26 irqpoll maxcpus=1 reset_devices hdb=cdrom memmap=exactmap memmap=640K@0K memmap=6080K@16384K memmap=124336K@23104K elfcorehdr=147440K memmap=56K#3276736K memmap=8K#3276792K memmap=4K$4173824K memmap=1024K$4175872K memmap=9216K$4185088K 2011-06-23 17:18:12 Misrouted IRQ fixup and polling support enabled 2011-06-23 17:18:12 This may significantly impact system performance 2011-06-23 17:18:12 ide_setup: hdb=cdrom 2011-06-23 17:18:12 Initializing CPU#0 2011-06-23 17:18:12 PID hash table entries: 512 (order: 9, 4096 bytes) 2011-06-23 17:18:12 irq 114, desc: ffffffff80445580, depth: 1, count: 0, unhandled: 0 2011-06-23 17:18:12 ->handle_irq(): ffffffff800bd021, handle_bad_irq+0x0/0x1f6 2011-06-23 17:18:12 ->chip(): ffffffff80320860, no_irq_chip+0x0/0x80 2011-06-23 17:18:12 ->action(): 0000000000000000 2011-06-23 17:18:12 IRQ_DISABLED set 2011-06-23 17:18:12 unexpected IRQ trap at vector 72 2011-06-23 17:18:12 Console: colour VGA+ 80x25 2011-06-23 17:18:12 Dentry cache hash table entries: 16384 (order: 5, 131072 bytes) 2011-06-23 17:18:12 Inode-cache hash table entries: 8192 (order: 4, 65536 bytes) 2011-06-23 17:18:12 Checking aperture... 2011-06-23 17:18:12 CPU 0: aperture @ 24000000 size 64 MB 2011-06-23 17:18:12 CPU 1: aperture @ 24000000 size 64 MB 2011-06-23 17:18:12 CPU 2: aperture @ 24000000 size 64 MB 2011-06-23 17:18:12 CPU 3: aperture @ 24000000 size 64 MB 2011-06-23 17:18:12 ACPI: DMAR not present 2011-06-23 17:18:12 Memory: 116700k/147440k available (2594k kernel code, 14356k reserved, 1652k data, 224k init) 2011-06-23 17:18:12 Calibrating delay loop (skipped), value calculated using timer frequency.. 4600.18 BogoMIPS (lpj=2300094) 2011-06-23 17:18:12 Security Framework v1.0.0 initialized 2011-06-23 17:18:12 SELinux: Disabled at boot. 2011-06-23 17:18:12 Capability LSM initialized 2011-06-23 17:18:12 Mount-cache hash table entries: 256 2011-06-23 17:18:12 CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line) 2011-06-23 17:18:12 CPU: L2 Cache: 512K (64 bytes/line) 2011-06-23 17:18:12 CPU 0/b -> Node 0 2011-06-23 17:18:12 CPU: Physical Processor ID: 2 2011-06-23 17:18:12 CPU: Processor Core ID: 3 2011-06-23 17:18:12 SMP alternatives: switching to UP code 2011-06-23 17:18:12 ACPI: Core revision 20060707 2011-06-23 17:18:12 ..MP-BIOS bug: 8254 timer not connected to IO-APIC 2011-06-23 17:18:12 Using local APIC timer interrupts. 2011-06-23 17:18:13 Detected 12.500 MHz APIC timer. 2011-06-23 17:18:13 Brought up 1 CPUs 2011-06-23 17:18:13 NMI watchdog testing PASSED. 2011-06-23 17:18:13 time.c: Using 25.000000 MHz WALL HPET GTOD HPET/TSC timer. 2011-06-23 17:18:13 time.c: Detected 2300.099 MHz processor. 2011-06-23 17:18:13 checking if image is initramfs... it is 2011-06-23 17:18:13 Freeing initrd memory: 3777k freed 2011-06-23 17:18:13 NET: Registered protocol family 16 2011-06-23 17:18:13 ACPI: bus type pci registered 2011-06-23 17:18:13 PCI: BIOS Bug: MCFG area at e0000000 is not E820-reserved 2011-06-23 17:18:13 PCI: Not using MMCONFIG. 2011-06-23 17:18:13 PCI: Using configuration type 1 2011-06-23 17:18:13 PCI: Using configuration type 1 for extended access 2011-06-23 17:18:13 ACPI: Interpreter enabled 2011-06-23 17:18:13 ACPI: Using IOAPIC for interrupt routing 2011-06-23 17:18:13 ACPI: No dock devices found. 2011-06-23 17:18:13 ACPI: PCI Root Bridge [PCIB] (0000:80) 2011-06-23 17:18:13 ACPI: PCI Root Bridge [PCI0] (0000:00) 2011-06-23 17:18:13 FOUND MCP55 CHIP 2011-06-23 17:18:13 cfg value is c1 2011-06-23 17:18:13 PCI: Transparent bridge - 0000:00:06.0 2011-06-23 17:18:13 ACPI: PCI Interrupt Link [LNKA] (IRQs 16 17 18 19) *0, disabled. 2011-06-23 17:18:13 ACPI: PCI Interrupt Link [LNKB] (IRQs 16 17 18 19) *0, disabled. 2011-06-23 17:18:13 ACPI: PCI Interrupt Link [LNKC] (IRQs 16 17 18 19) *0, disabled. 2011-06-23 17:18:13 ACPI: PCI Interrupt Link [LNKD] (IRQs 16 17 18 19) *0, disabled. 2011-06-23 17:18:13 ACPI: PCI Interrupt Link [LNEA] (IRQs 16 17 *18 19) 2011-06-23 17:18:13 ACPI: PCI Interrupt Link [LNEB] (IRQs 16 *17 18 19) 2011-06-23 17:18:13 ACPI: PCI Interrupt Link [LNEC] (IRQs 16 17 18 19) *0, disabled. 2011-06-23 17:18:13 ACPI: PCI Interrupt Link [LNED] (IRQs 16 17 18 *19) 2011-06-23 17:18:13 ACPI: PCI Interrupt Link [LUB0] (IRQs *20 21 22 23) 2011-06-23 17:18:13 ACPI: PCI Interrupt Link [LMAD] (IRQs 20 21 22 23) *0, disabled. 2011-06-23 17:18:13 ACPI: PCI Interrupt Link [LUB2] (IRQs 20 21 22 23) *0, disabled. 2011-06-23 17:18:13 ACPI: PCI Interrupt Link [LMAC] (IRQs 20 21 22 23) *0, disabled. 2011-06-23 17:18:13 ACPI: PCI Interrupt Link [LAZA] (IRQs 20 21 22 23) *0, disabled. 2011-06-23 17:18:13 ACPI: PCI Interrupt Link [LSMB] (IRQs 20 21 22 23) *0, disabled. 2011-06-23 17:18:13 ACPI: PCI Interrupt Link [LPMU] (IRQs 20 21 22 23) *0, disabled. 2011-06-23 17:18:13 ACPI: PCI Interrupt Link [LSA0] (IRQs 20 21 22 *23) 2011-06-23 17:18:13 ACPI: PCI Interrupt Link [LSA1] (IRQs 20 21 *22 23) 2011-06-23 17:18:13 ACPI: PCI Interrupt Link [LATA] (IRQs 20 21 22 23) *0, disabled. 2011-06-23 17:18:13 ACPI: PCI Interrupt Link [LSA2] (IRQs 20 *21 22 23) 2011-06-23 17:18:13 ACPI: PCI Interrupt Link [LN2A] (IRQs 40 41 42 43) *0, disabled. 2011-06-23 17:18:13 ACPI: PCI Interrupt Link [LN2B] (IRQs 40 41 42 43) *0, disabled. 2011-06-23 17:18:13 ACPI: PCI Interrupt Link [LN2C] (IRQs 40 41 42 43) *0, disabled. 2011-06-23 17:18:13 ACPI: PCI Interrupt Link [LN2D] (IRQs 40 41 42 43) *0, disabled. 2011-06-23 17:18:13 ACPI: PCI Interrupt Link [LE3A] (IRQs 40 41 42 43) *0, disabled. 2011-06-23 17:18:13 ACPI: PCI Interrupt Link [LE3B] (IRQs 40 41 42 43) *0, disabled. 2011-06-23 17:18:13 ACPI: PCI Interrupt Link [LE3C] (IRQs 40 41 42 *43) 2011-06-23 17:18:13 ACPI: PCI Interrupt Link [LE3D] (IRQs 40 41 42 43) *0, disabled. 2011-06-23 17:18:13 ACPI: PCI Interrupt Link [IIM0] (IRQs 44 45 46 47) *0, disabled. 2011-06-23 17:18:13 ACPI: PCI Interrupt Link [IIM1] (IRQs 44 45 46 47) *0, disabled. 2011-06-23 17:18:13 ACPI: PCI Interrupt Link [ISI0] (IRQs 44 45 46 47) *0, disabled. 2011-06-23 17:18:13 ACPI: PCI Interrupt Link [ISI1] (IRQs 44 45 46 47) *0, disabled. 2011-06-23 17:18:13 ACPI: PCI Interrupt Link [ISI2] (IRQs 44 45 46 47) *0, disabled. 2011-06-23 17:18:13 ACPI: PCI Interrupt Link [ILSM] (IRQs 44 45 46 47) *0, disabled. 2011-06-23 17:18:13 ACPI: PCI Interrupt Link [ILPM] (IRQs 44 45 46 47) *0, disabled. 2011-06-23 17:18:13 Linux Plug and Play Support v0.97 (c) Adam Belay 2011-06-23 17:18:13 pnp: PnP ACPI init 2011-06-23 17:18:13 pnp: PnP ACPI: found 16 devices 2011-06-23 17:18:13 usbcore: registered new driver usbfs 2011-06-23 17:18:13 usbcore: registered new driver hub 2011-06-23 17:18:13 PCI: Using ACPI for IRQ routing 2011-06-23 17:18:13 PCI: If a device doesn't work, try "pci=routeirq". If it helps, post a report 2011-06-23 17:18:13 NetLabel: Initializing 2011-06-23 17:18:13 NetLabel: domain hash size = 128 2011-06-23 17:18:13 NetLabel: protocols = UNLABELED CIPSOv4 2011-06-23 17:18:13 NetLabel: unlabeled traffic allowed by default 2011-06-23 17:18:13 hpet0: at MMIO 0xfed00000 (virtual 0xffffffffff5fe000), IRQs 2, 8, 31 2011-06-23 17:18:13 hpet0: 3 32-bit timers, 25000000 Hz 2011-06-23 17:18:13 ACPI: DMAR not present 2011-06-23 17:18:13 PCI-DMA: Disabling IOMMU. 2011-06-23 17:18:13 pnp: 00:09: iomem range 0xdf6c0000-0xdf6fffff has been reserved 2011-06-23 17:18:13 pnp: 00:09: iomem range 0xfee01000-0xfeefffff could not be reserved 2011-06-23 17:18:13 pnp: 00:0a: ioport range 0x3000-0x307f has been reserved 2011-06-23 17:18:13 pnp: 00:0a: ioport range 0x3080-0x30ff has been reserved 2011-06-23 17:18:13 pnp: 00:0a: ioport range 0x3400-0x347f has been reserved 2011-06-23 17:18:13 pnp: 00:0a: ioport range 0x3480-0x34ff has been reserved 2011-06-23 17:18:13 pnp: 00:0a: ioport range 0x3800-0x387f has been reserved 2011-06-23 17:18:13 pnp: 00:0a: ioport range 0x3880-0x38ff has been reserved 2011-06-23 17:18:13 pnp: 00:0a: iomem range 0xfee01000-0xfef00fff could not be reserved 2011-06-23 17:18:13 pnp: 00:0c: iomem range 0xfec00000-0xfec00fff could not be reserved 2011-06-23 17:18:13 pnp: 00:0c: iomem range 0xfee00000-0xfee00fff could not be reserved 2011-06-23 17:18:13 pnp: 00:0d: ioport range 0x680-0x6ff has been reserved 2011-06-23 17:18:13 pnp: 00:0d: ioport range 0x295-0x296 has been reserved 2011-06-23 17:18:13 pnp: 00:0e: iomem range 0xe0000000-0xefffffff has been reserved 2011-06-23 17:18:13 pnp: 00:0f: iomem range 0x0-0x9ffff could not be reserved 2011-06-23 17:18:13 pnp: 00:0f: iomem range 0xc0000-0xcffff could not be reserved 2011-06-23 17:18:13 pnp: 00:0f: iomem range 0xe0000-0xfffff could not be reserved 2011-06-23 17:18:13 pnp: 00:0f: iomem range 0x100000-0xcfffffff could not be reserved 2011-06-23 17:18:13 PCI: Bridge: 0000:80:0a.0 2011-06-23 17:18:13 IO window: disabled. 2011-06-23 17:18:13 MEM window: cfe00000-cfefffff 2011-06-23 17:18:13 PREFETCH window 0x00000000ce000000-0x00000000ce7fffff 2011-06-23 17:18:13 PCI: Bridge: 0000:80:0b.0 2011-06-23 17:18:13 IO window: disabled. 2011-06-23 17:18:13 MEM window: disabled. 2011-06-23 17:18:13 PREFETCH window: disabled. 2011-06-23 17:18:13 PCI: Bridge: 0000:80:0c.0 2011-06-23 17:18:13 IO window: disabled. 2011-06-23 17:18:13 MEM window: disabled. 2011-06-23 17:18:13 PREFETCH window: disabled. 2011-06-23 17:18:13 PCI: Bridge: 0000:80:0d.0 2011-06-23 17:18:13 IO window: disabled. 2011-06-23 17:18:13 MEM window: disabled. 2011-06-23 17:18:13 PREFETCH window: disabled. 2011-06-23 17:18:13 PCI: Bridge: 0000:80:0e.0 2011-06-23 17:18:13 IO window: disabled. 2011-06-23 17:18:13 MEM window: cff00000-cfffffff 2011-06-23 17:18:13 PREFETCH window 0x00000000ce800000-0x00000000ceffffff 2011-06-23 17:18:13 PCI: Bridge: 0000:80:0f.0 2011-06-23 17:18:13 IO window: disabled. 2011-06-23 17:18:13 MEM window: disabled. 2011-06-23 17:18:13 PREFETCH window: disabled. 2011-06-23 17:18:13 PCI: Bridge: 0000:00:06.0 2011-06-23 17:18:13 IO window: a000-afff 2011-06-23 17:18:13 MEM window: df700000-df7fffff 2011-06-23 17:18:13 PREFETCH window 0x00000000d0000000-0x00000000d7ffffff 2011-06-23 17:18:13 PCI: Bridge: 0000:00:0a.0 2011-06-23 17:18:13 IO window: b000-bfff 2011-06-23 17:18:13 MEM window: df800000-dfafffff 2011-06-23 17:18:13 PREFETCH window: disabled. 2011-06-23 17:18:13 PCI: Bridge: 0000:00:0b.0 2011-06-23 17:18:13 IO window: disabled. 2011-06-23 17:18:13 MEM window: disabled. 2011-06-23 17:18:13 PREFETCH window: disabled. 2011-06-23 17:18:13 PCI: Bridge: 0000:00:0c.0 2011-06-23 17:18:13 IO window: disabled. 2011-06-23 17:18:13 MEM window: disabled. 2011-06-23 17:18:13 PREFETCH window: disabled. 2011-06-23 17:18:13 PCI: Bridge: 0000:05:00.0 2011-06-23 17:18:13 IO window: c000-cfff 2011-06-23 17:18:13 MEM window: dfb00000-dfbfffff 2011-06-23 17:18:13 PREFETCH window: disabled. 2011-06-23 17:18:13 PCI: Bridge: 0000:05:00.1 2011-06-23 17:18:13 IO window: disabled. 2011-06-23 17:18:13 MEM window: disabled. 2011-06-23 17:18:13 PREFETCH window: disabled. 2011-06-23 17:18:13 PCI: Bridge: 0000:00:0d.0 2011-06-23 17:18:13 IO window: c000-cfff 2011-06-23 17:18:13 MEM window: dfb00000-dfbfffff 2011-06-23 17:18:13 PREFETCH window: disabled. 2011-06-23 17:18:13 PCI: Bridge: 0000:00:0e.0 2011-06-23 17:18:13 IO window: disabled. 2011-06-23 17:18:13 MEM window: disabled. 2011-06-23 17:18:13 PREFETCH window: disabled. 2011-06-23 17:18:13 PCI: Bridge: 0000:00:0f.0 2011-06-23 17:18:13 IO window: d000-dfff 2011-06-23 17:18:13 MEM window: dfc00000-dfffffff 2011-06-23 17:18:13 PREFETCH window: disabled. 2011-06-23 17:18:13 NET: Registered protocol family 2 2011-06-23 17:18:13 IP route cache hash table entries: 1024 (order: 1, 8192 bytes) 2011-06-23 17:18:13 TCP established hash table entries: 4096 (order: 4, 65536 bytes) 2011-06-23 17:18:13 TCP bind hash table entries: 2048 (order: 3, 32768 bytes) 2011-06-23 17:18:13 TCP: Hash tables configured (established 4096 bind 2048) 2011-06-23 17:18:13 TCP reno registered 2011-06-23 17:18:13 audit: initializing netlink socket (disabled) 2011-06-23 17:18:13 type=2000 audit(1308874692.610:1): initialized 2011-06-23 17:18:13 Total HugeTLB memory allocated, 0 2011-06-23 17:18:13 VFS: Disk quotas dquot_6.5.1 2011-06-23 17:18:13 Dquot-cache hash table entries: 512 (order 0, 4096 bytes) 2011-06-23 17:18:13 Initializing Cryptographic API 2011-06-23 17:18:13 alg: No test for crc32c (crc32c-generic) 2011-06-23 17:18:13 ksign: Installing public key data 2011-06-23 17:18:13 Loading keyring 2011-06-23 17:18:13 io scheduler noop registered 2011-06-23 17:18:13 io scheduler anticipatory registered 2011-06-23 17:18:13 io scheduler deadline registered (default) 2011-06-23 17:18:13 io scheduler cfq registered 2011-06-23 17:18:13 pci 0000:80:00.0: Enabling HT MSI Mapping 2011-06-23 17:18:13 pci 0000:80:0a.0: Enabling HT MSI Mapping 2011-06-23 17:18:13 pci 0000:80:0b.0: Enabling HT MSI Mapping 2011-06-23 17:18:13 pci 0000:80:0c.0: Enabling HT MSI Mapping 2011-06-23 17:18:13 pci 0000:80:0d.0: Enabling HT MSI Mapping 2011-06-23 17:18:13 pci 0000:80:0e.0: Enabling HT MSI Mapping 2011-06-23 17:18:13 pci 0000:80:0f.0: Enabling HT MSI Mapping 2011-06-23 17:18:13 pci 0000:00:00.0: Enabling HT MSI Mapping 2011-06-23 17:18:13 pci 0000:00:05.0: Enabling HT MSI Mapping 2011-06-23 17:18:13 pci 0000:00:05.1: Enabling HT MSI Mapping 2011-06-23 17:18:14 pci 0000:00:05.2: Enabling HT MSI Mapping 2011-06-23 17:18:14 pci 0000:00:06.0: Enabling HT MSI Mapping 2011-06-23 17:18:14 pci 0000:00:0a.0: Enabling HT MSI Mapping 2011-06-23 17:18:14 pci 0000:00:0b.0: Enabling HT MSI Mapping 2011-06-23 17:18:14 pci 0000:00:0c.0: Enabling HT MSI Mapping 2011-06-23 17:18:14 pci 0000:00:0d.0: Enabling HT MSI Mapping 2011-06-23 17:18:14 pci 0000:00:0e.0: Enabling HT MSI Mapping 2011-06-23 17:18:14 pci 0000:00:0f.0: Enabling HT MSI Mapping 2011-06-23 17:18:14 pci_hotplug: PCI Hot Plug PCI Core version: 0.5 2011-06-23 17:18:14 Real Time Clock Driver v1.12ac 2011-06-23 17:18:14 Non-volatile memory driver v1.2 2011-06-23 17:18:14 Linux agpgart interface v0.101 (c) Dave Jones 2011-06-23 17:18:14 Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing enabled Console [hyperion-mds1] departed by on pts/152 at 06-23 17:22.