2012-11-13T08:09:05+01:00 n-mds1 LustreError: 4211:0:(obd_class.h:521:obd_set_info_async()) obd_set_info_async: dev 0 no operation 2012-11-13T08:12:03+01:00 n-mds1 LustreError: 4211:0:(obd_class.h:521:obd_set_info_async()) obd_set_info_async: dev 0 no operation 2012-11-13T08:59:40+01:00 n-mds1 LustreError: 31980:0:(llog_cat.c:298:llog_cat_add_rec()) llog_write_rec -28: lh=ffff88042d450240 2012-11-13T10:37:46+01:00 n-mds1 LustreError: 138-a: nero-MDT0000: A client on nid 10.201.34.27@o2ib was evicted due to a lock blocking callback time out: rc -107 2012-11-13T14:57:46+01:00 n-mds1 LustreError: 138-a: nero-MDT0000: A client on nid 10.201.43.23@o2ib was evicted due to a lock blocking callback time out: rc -107 2012-11-13T15:31:06+01:00 n-mds1 LustreError: 138-a: nero-MDT0000: A client on nid 10.201.43.23@o2ib was evicted due to a lock blocking callback time out: rc -107 2012-11-13T15:50:52+01:00 n-mds1 LustreError: 0:0:(ldlm_lockd.c:357:waiting_locks_callback()) ### lock callback timer expired after 2119s: evicting client at 10.201.32.31@o2ib ns: mdt-ffff880c2a08a000 lock: ffff8804ad450240/0xb4f3993e21 786c1 lrc: 3/0,0 mode: PR/PR res: 8589985030/5616 bits 0x3 rrc: 3 type: IBT flags: 0x4000020 remote: 0x6d9f5e8ae7cb4595 expref: 270317 pid: 31786 timeout: 4710592781 2012-11-13T17:15:19+01:00 n-mds1 ------------[ cut here ]------------ 2012-11-13T17:15:19+01:00 n-mds1 kernel BUG at mm/slab.c:522! 2012-11-13T17:15:19+01:00 n-mds1 invalid opcode: 0000 [#1] 2012-11-13T17:15:19+01:00 n-mds1 SMP 2012-11-13T17:15:19+01:00 n-mds1 2012-11-13T17:15:19+01:00 n-mds1 last sysfs file: /sys/devices/pci0000:00/0000:00:1e.0/0000:01:03.0/local_cpus 2012-11-13T17:15:19+01:00 n-mds1 CPU 7 2012-11-13T17:15:19+01:00 n-mds1 2012-11-13T17:15:19+01:00 n-mds1 Modules linked in: 2012-11-13T17:15:19+01:00 n-mds1 cmm 2012-11-13T17:15:20+01:00 n-mds1 (U) 2012-11-13T17:15:20+01:00 n-mds1 osd_ldiskfs 2012-11-13T17:15:20+01:00 n-mds1 (U) 2012-11-13T17:15:20+01:00 n-mds1 mdt 2012-11-13T17:15:20+01:00 n-mds1 (U) 2012-11-13T17:15:20+01:00 n-mds1 mdd 2012-11-13T17:15:20+01:00 n-mds1 (U) 2012-11-13T17:15:20+01:00 n-mds1 mds 2012-11-13T17:15:20+01:00 n-mds1 (U) 2012-11-13T17:15:20+01:00 n-mds1 fsfilt_ldiskfs 2012-11-13T17:15:20+01:00 n-mds1 (U) 2012-11-13T17:15:20+01:00 n-mds1 exportfs 2012-11-13T17:15:20+01:00 n-mds1 mgs 2012-11-13T17:15:20+01:00 n-mds1 (U) 2012-11-13T17:15:20+01:00 n-mds1 mgc 2012-11-13T17:15:20+01:00 n-mds1 (U) 2012-11-13T17:15:20+01:00 n-mds1 ldiskfs 2012-11-13T17:15:20+01:00 n-mds1 (U) 2012-11-13T17:15:20+01:00 n-mds1 lustre 2012-11-13T17:15:20+01:00 n-mds1 (U) 2012-11-13T17:15:20+01:00 n-mds1 lquota 2012-11-13T17:15:20+01:00 n-mds1 (U) 2012-11-13T17:15:20+01:00 n-mds1 lov 2012-11-13T17:15:20+01:00 n-mds1 (U) 2012-11-13T17:15:20+01:00 n-mds1 dm_round_robin 2012-11-13T17:15:20+01:00 n-mds1 osc 2012-11-13T17:15:20+01:00 n-mds1 (U) 2012-11-13T17:15:20+01:00 n-mds1 mdc 2012-11-13T17:15:20+01:00 n-mds1 (U) 2012-11-13T17:15:20+01:00 n-mds1 fid 2012-11-13T17:15:20+01:00 n-mds1 (U) 2012-11-13T17:15:20+01:00 n-mds1 fld 2012-11-13T17:15:20+01:00 n-mds1 (U) 2012-11-13T17:15:20+01:00 n-mds1 ksocklnd 2012-11-13T17:15:20+01:00 n-mds1 (U) 2012-11-13T17:15:20+01:00 n-mds1 ko2iblnd 2012-11-13T17:15:20+01:00 n-mds1 (U) 2012-11-13T17:15:20+01:00 n-mds1 ptlrpc 2012-11-13T17:15:20+01:00 n-mds1 (U) 2012-11-13T17:15:20+01:00 n-mds1 obdclass 2012-11-13T17:15:20+01:00 n-mds1 (U) 2012-11-13T17:15:20+01:00 n-mds1 lnet 2012-11-13T17:15:20+01:00 n-mds1 (U) 2012-11-13T17:15:20+01:00 n-mds1 lvfs 2012-11-13T17:15:20+01:00 n-mds1 (U) 2012-11-13T17:15:20+01:00 n-mds1 libcfs 2012-11-13T17:15:20+01:00 n-mds1 (U) 2012-11-13T17:15:20+01:00 n-mds1 netconsole 2012-11-13T17:15:20+01:00 n-mds1 configfs 2012-11-13T17:15:20+01:00 n-mds1 qla2xxx 2012-11-13T17:15:20+01:00 n-mds1 scsi_transport_fc 2012-11-13T17:15:20+01:00 n-mds1 scsi_tgt 2012-11-13T17:15:20+01:00 n-mds1 autofs4 2012-11-13T17:15:20+01:00 n-mds1 lockd 2012-11-13T17:15:20+01:00 n-mds1 sunrpc 2012-11-13T17:15:20+01:00 n-mds1 ib_ipoib 2012-11-13T17:15:20+01:00 n-mds1 rdma_ucm 2012-11-13T17:15:20+01:00 n-mds1 ib_ucm 2012-11-13T17:15:20+01:00 n-mds1 ib_uverbs 2012-11-13T17:15:20+01:00 n-mds1 ib_umad 2012-11-13T17:15:20+01:00 n-mds1 rdma_cm 2012-11-13T17:15:20+01:00 n-mds1 ib_cm 2012-11-13T17:15:20+01:00 n-mds1 iw_cm 2012-11-13T17:15:20+01:00 n-mds1 ib_addr 2012-11-13T17:15:20+01:00 n-mds1 ipv6 2012-11-13T17:15:20+01:00 n-mds1 ib_sa 2012-11-13T17:15:20+01:00 n-mds1 microcode 2012-11-13T17:15:20+01:00 n-mds1 serio_raw 2012-11-13T17:15:20+01:00 n-mds1 i2c_i801 2012-11-13T17:15:20+01:00 n-mds1 iTCO_wdt 2012-11-13T17:15:20+01:00 n-mds1 iTCO_vendor_support 2012-11-13T17:15:20+01:00 n-mds1 e1000e 2012-11-13T17:15:20+01:00 n-mds1 ioatdma 2012-11-13T17:15:20+01:00 n-mds1 dca 2012-11-13T17:15:20+01:00 n-mds1 i7core_edac 2012-11-13T17:15:20+01:00 n-mds1 edac_core 2012-11-13T17:15:20+01:00 n-mds1 ses 2012-11-13T17:15:20+01:00 n-mds1 enclosure 2012-11-13T17:15:20+01:00 n-mds1 sg 2012-11-13T17:15:20+01:00 n-mds1 sfc 2012-11-13T17:15:20+01:00 n-mds1 i2c_algo_bit 2012-11-13T17:15:20+01:00 n-mds1 i2c_core 2012-11-13T17:15:20+01:00 n-mds1 mdio 2012-11-13T17:15:20+01:00 n-mds1 mlx4_ib 2012-11-13T17:15:20+01:00 n-mds1 ib_mad 2012-11-13T17:15:20+01:00 n-mds1 ib_core 2012-11-13T17:15:20+01:00 n-mds1 mlx4_en 2012-11-13T17:15:20+01:00 n-mds1 mlx4_core 2012-11-13T17:15:20+01:00 n-mds1 ext4 2012-11-13T17:15:20+01:00 n-mds1 mbcache 2012-11-13T17:15:20+01:00 n-mds1 jbd2 2012-11-13T17:15:20+01:00 n-mds1 sr_mod 2012-11-13T17:15:20+01:00 n-mds1 cdrom 2012-11-13T17:15:20+01:00 n-mds1 sd_mod 2012-11-13T17:15:20+01:00 n-mds1 crc_t10dif 2012-11-13T17:15:20+01:00 n-mds1 pata_acpi 2012-11-13T17:15:20+01:00 n-mds1 ata_generic 2012-11-13T17:15:20+01:00 n-mds1 ata_piix 2012-11-13T17:15:20+01:00 n-mds1 aacraid 2012-11-13T17:15:20+01:00 n-mds1 dm_multipath 2012-11-13T17:15:20+01:00 n-mds1 dm_mirror 2012-11-13T17:15:20+01:00 n-mds1 dm_region_hash 2012-11-13T17:15:20+01:00 n-mds1 dm_log 2012-11-13T17:15:20+01:00 n-mds1 dm_mod 2012-11-13T17:15:20+01:00 n-mds1 [last unloaded: scsi_wait_scan] 2012-11-13T17:15:20+01:00 n-mds1 2012-11-13T17:15:20+01:00 n-mds1 2012-11-13T17:15:20+01:00 n-mds1 Pid: 0, comm: swapper Not tainted 2.6.32-220.4.2.el6_lustre.x86_64 #1 2012-11-13T17:15:20+01:00 n-mds1 Supermicro X8DT6 2012-11-13T17:15:20+01:00 n-mds1 /X8DT6 2012-11-13T17:15:20+01:00 n-mds1 2012-11-13T17:15:20+01:00 n-mds1 RIP: 0010:[] 2012-11-13T17:15:20+01:00 n-mds1 [] kfree+0x29c/0x320 2012-11-13T17:15:20+01:00 n-mds1 RSP: 0018:ffff880655423dd0 EFLAGS: 00010002 2012-11-13T17:15:21+01:00 n-mds1 RAX: ffffea001f7fffc8 RBX: ffff8808ffffffff RCX: 0000000000000000 2012-11-13T17:15:21+01:00 n-mds1 RDX: 00c0000000000864 RSI: 000000000000005a RDI: ffff8808ffffffff 2012-11-13T17:15:21+01:00 n-mds1 RBP: ffff880655423e30 R08: 0000000000000000 R09: ffff8808ffffffff 2012-11-13T17:15:21+01:00 n-mds1 R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa04377be 2012-11-13T17:15:21+01:00 n-mds1 R13: 0000000000000282 R14: ffff8809216215f8 R15: ffff8806554312f0 2012-11-13T17:15:21+01:00 n-mds1 FS: 0000000000000000(0000) GS:ffff880655420000(0000) knlGS:0000000000000000 2012-11-13T17:15:21+01:00 n-mds1 CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b 2012-11-13T17:15:21+01:00 n-mds1 CR2: 000000000251b000 CR3: 0000000c2a1e3000 CR4: 00000000000006e0 2012-11-13T17:15:21+01:00 n-mds1 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 2012-11-13T17:15:21+01:00 n-mds1 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 2012-11-13T17:15:21+01:00 n-mds1 Process swapper (pid: 0, threadinfo ffff880c30fa2000, task ffff880630a66100) 2012-11-13T17:15:21+01:00 n-mds1 Stack: 2012-11-13T17:15:21+01:00 n-mds1 ffff880630a66100 2012-11-13T17:15:21+01:00 n-mds1 000000000000f4e8 2012-11-13T17:15:21+01:00 n-mds1 ffff880655423e50 2012-11-13T17:15:21+01:00 n-mds1 0000000000000286 2012-11-13T17:15:21+01:00 n-mds1 2012-11-13T17:15:21+01:00 n-mds1 kernel: 2012-11-13T17:15:21+01:00 n-mds1 ffff880655435fc0 2012-11-13T17:15:21+01:00 n-mds1 ffff880b563c8cc0 2012-11-13T17:15:21+01:00 n-mds1 ffff88063fc25100 2012-11-13T17:15:21+01:00 n-mds1 ffff8808ffffffff 2012-11-13T17:15:21+01:00 n-mds1 2012-11-13T17:15:21+01:00 n-mds1 kernel: 2012-11-13T17:15:21+01:00 n-mds1 0000000000000080 2012-11-13T17:15:21+01:00 n-mds1 0000000000000080 2012-11-13T17:15:21+01:00 n-mds1 ffff8809216215f8 2012-11-13T17:15:22+01:00 n-mds1 ffff8806554312f0 2012-11-13T17:15:22+01:00 n-mds1 2012-11-13T17:15:22+01:00 n-mds1 Call Trace: 2012-11-13T17:15:22+01:00 n-mds1 2012-11-13T17:15:22+01:00 n-mds1 2012-11-13T17:15:22+01:00 n-mds1 [] cfs_free+0xe/0x10 [libcfs] 2012-11-13T17:15:22+01:00 n-mds1 [] class_handle_free_cb+0xb0/0x100 [obdclass] 2012-11-13T17:15:22+01:00 n-mds1 [] __rcu_process_callbacks+0x10d/0x330 2012-11-13T17:15:22+01:00 n-mds1 [] rcu_process_callbacks+0x2b/0x50 2012-11-13T17:15:22+01:00 n-mds1 [] __do_softirq+0xc1/0x1d0 2012-11-13T17:15:22+01:00 n-mds1 [] ? hrtimer_interrupt+0x140/0x250 2012-11-13T17:15:22+01:00 n-mds1 [] call_softirq+0x1c/0x30 2012-11-13T17:15:22+01:00 n-mds1 [] do_softirq+0x65/0xa0 2012-11-13T17:15:22+01:00 n-mds1 [] irq_exit+0x85/0x90 2012-11-13T17:15:22+01:00 n-mds1 [] smp_apic_timer_interrupt+0x70/0x9b 2012-11-13T17:15:22+01:00 n-mds1 [] apic_timer_interrupt+0x13/0x20 2012-11-13T17:15:22+01:00 n-mds1 2012-11-13T17:15:22+01:00 n-mds1 2012-11-13T17:15:22+01:00 n-mds1 [] ? intel_idle+0xde/0x170 2012-11-13T17:15:22+01:00 n-mds1 [] ? intel_idle+0xc1/0x170 2012-11-13T17:15:22+01:00 n-mds1 [] ? sched_clock_cpu+0xcd/0x110 2012-11-13T17:15:22+01:00 n-mds1 [] cpuidle_idle_call+0xa7/0x140 2012-11-13T17:15:22+01:00 n-mds1 [] cpu_idle+0xb6/0x110 2012-11-13T17:15:22+01:00 n-mds1 [] start_secondary+0x202/0x245 2012-11-13T17:15:22+01:00 n-mds1 Code: 2012-11-13T17:15:22+01:00 n-mds1 4d 2012-11-13T17:15:22+01:00 n-mds1 c8 2012-11-13T17:15:22+01:00 n-mds1 89 2012-11-13T17:15:22+01:00 n-mds1 c2 2012-11-13T17:15:22+01:00 n-mds1 83 2012-11-13T17:15:22+01:00 n-mds1 c0 2012-11-13T17:15:22+01:00 n-mds1 01 2012-11-13T17:15:23+01:00 n-mds1 49 2012-11-13T17:15:23+01:00 n-mds1 89 2012-11-13T17:15:23+01:00 n-mds1 4c 2012-11-13T17:15:23+01:00 n-mds1 d4 2012-11-13T17:15:23+01:00 n-mds1 18 2012-11-13T17:15:23+01:00 n-mds1 41 2012-11-13T17:15:23+01:00 n-mds1 89 2012-11-13T17:15:23+01:00 n-mds1 04 2012-11-13T17:15:23+01:00 n-mds1 24 2012-11-13T17:15:23+01:00 n-mds1 66 2012-11-13T17:15:23+01:00 n-mds1 ff 2012-11-13T17:15:23+01:00 n-mds1 03 2012-11-13T17:15:23+01:00 n-mds1 66 2012-11-13T17:15:23+01:00 n-mds1 66 2012-11-13T17:15:23+01:00 n-mds1 90 2012-11-13T17:15:23+01:00 n-mds1 e9 2012-11-13T17:15:23+01:00 n-mds1 91 2012-11-13T17:15:23+01:00 n-mds1 fe 2012-11-13T17:15:23+01:00 n-mds1 ff 2012-11-13T17:15:23+01:00 n-mds1 ff 2012-11-13T17:15:23+01:00 n-mds1 0f 2012-11-13T17:15:23+01:00 n-mds1 0b 2012-11-13T17:15:23+01:00 n-mds1 eb 2012-11-13T17:15:23+01:00 n-mds1 fe 2012-11-13T17:15:23+01:00 n-mds1 48 2012-11-13T17:15:23+01:00 n-mds1 8b 2012-11-13T17:15:23+01:00 n-mds1 40 2012-11-13T17:15:23+01:00 n-mds1 10 2012-11-13T17:15:23+01:00 n-mds1 48 2012-11-13T17:15:23+01:00 n-mds1 8b 2012-11-13T17:15:23+01:00 n-mds1 10 2012-11-13T17:15:23+01:00 n-mds1 e9 2012-11-13T17:15:23+01:00 n-mds1 46 2012-11-13T17:15:23+01:00 n-mds1 fe 2012-11-13T17:15:23+01:00 n-mds1 ff 2012-11-13T17:15:23+01:00 n-mds1 ff 2012-11-13T17:15:23+01:00 n-mds1 syslog-ng[2221]: Error processing log message: <0f> 2012-11-13T17:15:23+01:00 n-mds1 0b 2012-11-13T17:15:23+01:00 n-mds1 66 2012-11-13T17:15:23+01:00 n-mds1 90 2012-11-13T17:15:23+01:00 n-mds1 eb 2012-11-13T17:15:23+01:00 n-mds1 fc 2012-11-13T17:15:23+01:00 n-mds1 48 2012-11-13T17:15:23+01:00 n-mds1 8b 2012-11-13T17:15:23+01:00 n-mds1 40 2012-11-13T17:15:23+01:00 n-mds1 10 2012-11-13T17:15:23+01:00 n-mds1 48 2012-11-13T17:15:23+01:00 n-mds1 8b 2012-11-13T17:15:23+01:00 n-mds1 10 2012-11-13T17:15:23+01:00 n-mds1 66 2012-11-13T17:15:23+01:00 n-mds1 85 2012-11-13T17:15:23+01:00 n-mds1 d2 2012-11-13T17:15:23+01:00 n-mds1 0f 2012-11-13T17:15:23+01:00 n-mds1 89 2012-11-13T17:15:23+01:00 n-mds1 d5 2012-11-13T17:15:23+01:00 n-mds1 fd 2012-11-13T17:15:23+01:00 n-mds1 ff 2012-11-13T17:15:23+01:00 n-mds1 2012-11-13T17:15:23+01:00 n-mds1 RIP 2012-11-13T17:15:23+01:00 n-mds1 [] kfree+0x29c/0x320 2012-11-13T17:15:23+01:00 n-mds1 RSP 2012-11-13T17:15:23+01:00 n-mds1 ---[ end trace ddbb06fb20789fb0 ]--- 2012-11-13T17:15:23+01:00 n-mds1 Kernel panic - not syncing: Fatal exception in interrupt 2012-11-13T17:15:23+01:00 n-mds1 Pid: 0, comm: swapper Tainted: G D ---------------- 2.6.32-220.4.2.el6_lustre.x86_64 #1 2012-11-13T17:15:23+01:00 n-mds1 Call Trace: 2012-11-13T17:15:23+01:00 n-mds1 2012-11-13T17:15:23+01:00 n-mds1 [] ? panic+0x78/0x143 2012-11-13T17:15:23+01:00 n-mds1 [] ? oops_end+0xf2/0x100 2012-11-13T17:15:23+01:00 n-mds1 [] ? die+0x5b/0x90 2012-11-13T17:15:23+01:00 n-mds1 [] ? do_trap+0xc4/0x160 2012-11-13T17:15:23+01:00 n-mds1 [] ? do_invalid_op+0x95/0xb0 2012-11-13T17:15:23+01:00 n-mds1 [] ? kfree+0x29c/0x320 2012-11-13T17:15:23+01:00 n-mds1 [] ? cfs_free+0xe/0x10 [libcfs] 2012-11-13T17:15:23+01:00 n-mds1 [] ? invalid_op+0x1b/0x20 2012-11-13T17:15:23+01:00 n-mds1 [] ? cfs_free+0xe/0x10 [libcfs] 2012-11-13T17:15:23+01:00 n-mds1 [] ? kfree+0x29c/0x320 2012-11-13T17:15:23+01:00 n-mds1 [] ? cfs_free+0xe/0x10 [libcfs] 2012-11-13T17:15:24+01:00 n-mds1 [] ? class_handle_free_cb+0xb0/0x100 [obdclass] 2012-11-13T17:15:24+01:00 n-mds1 [] ? __rcu_process_callbacks+0x10d/0x330 2012-11-13T17:15:24+01:00 n-mds1 [] ? rcu_process_callbacks+0x2b/0x50 2012-11-13T17:15:24+01:00 n-mds1 [] ? __do_softirq+0xc1/0x1d0 2012-11-13T17:15:24+01:00 n-mds1 [] ? hrtimer_interrupt+0x140/0x250 2012-11-13T17:15:24+01:00 n-mds1 [] ? call_softirq+0x1c/0x30 2012-11-13T17:15:24+01:00 n-mds1 [] ? do_softirq+0x65/0xa0 2012-11-13T17:15:24+01:00 n-mds1 [] ? irq_exit+0x85/0x90 2012-11-13T17:15:24+01:00 n-mds1 [] ? smp_apic_timer_interrupt+0x70/0x9b 2012-11-13T17:15:24+01:00 n-mds1 [] ? apic_timer_interrupt+0x13/0x20 2012-11-13T17:15:24+01:00 n-mds1 2012-11-13T17:15:24+01:00 n-mds1 [] ? intel_idle+0xde/0x170 2012-11-13T17:15:24+01:00 n-mds1 [] ? intel_idle+0xc1/0x170 2012-11-13T17:15:24+01:00 n-mds1 [] ? sched_clock_cpu+0xcd/0x110 2012-11-13T17:15:24+01:00 n-mds1 [] ? cpuidle_idle_call+0xa7/0x140 2012-11-13T17:15:24+01:00 n-mds1 [] ? cpu_idle+0xb6/0x110 2012-11-13T17:15:24+01:00 n-mds1 [] ? start_secondary+0x202/0x245 2012-11-13T18:04:33+01:00 n-mds1 SysRq : 2012-11-13T18:04:33+01:00 n-mds1 Resetting