------> n-mds1.log-20121109.gz <------ 2012-11-08T09:04:17+01:00 n-mds1 LustreError: 3809:0:(llog_cat.c:298:llog_cat_add_rec()) llog_write_rec -28: lh=ffff880482b99800 2012-11-08T10:37:10+01:00 n-mds1 LustreError: 138-a: nero-MDT0000: A client on nid 10.201.42.29@o2ib was evicted due to a lock blocking callback time out: rc -107 2012-11-08T10:51:41+01:00 n-mds1 LustreError: 3533:0:(obd_class.h:521:obd_set_info_async()) obd_set_info_async: dev 0 no operation 2012-11-08T15:16:39+01:00 n-mds1 LustreError: 138-a: nero-MDT0000: A client on nid 10.201.35.27@o2ib was evicted due to a lock blocking callback time out: rc -107 2012-11-08T20:14:25+01:00 n-mds1 general protection fault: 0000 [#1] 2012-11-08T20:14:25+01:00 n-mds1 SMP 2012-11-08T20:14:25+01:00 n-mds1 2012-11-08T20:14:25+01:00 n-mds1 last sysfs file: /sys/devices/system/cpu/cpu23/cache/index2/shared_cpu_map 2012-11-08T20:14:25+01:00 n-mds1 CPU 12 2012-11-08T20:14:25+01:00 n-mds1 2012-11-08T20:14:25+01:00 n-mds1 Modules linked in: 2012-11-08T20:14:25+01:00 n-mds1 nfs 2012-11-08T20:14:25+01:00 n-mds1 fscache 2012-11-08T20:14:25+01:00 n-mds1 nfs_acl 2012-11-08T20:14:25+01:00 n-mds1 auth_rpcgss 2012-11-08T20:14:25+01:00 n-mds1 cmm 2012-11-08T20:14:25+01:00 n-mds1 (U) 2012-11-08T20:14:25+01:00 n-mds1 osd_ldiskfs 2012-11-08T20:14:25+01:00 n-mds1 (U) 2012-11-08T20:14:25+01:00 n-mds1 mdt 2012-11-08T20:14:25+01:00 n-mds1 (U) 2012-11-08T20:14:25+01:00 n-mds1 mdd 2012-11-08T20:14:25+01:00 n-mds1 (U) 2012-11-08T20:14:25+01:00 n-mds1 mds 2012-11-08T20:14:25+01:00 n-mds1 (U) 2012-11-08T20:14:25+01:00 n-mds1 fsfilt_ldiskfs 2012-11-08T20:14:25+01:00 n-mds1 (U) 2012-11-08T20:14:25+01:00 n-mds1 exportfs 2012-11-08T20:14:25+01:00 n-mds1 mgs 2012-11-08T20:14:25+01:00 n-mds1 (U) 2012-11-08T20:14:25+01:00 n-mds1 mgc 2012-11-08T20:14:25+01:00 n-mds1 (U) 2012-11-08T20:14:25+01:00 n-mds1 ldiskfs 2012-11-08T20:14:25+01:00 n-mds1 (U) 2012-11-08T20:14:25+01:00 n-mds1 dm_round_robin 2012-11-08T20:14:25+01:00 n-mds1 lustre 2012-11-08T20:14:25+01:00 n-mds1 (U) 2012-11-08T20:14:25+01:00 n-mds1 lquota 2012-11-08T20:14:25+01:00 n-mds1 (U) 2012-11-08T20:14:25+01:00 n-mds1 lov 2012-11-08T20:14:25+01:00 n-mds1 (U) 2012-11-08T20:14:25+01:00 n-mds1 osc 2012-11-08T20:14:25+01:00 n-mds1 (U) 2012-11-08T20:14:25+01:00 n-mds1 mdc 2012-11-08T20:14:25+01:00 n-mds1 (U) 2012-11-08T20:14:25+01:00 n-mds1 fid 2012-11-08T20:14:25+01:00 n-mds1 (U) 2012-11-08T20:14:25+01:00 n-mds1 fld 2012-11-08T20:14:25+01:00 n-mds1 (U) 2012-11-08T20:14:25+01:00 n-mds1 ksocklnd 2012-11-08T20:14:25+01:00 n-mds1 (U) 2012-11-08T20:14:25+01:00 n-mds1 ko2iblnd 2012-11-08T20:14:25+01:00 n-mds1 (U) 2012-11-08T20:14:25+01:00 n-mds1 ptlrpc 2012-11-08T20:14:25+01:00 n-mds1 (U) 2012-11-08T20:14:25+01:00 n-mds1 obdclass 2012-11-08T20:14:25+01:00 n-mds1 (U) 2012-11-08T20:14:25+01:00 n-mds1 lnet 2012-11-08T20:14:25+01:00 n-mds1 (U) 2012-11-08T20:14:25+01:00 n-mds1 lvfs 2012-11-08T20:14:25+01:00 n-mds1 (U) 2012-11-08T20:14:26+01:00 n-mds1 libcfs 2012-11-08T20:14:26+01:00 n-mds1 (U) 2012-11-08T20:14:26+01:00 n-mds1 netconsole 2012-11-08T20:14:26+01:00 n-mds1 configfs 2012-11-08T20:14:26+01:00 n-mds1 qla2xxx 2012-11-08T20:14:26+01:00 n-mds1 scsi_transport_fc 2012-11-08T20:14:26+01:00 n-mds1 scsi_tgt 2012-11-08T20:14:26+01:00 n-mds1 autofs4 2012-11-08T20:14:26+01:00 n-mds1 lockd 2012-11-08T20:14:26+01:00 n-mds1 sunrpc 2012-11-08T20:14:26+01:00 n-mds1 ib_ipoib 2012-11-08T20:14:26+01:00 n-mds1 rdma_ucm 2012-11-08T20:14:26+01:00 n-mds1 ib_ucm 2012-11-08T20:14:26+01:00 n-mds1 ib_uverbs 2012-11-08T20:14:26+01:00 n-mds1 ib_umad 2012-11-08T20:14:26+01:00 n-mds1 rdma_cm 2012-11-08T20:14:26+01:00 n-mds1 ib_cm 2012-11-08T20:14:26+01:00 n-mds1 iw_cm 2012-11-08T20:14:26+01:00 n-mds1 ib_addr 2012-11-08T20:14:26+01:00 n-mds1 ipv6 2012-11-08T20:14:26+01:00 n-mds1 ib_sa 2012-11-08T20:14:26+01:00 n-mds1 microcode 2012-11-08T20:14:26+01:00 n-mds1 serio_raw 2012-11-08T20:14:26+01:00 n-mds1 i2c_i801 2012-11-08T20:14:26+01:00 n-mds1 iTCO_wdt 2012-11-08T20:14:26+01:00 n-mds1 iTCO_vendor_support 2012-11-08T20:14:26+01:00 n-mds1 e1000e 2012-11-08T20:14:26+01:00 n-mds1 ioatdma 2012-11-08T20:14:26+01:00 n-mds1 dca 2012-11-08T20:14:26+01:00 n-mds1 i7core_edac 2012-11-08T20:14:26+01:00 n-mds1 edac_core 2012-11-08T20:14:26+01:00 n-mds1 ses 2012-11-08T20:14:26+01:00 n-mds1 enclosure 2012-11-08T20:14:26+01:00 n-mds1 sg 2012-11-08T20:14:26+01:00 n-mds1 sfc 2012-11-08T20:14:26+01:00 n-mds1 i2c_algo_bit 2012-11-08T20:14:26+01:00 n-mds1 i2c_core 2012-11-08T20:14:26+01:00 n-mds1 mdio 2012-11-08T20:14:26+01:00 n-mds1 mlx4_ib 2012-11-08T20:14:26+01:00 n-mds1 ib_mad 2012-11-08T20:14:26+01:00 n-mds1 ib_core 2012-11-08T20:14:26+01:00 n-mds1 mlx4_en 2012-11-08T20:14:26+01:00 n-mds1 mlx4_core 2012-11-08T20:14:26+01:00 n-mds1 ext4 2012-11-08T20:14:26+01:00 n-mds1 mbcache 2012-11-08T20:14:26+01:00 n-mds1 jbd2 2012-11-08T20:14:26+01:00 n-mds1 sr_mod 2012-11-08T20:14:26+01:00 n-mds1 cdrom 2012-11-08T20:14:26+01:00 n-mds1 sd_mod 2012-11-08T20:14:26+01:00 n-mds1 crc_t10dif 2012-11-08T20:14:26+01:00 n-mds1 pata_acpi 2012-11-08T20:14:26+01:00 n-mds1 ata_generic 2012-11-08T20:14:26+01:00 n-mds1 ata_piix 2012-11-08T20:14:26+01:00 n-mds1 aacraid 2012-11-08T20:14:26+01:00 n-mds1 dm_multipath 2012-11-08T20:14:26+01:00 n-mds1 dm_mirror 2012-11-08T20:14:26+01:00 n-mds1 dm_region_hash 2012-11-08T20:14:26+01:00 n-mds1 dm_log 2012-11-08T20:14:26+01:00 n-mds1 dm_mod 2012-11-08T20:14:26+01:00 n-mds1 [last unloaded: scsi_wait_scan] 2012-11-08T20:14:26+01:00 n-mds1 2012-11-08T20:14:26+01:00 n-mds1 2012-11-08T20:14:26+01:00 n-mds1 Pid: 3845, comm: mdt_52 Not tainted 2.6.32-220.4.2.el6_lustre.x86_64 #1 2012-11-08T20:14:26+01:00 n-mds1 Supermicro X8DT6 2012-11-08T20:14:26+01:00 n-mds1 /X8DT6 2012-11-08T20:14:26+01:00 n-mds1 2012-11-08T20:14:26+01:00 n-mds1 RIP: 0010:[] 2012-11-08T20:14:26+01:00 n-mds1 [] osd_trans_stop+0xeb/0x390 [osd_ldiskfs] 2012-11-08T20:14:26+01:00 n-mds1 RSP: 0018:ffff880bd116da70 EFLAGS: 00010202 2012-11-08T20:14:26+01:00 n-mds1 RAX: 0006000100000002 RBX: ffff8802e46b92c0 RCX: ffff8802e46b92c0 2012-11-08T20:14:26+01:00 n-mds1 RDX: ffff88012b15e790 RSI: ffff8802e4464318 RDI: ffff8802e46b9318 2012-11-08T20:14:26+01:00 n-mds1 RBP: ffff880bd116daa0 R08: ffff88012b15e790 R09: ffff8802e46b92c0 2012-11-08T20:14:27+01:00 n-mds1 R10: ffff8805d3bc5fc0 R11: 0000000000000000 R12: ffff8801e2e80c78 2012-11-08T20:14:27+01:00 n-mds1 R13: ffff8805d56ca000 R14: ffff8805d5630ac0 R15: ffff880c2becfe8c 2012-11-08T20:14:27+01:00 n-mds1 FS: 00007f878c7bc700(0000) GS:ffff8800282c0000(0000) knlGS:0000000000000000 2012-11-08T20:14:27+01:00 n-mds1 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b 2012-11-08T20:14:27+01:00 n-mds1 CR2: 00007f238f4ff0a0 CR3: 0000000001a85000 CR4: 00000000000006e0 2012-11-08T20:14:27+01:00 n-mds1 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 2012-11-08T20:14:27+01:00 n-mds1 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 2012-11-08T20:14:27+01:00 n-mds1 Process mdt_52 (pid: 3845, threadinfo ffff880bd116c000, task ffff880bd116b500) 2012-11-08T20:14:27+01:00 n-mds1 Stack: 2012-11-08T20:14:27+01:00 n-mds1 ffff880bd116dab0 2012-11-08T20:14:27+01:00 n-mds1 0000000000000014 2012-11-08T20:14:27+01:00 n-mds1 0000000000000000 2012-11-08T20:14:27+01:00 n-mds1 0000000000000000 2012-11-08T20:14:27+01:00 n-mds1 2012-11-08T20:14:27+01:00 n-mds1 kernel: 2012-11-08T20:14:27+01:00 n-mds1 ffff8805d5630ac0 2012-11-08T20:14:27+01:00 n-mds1 ffff880628562a00 2012-11-08T20:14:27+01:00 n-mds1 ffff880bd116dab0 2012-11-08T20:14:27+01:00 n-mds1 ffffffffa0cde12a 2012-11-08T20:14:27+01:00 n-mds1 2012-11-08T20:14:27+01:00 n-mds1 kernel: 2012-11-08T20:14:27+01:00 n-mds1 ffff880bd116dbb0 2012-11-08T20:14:27+01:00 n-mds1 ffffffffa0cc1036 2012-11-08T20:14:27+01:00 n-mds1 0000000000000000 2012-11-08T20:14:27+01:00 n-mds1 ffffc90039eb7120 2012-11-08T20:14:27+01:00 n-mds1 2012-11-08T20:14:27+01:00 n-mds1 Call Trace: 2012-11-08T20:14:27+01:00 n-mds1 [] mdd_trans_stop+0x1a/0x20 [mdd] 2012-11-08T20:14:27+01:00 n-mds1 [] mdd_attr_set+0xbf6/0x2030 [mdd] 2012-11-08T20:14:27+01:00 n-mds1 [] ? ldlm_completion_ast+0x0/0x6d0 [ptlrpc] 2012-11-08T20:14:28+01:00 n-mds1 [] ? lustre_msg_get_versions+0x6c/0xb0 [ptlrpc] 2012-11-08T20:14:28+01:00 n-mds1 [] cml_attr_set+0x6c/0x160 [cmm] 2012-11-08T20:14:28+01:00 n-mds1 [] mdt_attr_set+0x268/0x4b0 [mdt] 2012-11-08T20:14:28+01:00 n-mds1 [] mdt_reint_setattr+0x34d/0x1060 [mdt] 2012-11-08T20:14:28+01:00 n-mds1 [] mdt_reint_rec+0x4b/0xa0 [mdt] 2012-11-08T20:14:28+01:00 n-mds1 [] mdt_reint_internal+0x479/0x7b0 [mdt] 2012-11-08T20:14:28+01:00 n-mds1 [] mdt_reint+0x4e/0xb0 [mdt] 2012-11-08T20:14:28+01:00 n-mds1 [] mdt_handle_common+0x74d/0x1400 [mdt] 2012-11-08T20:14:28+01:00 n-mds1 [] mdt_regular_handle+0x15/0x20 [mdt] 2012-11-08T20:14:28+01:00 n-mds1 [] ptlrpc_server_handle_request+0x3c1/0xcb0 [ptlrpc] 2012-11-08T20:14:28+01:00 n-mds1 [] ? cfs_timer_arm+0xe/0x10 [libcfs] 2012-11-08T20:14:28+01:00 n-mds1 [] ? lc_watchdog_touch+0x79/0x110 [libcfs] 2012-11-08T20:14:28+01:00 n-mds1 [] ? ptlrpc_wait_event+0xb2/0x2c0 [ptlrpc] 2012-11-08T20:14:28+01:00 n-mds1 [] ? __wake_up+0x53/0x70 2012-11-08T20:14:28+01:00 n-mds1 [] ptlrpc_main+0x71f/0x1210 [ptlrpc] 2012-11-08T20:14:28+01:00 n-mds1 [] ? ptlrpc_main+0x0/0x1210 [ptlrpc] 2012-11-08T20:14:28+01:00 n-mds1 [] child_rip+0xa/0x20 2012-11-08T20:14:29+01:00 n-mds1 [] ? ptlrpc_main+0x0/0x1210 [ptlrpc] 2012-11-08T20:14:29+01:00 n-mds1 [] ? ptlrpc_main+0x0/0x1210 [ptlrpc] 2012-11-08T20:14:29+01:00 n-mds1 [] ? child_rip+0x0/0x20 2012-11-08T20:14:29+01:00 n-mds1 Code: 2012-11-08T20:14:29+01:00 n-mds1 e0 2012-11-08T20:14:29+01:00 n-mds1 4c 2012-11-08T20:14:29+01:00 n-mds1 89 2012-11-08T20:14:29+01:00 n-mds1 f8 2012-11-08T20:14:29+01:00 n-mds1 66 2012-11-08T20:14:29+01:00 n-mds1 ff 2012-11-08T20:14:29+01:00 n-mds1 00 2012-11-08T20:14:29+01:00 n-mds1 66 2012-11-08T20:14:29+01:00 n-mds1 66 2012-11-08T20:14:29+01:00 n-mds1 90 2012-11-08T20:14:29+01:00 n-mds1 41 2012-11-08T20:14:29+01:00 n-mds1 83 2012-11-08T20:14:29+01:00 n-mds1 bd 2012-11-08T20:14:29+01:00 n-mds1 90 2012-11-08T20:14:29+01:00 n-mds1 09 2012-11-08T20:14:29+01:00 n-mds1 00 2012-11-08T20:14:29+01:00 n-mds1 00 2012-11-08T20:14:29+01:00 n-mds1 01 2012-11-08T20:14:29+01:00 n-mds1 0f 2012-11-08T20:14:29+01:00 n-mds1 85 2012-11-08T20:14:29+01:00 n-mds1 02 2012-11-08T20:14:29+01:00 n-mds1 02 2012-11-08T20:14:29+01:00 n-mds1 00 2012-11-08T20:14:29+01:00 n-mds1 00 2012-11-08T20:14:29+01:00 n-mds1 41 2012-11-08T20:14:29+01:00 n-mds1 c7 2012-11-08T20:14:29+01:00 n-mds1 85 2012-11-08T20:14:29+01:00 n-mds1 90 2012-11-08T20:14:29+01:00 n-mds1 09 2012-11-08T20:14:29+01:00 n-mds1 00 2012-11-08T20:14:29+01:00 n-mds1 00 2012-11-08T20:14:29+01:00 n-mds1 00 2012-11-08T20:14:29+01:00 n-mds1 00 2012-11-08T20:14:29+01:00 n-mds1 00 2012-11-08T20:14:29+01:00 n-mds1 00 2012-11-08T20:14:29+01:00 n-mds1 48 2012-11-08T20:14:29+01:00 n-mds1 8b 2012-11-08T20:14:29+01:00 n-mds1 03 2012-11-08T20:14:29+01:00 n-mds1 48 2012-11-08T20:14:29+01:00 n-mds1 85 2012-11-08T20:14:29+01:00 n-mds1 c0 2012-11-08T20:14:29+01:00 n-mds1 74 2012-11-08T20:14:29+01:00 n-mds1 09 2012-11-08T20:14:29+01:00 n-mds1 2012-11-08T20:14:29+01:00 n-mds1 8b 2012-11-08T20:14:29+01:00 n-mds1 40 2012-11-08T20:14:29+01:00 n-mds1 08 2012-11-08T20:14:29+01:00 n-mds1 f6 2012-11-08T20:14:29+01:00 n-mds1 00 2012-11-08T20:14:29+01:00 n-mds1 01 2012-11-08T20:14:29+01:00 n-mds1 74 2012-11-08T20:14:29+01:00 n-mds1 1e 2012-11-08T20:14:29+01:00 n-mds1 41 2012-11-08T20:14:29+01:00 n-mds1 8b 2012-11-08T20:14:29+01:00 n-mds1 bd 2012-11-08T20:14:29+01:00 n-mds1 88 2012-11-08T20:14:29+01:00 n-mds1 09 2012-11-08T20:14:29+01:00 n-mds1 00 2012-11-08T20:14:29+01:00 n-mds1 00 2012-11-08T20:14:29+01:00 n-mds1 85 2012-11-08T20:14:29+01:00 n-mds1 ff 2012-11-08T20:14:29+01:00 n-mds1 0f 2012-11-08T20:14:29+01:00 n-mds1 85 2012-11-08T20:14:29+01:00 n-mds1 13 2012-11-08T20:14:29+01:00 n-mds1 2012-11-08T20:14:29+01:00 n-mds1 RIP 2012-11-08T20:14:29+01:00 n-mds1 [] osd_trans_stop+0xeb/0x390 [osd_ldiskfs] 2012-11-08T20:14:29+01:00 n-mds1 RSP 2012-11-08T20:14:29+01:00 n-mds1 ---[ end trace 69a06040c21c938c ]--- 2012-11-08T20:14:29+01:00 n-mds1 Kernel panic - not syncing: Fatal exception 2012-11-08T20:14:29+01:00 n-mds1 Pid: 3845, comm: mdt_52 Tainted: G D ---------------- 2.6.32-220.4.2.el6_lustre.x86_64 #1 2012-11-08T20:14:29+01:00 n-mds1 Call Trace: 2012-11-08T20:14:29+01:00 n-mds1 [] ? panic+0x78/0x143 2012-11-08T20:14:29+01:00 n-mds1 [] ? oops_end+0xe4/0x100 2012-11-08T20:14:29+01:00 n-mds1 [] ? die+0x5b/0x90 2012-11-08T20:14:29+01:00 n-mds1 [] ? do_general_protection+0x152/0x160 2012-11-08T20:14:29+01:00 n-mds1 [] ? osd_xattr_set+0x14a/0x1d0 [osd_ldiskfs] 2012-11-08T20:14:29+01:00 n-mds1 [] ? general_protection+0x25/0x30 2012-11-08T20:14:30+01:00 n-mds1 [] ? osd_trans_stop+0xeb/0x390 [osd_ldiskfs] 2012-11-08T20:14:30+01:00 n-mds1 [] ? mdd_trans_stop+0x1a/0x20 [mdd] 2012-11-08T20:14:30+01:00 n-mds1 [] ? mdd_attr_set+0xbf6/0x2030 [mdd] 2012-11-08T20:14:30+01:00 n-mds1 [] ? ldlm_completion_ast+0x0/0x6d0 [ptlrpc] 2012-11-08T20:14:30+01:00 n-mds1 [] ? lustre_msg_get_versions+0x6c/0xb0 [ptlrpc] 2012-11-08T20:14:30+01:00 n-mds1 [] ? cml_attr_set+0x6c/0x160 [cmm] 2012-11-08T20:14:30+01:00 n-mds1 [] ? mdt_attr_set+0x268/0x4b0 [mdt] 2012-11-08T20:14:30+01:00 n-mds1 [] ? mdt_reint_setattr+0x34d/0x1060 [mdt] 2012-11-08T20:14:30+01:00 n-mds1 [] ? mdt_reint_rec+0x4b/0xa0 [mdt] 2012-11-08T20:14:30+01:00 n-mds1 [] ? mdt_reint_internal+0x479/0x7b0 [mdt] 2012-11-08T20:14:30+01:00 n-mds1 [] ? mdt_reint+0x4e/0xb0 [mdt] 2012-11-08T20:14:30+01:00 n-mds1 [] ? mdt_handle_common+0x74d/0x1400 [mdt] 2012-11-08T20:14:30+01:00 n-mds1 [] ? mdt_regular_handle+0x15/0x20 [mdt] 2012-11-08T20:14:30+01:00 n-mds1 [] ? ptlrpc_server_handle_request+0x3c1/0xcb0 [ptlrpc] 2012-11-08T20:14:30+01:00 n-mds1 [] ? cfs_timer_arm+0xe/0x10 [libcfs] 2012-11-08T20:14:30+01:00 n-mds1 [] ? lc_watchdog_touch+0x79/0x110 [libcfs] 2012-11-08T20:14:31+01:00 n-mds1 [] ? ptlrpc_wait_event+0xb2/0x2c0 [ptlrpc] 2012-11-08T20:14:31+01:00 n-mds1 [] ? __wake_up+0x53/0x70 2012-11-08T20:14:31+01:00 n-mds1 [] ? ptlrpc_main+0x71f/0x1210 [ptlrpc] 2012-11-08T20:14:31+01:00 n-mds1 [] ? ptlrpc_main+0x0/0x1210 [ptlrpc] 2012-11-08T20:14:31+01:00 n-mds1 [] ? child_rip+0xa/0x20 2012-11-08T20:14:31+01:00 n-mds1 [] ? ptlrpc_main+0x0/0x1210 [ptlrpc] 2012-11-08T20:14:31+01:00 n-mds1 [] ? ptlrpc_main+0x0/0x1210 [ptlrpc] 2012-11-08T20:14:31+01:00 n-mds1 [] ? child_rip+0x0/0x20