2011-08-02 19:46:42 hyperion-mds1 login: Console [hyperion-mds1] departed by on pts/156 at 08-02 19:46. 2011-08-02 19:47:04 nodeup: starting at 08/02/11 19:47:04. 2011-08-02 19:47:04 nodeup: Turning on panic_on_pci_parity 2011-08-02 19:47:04 nodeup: mounting NFS filesystems 2011-08-02 19:47:04 mount_rfs: mounting nfs file systems 2011-08-02 19:47:04 mount_rfs: filesystems mounted successfuly 2011-08-02 19:47:04 nodeup: mounting Lustre filesystems(If present) 2011-08-02 19:47:05 mount_rfs: node is not a compute, login, or lustre_client node 2011-08-02 19:47:05 nodeup: complete at 08/02/11 19:47:05. 2011-08-02 19:47:21 Lustre: OBD class driver, http://wiki.whamcloud.com/ 2011-08-02 19:47:21 Lustre: Lustre Version: 2.0.66 2011-08-02 19:47:21 Lustre: Build Version: jenkins-gadb6cea-PRISTINE-2.6.18-238.12.1.el5_lustre 2011-08-02 19:47:21 Lustre: Lustre LU module (ffffffff88742be0). 2011-08-02 19:47:23 Lustre: Lustre OSC module (ffffffff889268a0). 2011-08-02 19:47:24 Lustre: Lustre LOV module (ffffffff88995660). 2011-08-02 19:47:25 Lustre: Lustre client module (ffffffff88a36a20). 2011-08-02 19:47:26 init dynlocks cache 2011-08-02 19:47:26 ldiskfs created from ext4-2.6-rhel5 2011-08-02 19:47:36 LDISKFS-fs (md0): recovery complete 2011-08-02 19:47:36 LDISKFS-fs (md0): mounted filesystem with ordered data mode 2011-08-02 19:47:36 JBD: barrier-based sync failed on md0-8 - disabling barriers 2011-08-02 19:47:38 LDISKFS-fs (md0): mounted filesystem with ordered data mode 2011-08-02 19:47:38 Lustre: MGS MGS started 2011-08-02 19:47:38 Lustre: 24171:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.120.125@o2ib->MGC192.168.120.125@o2ib_0 netid 90000: select flavor null 2011-08-02 19:47:38 Lustre: 24557:0:(ldlm_lib.c:874:target_handle_connect()) MGS: connection from fff79cb1-731b-11a9-295d-991e47ff62d6@0@lo t0 exp 0000000000000000 cur 1312339658 last 0 2011-08-02 19:47:38 Lustre: 24557:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGS->MGC192.168.120.125@o2ib_0 netid 90000: select flavor null 2011-08-02 19:47:38 Lustre: MGC192.168.120.125@o2ib: Reactivating import 2011-08-02 19:47:38 Lustre: 24556:0:(ldlm_lib.c:874:target_handle_connect()) MGS: connection from be513c1c-1a2c-d19c-d9b8-f71ee8e11313@192.168.114.125@o2ib t0 exp 0000000000000000 cur 1312339658 last 0 2011-08-02 19:47:38 JBD: barrier-based sync failed on md0-8 - disabling barriers 2011-08-02 19:47:39 Lustre: 24556:0:(ldlm_lib.c:874:target_handle_connect()) MGS: connection from c3065547-b103-9ea9-de94-cff16431e0d1@192.168.114.81@o2ib t0 exp 0000000000000000 cur 1312339659 last 0 2011-08-02 19:47:39 Lustre: 24556:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 4 previous similar messages 2011-08-02 19:47:39 Lustre: 24557:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGS->NET_0x50000c0a87250_UUID netid 50000: select flavor null 2011-08-02 19:47:39 Lustre: 24557:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 5 previous similar messages 2011-08-02 19:47:40 Lustre: Enabling ACL 2011-08-02 19:47:40 Lustre: 24556:0:(ldlm_lib.c:874:target_handle_connect()) MGS: connection from 0f9a4ec4-c1a6-591f-3dc3-8f404ba72b2a@192.168.114.136@o2ib t0 exp 0000000000000000 cur 1312339660 last 0 2011-08-02 19:47:40 Lustre: 24556:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 2 previous similar messages 2011-08-02 19:47:40 Lustre: 24556:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGS->NET_0x50000c0a87288_UUID netid 50000: select flavor null 2011-08-02 19:47:40 Lustre: 24556:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 3 previous similar messages 2011-08-02 19:47:40 Lustre: lustre-MDT0000: used disk, loading 2011-08-02 19:47:40 Lustre: 24562:0:(ldlm_lib.c:1896:target_recovery_init()) RECOVERY: service lustre-MDT0000, 125 recoverable clients, last_transno 4329017803 2011-08-02 19:47:40 LustreError: 24659:0:(ldlm_lib.c:1733:target_recovery_thread()) lustre-MDT0000: started recovery thread pid 24659 2011-08-02 19:47:40 Lustre: 24562:0:(mds_lov.c:1004:mds_notify()) MDS mdd_obd-lustre-MDT0000: add target lustre-OST0000_UUID 2011-08-02 19:47:40 Lustre: 24562:0:(mds_lov.c:1004:mds_notify()) MDS mdd_obd-lustre-MDT0000: add target lustre-OST0001_UUID 2011-08-02 19:47:40 Lustre: 24249:0:(mds_lov.c:1024:mds_notify()) MDS mdd_obd-lustre-MDT0000: in recovery, not resetting orphans on lustre-OST0000_UUID 2011-08-02 19:47:40 Lustre: 24249:0:(mds_lov.c:1024:mds_notify()) MDS mdd_obd-lustre-MDT0000: in recovery, not resetting orphans on lustre-OST0001_UUID 2011-08-02 19:47:54 Lustre: 24556:0:(ldlm_lib.c:874:target_handle_connect()) MGS: connection from 1b2a907c-8c4b-3f81-e8e2-3bf4c2dddf5f@192.168.114.145@o2ib t0 exp 0000000000000000 cur 1312339674 last 0 2011-08-02 19:47:54 Lustre: 24556:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGS->NET_0x50000c0a87291_UUID netid 50000: select flavor null 2011-08-02 19:47:54 Lustre: 24556:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 8 previous similar messages 2011-08-02 19:47:55 Lustre: 24661:0:(ldlm_lib.c:2022:target_queue_recovery_request()) Next recovery transno: 4329017804, current: 4329017829, replaying 2011-08-02 19:47:55 Lustre: 24661:0:(ldlm_lib.c:2022:target_queue_recovery_request()) Next recovery transno: 4329017804, current: 4329017828, replaying 2011-08-02 19:47:56 Lustre: 24661:0:(ldlm_lib.c:2022:target_queue_recovery_request()) Next recovery transno: 4329017804, current: 4329017880, replaying 2011-08-02 19:47:56 Lustre: 24661:0:(ldlm_lib.c:2022:target_queue_recovery_request()) Skipped 2 previous similar messages 2011-08-02 19:47:57 Lustre: 24661:0:(ldlm_lib.c:2022:target_queue_recovery_request()) Next recovery transno: 4329017804, current: 4329017867, replaying 2011-08-02 19:47:57 Lustre: 24661:0:(ldlm_lib.c:2022:target_queue_recovery_request()) Skipped 4 previous similar messages 2011-08-02 19:47:59 Lustre: 24556:0:(ldlm_lib.c:874:target_handle_connect()) MGS: connection from 939db2a3-5d0f-1372-6c69-bc6246ed9686@192.168.114.104@o2ib t0 exp 0000000000000000 cur 1312339679 last 0 2011-08-02 19:47:59 Lustre: 24661:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-MDT0000->NET_0x50000c0a87268_UUID netid 50000: select flavor null 2011-08-02 19:47:59 Lustre: 24661:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 28 previous similar messages 2011-08-02 19:47:59 Lustre: 24661:0:(ldlm_lib.c:2022:target_queue_recovery_request()) Next recovery transno: 4329017804, current: 4329017820, replaying 2011-08-02 19:47:59 Lustre: 24661:0:(ldlm_lib.c:2022:target_queue_recovery_request()) Skipped 4 previous similar messages 2011-08-02 19:47:59 Lustre: 24556:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 29 previous similar messages 2011-08-02 19:48:03 Lustre: 24661:0:(ldlm_lib.c:2022:target_queue_recovery_request()) Next recovery transno: 4324919280, current: 4329017815, replaying 2011-08-02 19:48:03 Lustre: 24661:0:(ldlm_lib.c:2022:target_queue_recovery_request()) Skipped 11 previous similar messages 2011-08-02 19:48:10 Lustre: 24556:0:(ldlm_lib.c:874:target_handle_connect()) MGS: connection from 3303e411-8bdb-3d16-6cce-abd3168008c4@192.168.120.87@o2ib t0 exp 0000000000000000 cur 1312339690 last 0 2011-08-02 19:48:10 Lustre: 24556:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 34 previous similar messages 2011-08-02 19:48:10 Lustre: 24556:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGS->192.168.120.87@o2ib netid 50000: select flavor null 2011-08-02 19:48:10 Lustre: 24556:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 35 previous similar messages 2011-08-02 19:48:19 Lustre: 24661:0:(ldlm_lib.c:2022:target_queue_recovery_request()) Next recovery transno: 4324919280, current: 4329017916, replaying 2011-08-02 19:48:19 Lustre: 24661:0:(ldlm_lib.c:2022:target_queue_recovery_request()) Skipped 5 previous similar messages 2011-08-02 19:48:27 Lustre: 24556:0:(ldlm_lib.c:874:target_handle_connect()) MGS: connection from 81cc878e-2169-d2cf-97b5-efc956dbc8e5@192.168.114.128@o2ib t0 exp 0000000000000000 cur 1312339706 last 0 2011-08-02 19:48:27 Lustre: 24556:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 64 previous similar messages 2011-08-02 19:48:27 Lustre: 24556:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGS->NET_0x50000c0a87280_UUID netid 50000: select flavor null 2011-08-02 19:48:27 Lustre: 24556:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 64 previous similar messages 2011-08-02 19:48:44 Lustre: 24661:0:(ldlm_lib.c:2022:target_queue_recovery_request()) Next recovery transno: 4324919280, current: 4329017811, replaying 2011-08-02 19:48:44 Lustre: 24661:0:(ldlm_lib.c:2022:target_queue_recovery_request()) Skipped 44 previous similar messages 2011-08-02 19:48:55 LustreError: 24659:0:(ldlm_lib.c:1506:check_for_next_transno()) lustre-MDT0000: waking for gap in transno, VBR is OFF (skip: 4324919281, ql: 124, comp: 1, conn: 125, next: 4329017804, last_committed: 4329017803) 2011-08-02 19:49:16 Lustre: 24660:0:(ldlm_lib.c:2022:target_queue_recovery_request()) Next recovery transno: 4329019149, current: 4329019279, replaying 2011-08-02 19:49:16 Lustre: 24660:0:(ldlm_lib.c:2022:target_queue_recovery_request()) Skipped 1392 previous similar messages 2011-08-02 19:49:44 LustreError: 24659:0:(ldlm_lib.c:1506:check_for_next_transno()) lustre-MDT0000: waking for gap in transno, VBR is OFF (skip: 4329021557, ql: 121, comp: 4, conn: 125, next: 4329021558, last_committed: 4329021229) 2011-08-02 19:49:44 LustreError: 24659:0:(ldlm_lib.c:1506:check_for_next_transno()) lustre-MDT0000: waking for gap in transno, VBR is OFF (skip: 4329021581, ql: 121, comp: 4, conn: 125, next: 4329021582, last_committed: 4329021229) 2011-08-02 19:49:54 Lustre: lustre-MDT0000: sending delayed replies to recovered clients 2011-08-02 19:49:54 Lustre: 24659:0:(mds_lov.c:1024:mds_notify()) MDS mdd_obd-lustre-MDT0000: in recovery, not resetting orphans on lustre-OST0000_UUID 2011-08-02 19:49:54 Lustre: 24659:0:(mds_lov.c:1024:mds_notify()) Skipped 6 previous similar messages 2011-08-02 19:49:55 Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0007_UUID now active, resetting orphans 2011-08-02 19:49:55 Lustre: Skipped 3 previous similar messages 2011-08-02 19:49:55 Lustre: MDS mdd_obd-lustre-MDT0000: lustre-OST0000_UUID now active, resetting orphans 2011-08-02 19:49:55 Lustre: Skipped 3 previous similar messages 2011-08-02 19:53:57 Lustre: 24556:0:(ldlm_lib.c:874:target_handle_connect()) MGS: connection from 2a257e22-d98c-fd71-103a-039094cd0722@192.168.114.71@o2ib t0 exp 0000000000000000 cur 1312340037 last 0 2011-08-02 19:53:57 Lustre: 24556:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 113 previous similar messages 2011-08-02 19:53:57 Lustre: 24556:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGS->NET_0x50000c0a87247_UUID netid 50000: select flavor null 2011-08-02 19:53:57 Lustre: 24556:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 111 previous similar messages Console [hyperion-mds1] log at 2011-08-02 20:00:00 PDT. 2011-08-02 20:32:42 LustreError: 24944:0:(ldlm_lock.c:613:ldlm_add_bl_work_item()) ASSERTION(cfs_list_empty(&lock->l_bl_ast)) failed 2011-08-02 20:32:42 LustreError: 24944:0:(ldlm_lock.c:613:ldlm_add_bl_work_item()) LBUG 2011-08-02 20:32:42 Pid: 24944, comm: mdt_147 2011-08-02 20:32:42 2011-08-02 20:32:42 Call Trace: 2011-08-02 20:32:42 Aug 2 20:32:42 [] libcfs_debug_dumpstack+0x51/0x60 [libcfs] 2011-08-02 20:32:42 hyperion-mds1 Lu [] lbug_with_loc+0x7a/0xd0 [libcfs] 2011-08-02 20:32:42 streError: 24944 [] cfs_tracefile_init+0x0/0x10a [libcfs] 2011-08-02 20:32:42 :0:(ldlm_lock.c: [] ldlm_add_bl_work_item+0xbe/0x130 [ptlrpc] 2011-08-02 20:32:42 613:ldlm_add_bl_ [] ldlm_add_ast_work_item+0xb3/0x130 [ptlrpc] 2011-08-02 20:32:42 work_item()) ASS [] enqueue_task+0x41/0x56 2011-08-02 20:32:42 ERTION(cfs_list_ [] ldlm_inodebits_compat_queue+0x25a/0x360 [ptlrpc] 2011-08-02 20:32:42 empty(&lock->l_b [] ldlm_res_hop_fid_hash+0x1cb/0x1f0 [ptlrpc] 2011-08-02 20:32:42 l_ast)) failed 2011-08-02 20:32:42 A [] try_to_wake_up+0x472/0x484 2011-08-02 20:32:42 ug 2 20:32:42 h [] ldlm_process_inodebits_lock+0x2b6/0x450 [ptlrpc] 2011-08-02 20:32:42 yperion-mds1 Lus [] ldlm_resource_get+0x127/0x6bd [ptlrpc] 2011-08-02 20:32:42 treError: 24944: [] ldlm_lock_enqueue+0x84e/0xa20 [ptlrpc] 2011-08-02 20:32:42 0:(ldlm_lock.c:6 [] ldlm_lock_create+0x59b/0xab0 [ptlrpc] 2011-08-02 20:32:42 13:ldlm_add_bl_w [] cfs_hash_bd_lookup_intent+0x83/0x130 [libcfs] 2011-08-02 20:32:42 ork_item()) LBUG [] ldlm_cli_enqueue_local+0x363/0x500 [ptlrpc] 2011-08-02 20:32:42 2011-08-02 20:32:42 [] lu_object_find_at+0xa0/0x700 [obdclass] 2011-08-02 20:32:42 [] mdd_lookup+0x8e/0x110 [mdd] 2011-08-02 20:32:42 [] ldlm_completion_ast+0x0/0x6a0 [ptlrpc] 2011-08-02 20:32:42 [] mdt_blocking_ast+0x0/0x2e0 [mdt] 2011-08-02 20:32:42 [] mdt_object_lock+0x8f6/0xa40 [mdt] 2011-08-02 20:32:42 [] mdt_blocking_ast+0x0/0x2e0 [mdt] 2011-08-02 20:32:42 [] ldlm_completion_ast+0x0/0x6a0 [ptlrpc] 2011-08-02 20:32:42 [] mdt_reint_unlink+0x813/0xe00 [mdt] 2011-08-02 20:32:42 [] mdt_root_squash+0x33/0x440 [mdt] 2011-08-02 20:32:42 [] mdt_unlink_unpack+0x388/0x520 [mdt] 2011-08-02 20:32:42 [] mdt_reint_rec+0x7a/0x100 [mdt] 2011-08-02 20:32:42 [] lustre_msg_get_flags+0x1f/0x80 [ptlrpc] 2011-08-02 20:32:42 [] mdt_reint_internal+0x8f4/0x9b0 [mdt] 2011-08-02 20:32:42 [] mdt_reint_opcode+0x8f/0x150 [mdt] 2011-08-02 20:32:42 [] mdt_reint+0x82/0x100 [mdt] 2011-08-02 20:32:42 [] lustre_msg_get_opc+0x1e/0xf0 [ptlrpc] 2011-08-02 20:32:42 [] mdt_handle_common+0x119f/0x1700 [mdt] 2011-08-02 20:32:42 [] lustre_msg_get_opc+0x1e/0xf0 [ptlrpc] 2011-08-02 20:32:42 [] ptlrpc_server_handle_request+0x8b7/0xf10 [ptlrpc] 2011-08-02 20:32:42 [] try_to_wake_up+0x472/0x484 2011-08-02 20:32:42 [] __mod_timer+0x100/0x10f 2011-08-02 20:32:42 [] _spin_lock_bh+0x9/0x14 2011-08-02 20:32:42 [] lc_watchdog_del_pending+0x16/0x70 [libcfs] 2011-08-02 20:32:42 [] lc_watchdog_touch+0xbf/0x120 [libcfs] 2011-08-02 20:32:42 [] ptlrpc_main+0x136e/0x1560 [ptlrpc] 2011-08-02 20:32:42 [] ptlrpc_main+0x0/0x1560 [ptlrpc] 2011-08-02 20:32:42 [] child_rip+0xa/0x11 2011-08-02 20:32:42 [] ptlrpc_main+0x0/0x1560 [ptlrpc] 2011-08-02 20:32:42 [] ptlrpc_main+0x0/0x1560 [ptlrpc] 2011-08-02 20:32:42 [] child_rip+0x0/0x11 2011-08-02 20:32:42 2011-08-02 20:32:42 Kernel panic - not syncing: LBUG 2011-08-02 20:32:42 <2>REWRITING MCP55 CFG REG 2011-08-02 20:32:42 CFG = c1 2011-08-02 20:32:43 Linux version 2.6.18-238.12.1.el5_lustre (jenkins@client-3.lab.whamcloud.com) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-50)) #1 SMP Tue Jul 26 09:15:43 PDT 2011 2011-08-02 20:32:43 Command line: initrd=initrd-2.6.18-238.12.1.el5_lustre console=ttyS0,115200n8 elevator=deadline swiotlb=65536 selinux=0 BOOT_IMAGE=vmlinuz-2.6.18-238.12.1.el5_lustre BOOTIF=01-00-30-48-5f-28-26 irqpoll maxcpus=1 reset_devices hdb=cdrom memmap=exactmap memmap=640K@0K memmap=6080K@16384K memmap=124336K@23104K elfcorehdr=147440K memmap=56K#3276736K memmap=8K#3276792K memmap=4K$4173824K memmap=1024K$4175872K memmap=9216K$4185088K