Apr 14 18:54:59 hostname kernel: [903595.045729] LustreError: 8117:0:(ofd_grant.c:352:ofd_grant_incoming()) fscratch-OST001d: cli 8c1795e2-8806-4e65-5865-4e42489eac9b/ffff8807dee68400 dirty 33554432 pend 0 grant -54657024
Apr 14 18:54:59 hostname kernel: [903595.193740] LustreError: 8117:0:(ofd_grant.c:354:ofd_grant_incoming()) LBUG
Apr 14 18:54:59 hostname kernel: [903595.267358] Pid: 8117, comm: ll_ost_io03_054
Apr 14 18:54:59 hostname kernel: [903595.338103]
Apr 14 18:54:59 hostname kernel: [903595.338105] Call Trace:
Apr 14 18:54:59 hostname kernel: [903595.472914]  [<ffffffffa04db8f5>] libcfs_debug_dumpstack+0x55/0x80 [libcfs]
Apr 14 18:54:59 hostname kernel: [903595.546178]  [<ffffffffa04dbef7>] lbug_with_loc+0x47/0xb0 [libcfs]
Apr 14 18:54:59 hostname kernel: [903595.617899]  [<ffffffffa1131a61>] ofd_grant_incoming+0x5d1/0x610 [ofd]
Apr 14 18:54:59 hostname kernel: [903595.689525]  [<ffffffffa1132403>] ofd_grant_prepare_write+0x2a3/0xeb0 [ofd]
Apr 14 18:54:59 hostname kernel: [903595.760638]  [<ffffffffa112aaff>] ofd_preprw+0x40f/0x1620 [ofd]
Apr 14 18:54:59 hostname kernel: [903595.829731]  [<ffffffffa0bc4c84>] ? sptlrpc_svc_alloc_rs+0x74/0x360 [ptlrpc]
Apr 14 18:54:59 hostname kernel: [903595.899044]  [<ffffffffa10bfe51>] obd_preprw+0x121/0x390 [ost]
Apr 14 18:54:59 hostname kernel: [903595.968059]  [<ffffffffa10c9c22>] ost_brw_write+0x982/0x15d0 [ost]
Apr 14 18:55:00 hostname kernel: [903596.037004]  [<ffffffffa0bd4a85>] ? null_authorize+0x75/0x100 [ptlrpc]
Apr 14 18:55:00 hostname kernel: [903596.106258]  [<ffffffffa0b9455e>] ? ptlrpc_send_reply+0x28e/0x7f0 [ptlrpc]
Apr 14 18:55:00 hostname kernel: [903596.175196]  [<ffffffff812925f9>] ? cpumask_next_and+0x29/0x50
Apr 14 18:55:00 hostname kernel: [903596.242052]  [<ffffffffa0b9c1cc>] ? lustre_msg_get_version+0x8c/0x100 [ptlrpc]
Apr 14 18:55:00 hostname kernel: [903596.309693]  [<ffffffffa0b9c328>] ? lustre_msg_check_version+0xe8/0x100 [ptlrpc]
Apr 14 18:55:00 hostname kernel: [903596.376660]  [<ffffffffa10d079c>] ost_handle+0x439c/0x44d0 [ost]
Apr 14 18:55:00 hostname kernel: [903596.442008]  [<ffffffffa0ba31fb>] ? ptlrpc_update_export_timer+0x4b/0x560 [ptlrpc]
Apr 14 18:55:00 hostname kernel: [903596.508592]  [<ffffffffa0baa2c5>] ptlrpc_server_handle_request+0x385/0xc00 [ptlrpc]
Apr 14 18:55:00 hostname kernel: [903596.574391]  [<ffffffffa04dc52e>] ? cfs_timer_arm+0xe/0x10 [libcfs]
Apr 14 18:55:00 hostname kernel: [903596.637640]  [<ffffffffa04ed845>] ? lc_watchdog_touch+0x65/0x170 [libcfs]
Apr 14 18:55:00 hostname kernel: [903596.700960]  [<ffffffffa0ba2cf9>] ? ptlrpc_wait_event+0xb9/0x2e0 [ptlrpc]
Apr 14 18:55:00 hostname kernel: [903596.762899]  [<ffffffff81059a49>] ? __wake_up_common+0x59/0x90
Apr 14 18:55:00 hostname kernel: [903596.822704]  [<ffffffffa0baca4d>] ptlrpc_main+0xaed/0x1930 [ptlrpc]
Apr 14 18:55:00 hostname kernel: [903596.882204]  [<ffffffffa0babf60>] ? ptlrpc_main+0x0/0x1930 [ptlrpc]
Apr 14 18:55:00 hostname kernel: [903596.940466]  [<ffffffff810a18be>] kthread+0x9e/0xc0
Apr 14 18:55:01 hostname kernel: [903596.995922]  [<ffffffff8100c2ca>] child_rip+0xa/0x20
Apr 14 18:55:01 hostname kernel: [903597.050397]  [<ffffffff810a1820>] ? kthread+0x0/0xc0
Apr 14 18:55:01 hostname kernel: [903597.103548]  [<ffffffff8100c2c0>] ? child_rip+0x0/0x20
Apr 14 18:55:01 hostname kernel: [903597.155670]
Apr 14 18:55:02 hostname kernel: [903597.203132] Kernel panic - not syncing: LBUG
Apr 14 18:55:02 hostname kernel: [903597.252140] Pid: 8117, comm: ll_ost_io03_054 Tainted: P           -- ------------    2.6.32-573.7.1.1chaos.ch5.4.x86_64 #1
Apr 14 18:55:02 hostname kernel: [903597.309381] Call Trace:
Apr 14 18:55:02 hostname kernel: [903597.355368]  [<ffffffff8153ab99>] ? panic+0xa7/0x16f
Apr 14 18:55:02 hostname kernel: [903597.403529]  [<ffffffffa04dbf4b>] ? lbug_with_loc+0x9b/0xb0 [libcfs]
Apr 14 18:55:02 hostname kernel: [903597.452599]  [<ffffffffa1131a61>] ? ofd_grant_incoming+0x5d1/0x610 [ofd]
Apr 14 18:55:02 hostname kernel: [903597.500920]  [<ffffffffa1132403>] ? ofd_grant_prepare_write+0x2a3/0xeb0 [ofd]
Apr 14 18:55:02 hostname kernel: [903597.548722]  [<ffffffffa112aaff>] ? ofd_preprw+0x40f/0x1620 [ofd]
Apr 14 18:55:02 hostname kernel: [903597.594305]  [<ffffffffa0bc4c84>] ? sptlrpc_svc_alloc_rs+0x74/0x360 [ptlrpc]
Apr 14 18:55:02 hostname kernel: [903597.640262]  [<ffffffffa10bfe51>] ? obd_preprw+0x121/0x390 [ost]
Apr 14 18:55:02 hostname kernel: [903597.683942]  [<ffffffffa10c9c22>] ? ost_brw_write+0x982/0x15d0 [ost]
Apr 14 18:55:02 hostname kernel: [903597.727061]  [<ffffffffa0bd4a85>] ? null_authorize+0x75/0x100 [ptlrpc]
Apr 14 18:55:02 hostname kernel: [903597.769524]  [<ffffffffa0b9455e>] ? ptlrpc_send_reply+0x28e/0x7f0 [ptlrpc]
Apr 14 18:55:02 hostname kernel: [903597.811352]  [<ffffffff812925f9>] ? cpumask_next_and+0x29/0x50
Apr 14 18:55:02 hostname kernel: [903597.851040]  [<ffffffffa0b9c1cc>] ? lustre_msg_get_version+0x8c/0x100 [ptlrpc]
Apr 14 18:55:02 hostname kernel: [903597.891627]  [<ffffffffa0b9c328>] ? lustre_msg_check_version+0xe8/0x100 [ptlrpc]
Apr 14 18:55:02 hostname kernel: [903597.931382]  [<ffffffffa10d079c>] ? ost_handle+0x439c/0x44d0 [ost]
Apr 14 18:55:02 hostname kernel: [903597.969257]  [<ffffffffa0ba31fb>] ? ptlrpc_update_export_timer+0x4b/0x560 [ptlrpc]
Apr 14 18:55:02 hostname kernel: [903598.008756]  [<ffffffffa0baa2c5>] ? ptlrpc_server_handle_request+0x385/0xc00 [ptlrpc]
Apr 14 18:55:02 hostname kernel: [903598.048941]  [<ffffffffa04dc52e>] ? cfs_timer_arm+0xe/0x10 [libcfs]
Apr 14 18:55:02 hostname kernel: [903598.087582]  [<ffffffffa04ed845>] ? lc_watchdog_touch+0x65/0x170 [libcfs]
Apr 14 18:55:02 hostname kernel: [903598.126609]  [<ffffffffa0ba2cf9>] ? ptlrpc_wait_event+0xb9/0x2e0 [ptlrpc]
Apr 14 18:55:05 hostname kernel: [903598.165500]  [<ffffffff81059a49>] ? __wake_up_common+0x59/0x90
Apr 14 18:55:05 hostname kernel: [903598.203193]  [<ffffffffa0baca4d>] ? ptlrpc_main+0xaed/0x1930 [ptlrpc]
Apr 14 18:55:05 hostname kernel: [903598.241713]  [<ffffffffa0babf60>] ? ptlrpc_main+0x0/0x1930 [ptlrpc]
Apr 14 18:55:05 hostname kernel: [903598.279748]  [<ffffffff810a18be>] ? kthread+0x9e/0xc0
Apr 14 18:55:05 hostname kernel: [903598.316041]  [<ffffffff8100c2ca>] ? child_rip+0xa/0x20
Apr 14 18:55:05 hostname kernel: [903598.352138]  [<ffffffff810a1820>] ? kthread+0x0/0xc0
Apr 14 18:55:05 hostname kernel: [903598.387861]  [<ffffffff8100c2c0>] ? child_rip+0x0/0x20
Apr 14 18:55:05 hostname kernel: [903598.423647] LustreError: dumping log to /admin/lustre-logs/hostname.1460681702.8117
Apr 14 18:55:05 hostname kernel: [903598.462754] ------------[ cut here ]------------
Apr 14 18:55:05 hostname kernel: [903598.498514] WARNING: at kernel/softirq.c:159 local_bh_enable_ip+0x7d/0xb0() (Tainted: P           -- ------------   )
Apr 14 18:55:05 hostname kernel: [903598.542367] Hardware name: PowerEdge R720
Apr 14 18:55:05 hostname kernel: [903598.578505] Modules linked in: osp(U) ofd(U) lfsck(U) ost(U) mgc(U) osd_zfs(U) lquota(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) dm_round_robin zfs(P)(U) zcommon(P)(U) znvpair(P)(U) spl(U) zlib_deflate zavl(P)(U) zunicode(P)(U) scsi_dh_rdac sg sd_mod crc_t10dif ib_srp scsi_transport_srp scsi_tgt ksocklnd(U) ko2iblnd(U) lnet(U) sha512_generic crc32c_intel libcfs(U) ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm dm_mirror dm_region_hash dm_log dm_multipath dm_mod vhost_net macvtap macvlan tun kvm_intel kvm uinput ipmi_devintf joydev microcode wmi power_meter acpi_ipmi ipmi_si ipmi_msghandler iTCO_wdt iTCO_vendor_support ixgbe(U) mlx4_ib ib_sa ib_mad ib_core ib_addr ipv6 mlx4_en mlx4_core dcdbas ahci sb_edac edac_core lpc_ich mfd_core shpchp nfs nfs_acl auth_rpcgss fscache lockd sunrpc e1000 broadcom tg3 igb ptp pps_core i2c_algo_bit i2c_core dca bnx2 [last unloaded: scsi_tgt]
Apr 14 18:55:05 hostname kernel: [903598.964165] Pid: 8117, comm: ll_ost_io03_054 Tainted: P           -- ------------    2.6.32-573.7.1.1chaos.ch5.4.x86_64 #1
Apr 14 18:55:05 hostname kernel: [903599.017612] Call Trace:
Apr 14 18:55:05 hostname kernel: [903599.060857]  [<ffffffff810776c1>] ? warn_slowpath_common+0x91/0xe0
Apr 14 18:55:05 hostname kernel: [903599.109199]  [<ffffffffa00ce4a0>] ? call_reserve+0x0/0x60 [sunrpc]
Apr 14 18:55:05 hostname kernel: [903599.157780]  [<ffffffff8107772a>] ? warn_slowpath_null+0x1a/0x20
Apr 14 18:55:05 hostname kernel: [903599.206335]  [<ffffffff8107fe4d>] ? local_bh_enable_ip+0x7d/0xb0
Apr 14 18:55:05 hostname kernel: [903599.254720]  [<ffffffff8153e9db>] ? _spin_unlock_bh+0x1b/0x20
Apr 14 18:55:05 hostname kernel: [903599.302827]  [<ffffffffa00d336d>] ? xprt_lock_and_alloc_slot+0x4d/0x90 [sunrpc]
Apr 14 18:55:05 hostname kernel: [903599.352394]  [<ffffffffa00d1bf3>] ? xprt_reserve+0x73/0xd0 [sunrpc]
Apr 14 18:55:05 hostname kernel: [903599.399968]  [<ffffffffa00ce4d4>] ? call_reserve+0x34/0x60 [sunrpc]
Apr 14 18:55:05 hostname kernel: [903599.447295]  [<ffffffffa00d98e7>] ? __rpc_execute+0x77/0x350 [sunrpc]
Apr 14 18:55:05 hostname kernel: [903599.494724]  [<ffffffff810a1bd7>] ? bit_waitqueue+0x17/0xd0
Apr 14 18:55:05 hostname kernel: [903599.540954]  [<ffffffffa00d9c21>] ? rpc_execute+0x61/0xa0 [sunrpc]
Apr 14 18:55:05 hostname kernel: [903599.587941]  [<ffffffffa00d0465>] ? rpc_run_task+0x75/0x90 [sunrpc]
Apr 14 18:55:05 hostname kernel: [903599.634919]  [<ffffffffa00d0582>] ? rpc_call_sync+0x42/0x70 [sunrpc]
Apr 14 18:55:05 hostname kernel: [903599.681819]  [<ffffffffa018801a>] ? nfs3_rpc_call.clone.0+0x9a/0x100 [nfs]
Apr 14 18:55:05 hostname kernel: [903599.729276]  [<ffffffffa018829a>] ? nfs3_proc_access+0xaa/0x180 [nfs]
Apr 14 18:55:05 hostname kernel: [903599.776184]  [<ffffffffa0170938>] ? nfs_do_access+0x198/0x260 [nfs]
Apr 14 18:55:05 hostname kernel: [903599.822818]  [<ffffffffa0170ad7>] ? nfs_permission+0xd7/0x220 [nfs]
Apr 14 18:55:05 hostname kernel: [903599.869205]  [<ffffffff811a2f00>] ? __link_path_walk+0x90/0x850
Apr 14 18:55:05 hostname kernel: [903599.915060]  [<ffffffff811a3dba>] ? path_walk+0x6a/0xe0
Apr 14 18:55:05 hostname kernel: [903599.959820]  [<ffffffff811a5c1a>] ? do_filp_open+0x1fa/0xd20
Apr 14 18:55:05 hostname kernel: [903600.005019]  [<ffffffff8134e610>] ? vt_console_print+0x260/0x330
Apr 14 18:55:05 hostname kernel: [903600.050460]  [<ffffffff810a7ecf>] ? up+0x2f/0x50
Apr 14 18:55:05 hostname kernel: [903600.093890]  [<ffffffff810778ba>] ? _call_console_drivers+0x4a/0x80
Apr 14 18:55:05 hostname kernel: [903600.139273]  [<ffffffff81077fcf>] ? release_console_sem+0x1cf/0x220
Apr 14 18:55:05 hostname kernel: [903600.184397]  [<ffffffff811a6780>] ? filp_open+0x40/0x50
Apr 14 18:55:05 hostname kernel: [903600.228166]  [<ffffffffa04ec700>] ? cfs_tracefile_dump_all_pages+0x30/0x2e0 [libcfs]
Apr 14 18:55:05 hostname kernel: [903600.275396]  [<ffffffff8153aca2>] ? printk+0x41/0x47
Apr 14 18:55:05 hostname kernel: [903600.319279]  [<ffffffffa04e6725>] ? libcfs_debug_dumplog_internal+0xb5/0xd0 [libcfs]
Apr 14 18:55:05 hostname kernel: [903600.366905]  [<ffffffffa04db7a1>] ? panic_notifier+0x71/0x90 [libcfs]
Apr 14 18:55:05 hostname kernel: [903600.413268]  [<ffffffff81541a15>] ? notifier_call_chain+0x55/0x80
Apr 14 18:55:05 hostname kernel: [903600.459263]  [<ffffffff81541a7a>] ? atomic_notifier_call_chain+0x1a/0x20
Apr 14 18:55:05 hostname kernel: [903600.506204]  [<ffffffff8153abc4>] ? panic+0xd2/0x16f
Apr 14 18:55:05 hostname kernel: [903600.551205]  [<ffffffffa04dbf4b>] ? lbug_with_loc+0x9b/0xb0 [libcfs]
Apr 14 18:55:05 hostname kernel: [903600.598199]  [<ffffffffa1131a61>] ? ofd_grant_incoming+0x5d1/0x610 [ofd]
Apr 14 18:55:05 hostname kernel: [903600.645207]  [<ffffffffa1132403>] ? ofd_grant_prepare_write+0x2a3/0xeb0 [ofd]
Apr 14 18:55:05 hostname kernel: [903600.692332]  [<ffffffffa112aaff>] ? ofd_preprw+0x40f/0x1620 [ofd]
Apr 14 18:55:05 hostname kernel: [903600.737554]  [<ffffffffa0bc4c84>] ? sptlrpc_svc_alloc_rs+0x74/0x360 [ptlrpc]
Apr 14 18:55:05 hostname kernel: [903600.783415]  [<ffffffffa10bfe51>] ? obd_preprw+0x121/0x390 [ost]
Apr 14 18:55:05 hostname kernel: [903600.827220]  [<ffffffffa10c9c22>] ? ost_brw_write+0x982/0x15d0 [ost]
Apr 14 18:55:05 hostname kernel: [903600.870484]  [<ffffffffa0bd4a85>] ? null_authorize+0x75/0x100 [ptlrpc]
Apr 14 18:55:05 hostname kernel: [903600.912874]  [<ffffffffa0b9455e>] ? ptlrpc_send_reply+0x28e/0x7f0 [ptlrpc]
Apr 14 18:55:05 hostname kernel: [903600.954608]  [<ffffffff812925f9>] ? cpumask_next_and+0x29/0x50
Apr 14 18:55:05 hostname kernel: [903600.994082]  [<ffffffffa0b9c1cc>] ? lustre_msg_get_version+0x8c/0x100 [ptlrpc]
Apr 14 18:55:05 hostname kernel: [903601.034357]  [<ffffffffa0b9c328>] ? lustre_msg_check_version+0xe8/0x100 [ptlrpc]
Apr 14 18:55:05 hostname kernel: [903601.073790]  [<ffffffffa10d079c>] ? ost_handle+0x439c/0x44d0 [ost]
Apr 14 18:55:05 hostname kernel: [903601.110926]  [<ffffffffa0ba31fb>] ? ptlrpc_update_export_timer+0x4b/0x560 [ptlrpc]
Apr 14 18:55:05 hostname kernel: [903601.149161]  [<ffffffffa0baa2c5>] ? ptlrpc_server_handle_request+0x385/0xc00 [ptlrpc]
Apr 14 18:55:05 hostname kernel: [903601.187091]  [<ffffffffa04dc52e>] ? cfs_timer_arm+0xe/0x10 [libcfs]
Apr 14 18:55:05 hostname kernel: [903601.223052]  [<ffffffffa04ed845>] ? lc_watchdog_touch+0x65/0x170 [libcfs]
Apr 14 18:55:05 hostname kernel: [903601.259745]  [<ffffffffa0ba2cf9>] ? ptlrpc_wait_event+0xb9/0x2e0 [ptlrpc]
Apr 14 18:55:05 hostname kernel: [903601.296392]  [<ffffffff81059a49>] ? __wake_up_common+0x59/0x90
Apr 14 18:55:05 hostname kernel: [903601.332041]  [<ffffffffa0baca4d>] ? ptlrpc_main+0xaed/0x1930 [ptlrpc]
Apr 14 18:55:05 hostname kernel: [903601.368544]  [<ffffffffa0babf60>] ? ptlrpc_main+0x0/0x1930 [ptlrpc]
Apr 14 18:55:05 hostname kernel: [903601.404434]  [<ffffffff810a18be>] ? kthread+0x9e/0xc0
Apr 14 18:55:05 hostname kernel: [903601.438789]  [<ffffffff8100c2ca>] ? child_rip+0xa/0x20
Apr 14 18:55:05 hostname kernel: [903601.473315]  [<ffffffff810a1820>] ? kthread+0x0/0xc0
Apr 14 18:55:05 hostname kernel: [903601.507420]  [<ffffffff8100c2c0>] ? child_rip+0x0/0x20
Apr 14 18:55:05 hostname kernel: [903601.541635] ---[ end trace 2f456923f2cd5a40 ]---