Nov 15 23:48:09 <user.notice> n6-1-21.local 5645:0:(client.c:2089:__ptlrpc_free_req()) LBUG

Nov 15 23:48:09 <user.notice> n6-1-21.local Pid[]: 5645, comm: lcg-cp
Nov 15 23:48:09 <user.notice> n6-1-21.local []: 
Nov 15 23:48:09 <user.notice> n6-1-21.local Call[]: Trace:
Nov 15 23:48:09 <user.notice> n6-1-21.local [<ffffffff883415f1>]: libcfs_debug_dumpstack+0x51/0x60 [libcfs]
Nov 15 23:48:09 <user.notice> n6-1-21.local [<ffffffff88341b2a>]: lbug_with_loc+0x7a/0xd0 [libcfs]
Nov 15 23:48:09 <user.notice> n6-1-21.local [<ffffffff88a1b32c>]: __ptlrpc_req_finished+0x44c/0x850 [ptlrpc]
Nov 15 23:48:09 <user.notice> n6-1-21.local [<ffffffff88b5dbbd>]: mdc_enqueue+0xd3d/0x1100 [mdc]
Nov 15 23:48:09 <user.notice> n6-1-21.local [<ffffffff88b3694d>]: seq_client_alloc_fid+0xcd/0x810 [fid]
Nov 15 23:48:09 <user.notice> n6-1-21.local [<ffffffff88b5f02c>]: mdc_intent_lock+0x46c/0x610 [mdc]
Nov 15 23:48:09 <user.notice> n6-1-21.local [<ffffffff88d1fcba>]: __lmv_fid_alloc+0x44a/0x5b0 [lmv]
Nov 15 23:48:09 <user.notice> n6-1-21.local [<ffffffff80009001>]: __handle_mm_fault+0x892/0x103b
Nov 15 23:48:09 <user.notice> n6-1-21.local [<ffffffff88cad820>]: ll_md_blocking_ast+0x0/0x580 [lustre]
Nov 15 23:48:09 <user.notice> n6-1-21.local [<ffffffff88a04cd0>]: ldlm_completion_ast+0x0/0x6a0 [ptlrpc]
Nov 15 23:48:09 <user.notice> n6-1-21.local [<ffffffff88d3f687>]: lmv_object_find+0x77/0xf0 [lmv]
Nov 15 23:48:09 <user.notice> n6-1-21.local [<ffffffff88d3e4c6>]: lmv_intent_open+0x9c6/0x12c0 [lmv]
Nov 15 23:48:09 <user.notice> n6-1-21.local [<ffffffff88cad820>]: ll_md_blocking_ast+0x0/0x580 [lustre]
Nov 15 23:48:09 <user.notice> n6-1-21.local [<ffffffff8000d5a8>]: dput+0x2c/0x114
Nov 15 23:48:09 <user.notice> n6-1-21.local [<ffffffff8899133b>]: nfs_lookup_revalidate+0x2c0/0x445 [nfs]
Nov 15 23:48:09 <user.notice> n6-1-21.local [<ffffffff88d3f084>]: lmv_intent_lock+0x2c4/0x390 [lmv]
Nov 15 23:48:09 <user.notice> n6-1-21.local [<ffffffff88cad820>]: ll_md_blocking_ast+0x0/0x580 [lustre]
Nov 15 23:48:09 <user.notice> n6-1-21.local [<ffffffff88c8a5d5>]: ll_prep_md_op_data+0x385/0x460 [lustre]
Nov 15 23:48:09 <user.notice> n6-1-21.local [<ffffffff8000985a>]: __d_lookup+0xb0/0xff
Nov 15 23:48:09 <user.notice> n6-1-21.local [<ffffffff88cad078>]: ll_lookup_it+0x8c8/0xc90 [lustre]
Nov 15 23:48:09 <user.notice> n6-1-21.local [<ffffffff88cad820>]: ll_md_blocking_ast+0x0/0x580 [lustre]
Nov 15 23:48:09 <user.notice> n6-1-21.local [<ffffffff8008dff0>]: dequeue_task+0x18/0x37
Nov 15 23:48:09 <user.notice> n6-1-21.local [<ffffffff88cae1a6>]: ll_lookup_nd+0x1e6/0x400 [lustre]
Nov 15 23:48:09 <user.notice> n6-1-21.local [<ffffffff80022abd>]: d_alloc+0x176/0x1ab
Nov 15 23:48:09 <user.notice> n6-1-21.local [<ffffffff80036cb4>]: __lookup_hash+0x10b/0x12f
Nov 15 23:48:09 <user.notice> n6-1-21.local [<ffffffff8001b255>]: open_namei+0xf2/0x718
Nov 15 23:48:09 <user.notice> n6-1-21.local [<ffffffff8002771b>]: do_filp_open+0x1c/0x38
Nov 15 23:48:09 <user.notice> n6-1-21.local [<ffffffff8001a089>]: do_sys_open+0x44/0xbe
Nov 15 23:48:09 <user.notice> n6-1-21.local [<ffffffff8005d28d>]: tracesys+0xd5/0xe0
Nov 15 23:48:09 <user.notice> n6-1-21.local []: 
Nov 15 23:48:12 <user.notice> n6-1-21.local Kernel[]: panic - not syncing: LBUG
Nov 15 23:48:12 <user.notice> n6-1-21.local []: 
Nov 15 23:48:12 <kern.emerg> n6-1-21.local kernel[]: Dumping qib trace buffer from panic
Nov 15 23:48:12 <user.notice> n6-1-21.local Done[]: dumping qib trace buffer





Nov 16 08:34:36 <user.notice> n16-1-2.local 3311:0:(client.c:2089:__ptlrpc_free_req()) LBUG
Nov 16 08:34:36 <user.notice> n16-1-2.local Pid[]: 3311, comm: lcg-ls
Nov 16 08:34:36 <user.notice> n16-1-2.local []: 
Nov 16 08:34:36 <user.notice> n16-1-2.local Call[]: Trace:
Nov 16 08:34:36 <user.notice> n16-1-2.local [<ffffffff888495f1>]: libcfs_debug_dumpstack+0x51/0x60 [libcfs]
Nov 16 08:34:36 <user.notice> n16-1-2.local [<ffffffff88849b2a>]: lbug_with_loc+0x7a/0xd0 [libcfs]
Nov 16 08:34:36 <user.notice> n16-1-2.local [<ffffffff889f132c>]: __ptlrpc_req_finished+0x44c/0x850 [ptlrpc]
Nov 16 08:34:36 <user.notice> n16-1-2.local [<ffffffff88b64bbd>]: mdc_enqueue+0xd3d/0x1100 [mdc]
Nov 16 08:34:36 <user.notice> n16-1-2.local [<ffffffff88941b11>]: cl_lock_counters+0x21/0x70 [obdclass]
Nov 16 08:34:36 <user.notice> n16-1-2.local [<ffffffff88b3d94d>]: seq_client_alloc_fid+0xcd/0x810 [fid]
Nov 16 08:34:36 <user.notice> n16-1-2.local [<ffffffff88b6602c>]: mdc_intent_lock+0x46c/0x610 [mdc]
Nov 16 08:34:36 <user.notice> n16-1-2.local [<ffffffff88d31cba>]: __lmv_fid_alloc+0x44a/0x5b0 [lmv]
Nov 16 08:34:36 <user.notice> n16-1-2.local [<ffffffff88cb4820>]: ll_md_blocking_ast+0x0/0x580 [lustre]
Nov 16 08:34:36 <user.notice> n16-1-2.local [<ffffffff889dacd0>]: ldlm_completion_ast+0x0/0x6a0 [ptlrpc]
Nov 16 08:34:36 <user.notice> n16-1-2.local [<ffffffff88d51687>]: lmv_object_find+0x77/0xf0 [lmv]
Nov 16 08:34:36 <user.notice> n16-1-2.local [<ffffffff88d504c6>]: lmv_intent_open+0x9c6/0x12c0 [lmv]
Nov 16 08:34:36 <user.notice> n16-1-2.local [<ffffffff88cb4820>]: ll_md_blocking_ast+0x0/0x580 [lustre]
Nov 16 08:34:36 <user.notice> n16-1-2.local [<ffffffff8894485e>]: cl_lock_disclosure+0x8e/0xf0 [obdclass]
Nov 16 08:34:36 <user.notice> n16-1-2.local [<ffffffff8002dfd7>]: __wake_up+0x38/0x4f
Nov 16 08:34:36 <user.notice> n16-1-2.local [<ffffffff889417ed>]: cl_lock_hold_mod+0x2d/0x80 [obdclass]
Nov 16 08:34:36 <user.notice> n16-1-2.local [<ffffffff88d51084>]: lmv_intent_lock+0x2c4/0x390 [lmv]
Nov 16 08:34:36 <user.notice> n16-1-2.local [<ffffffff88cb4820>]: ll_md_blocking_ast+0x0/0x580 [lustre]
Nov 16 08:34:36 <user.notice> n16-1-2.local [<ffffffff88c915d5>]: ll_prep_md_op_data+0x385/0x460 [lustre]
Nov 16 08:34:36 <user.notice> n16-1-2.local [<ffffffff88cb4078>]: ll_lookup_it+0x8c8/0xc90 [lustre]
Nov 16 08:34:36 <user.notice> n16-1-2.local [<ffffffff88cb4820>]: ll_md_blocking_ast+0x0/0x580 [lustre]
Nov 16 08:34:36 <user.notice> n16-1-2.local [<ffffffff8008dff0>]: dequeue_task+0x18/0x37
Nov 16 08:34:36 <user.notice> n16-1-2.local [<ffffffff88cb51a6>]: ll_lookup_nd+0x1e6/0x400 [lustre]
Nov 16 08:34:36 <user.notice> n16-1-2.local [<ffffffff80022abd>]: d_alloc+0x176/0x1ab
Nov 16 08:34:36 <user.notice> n16-1-2.local [<ffffffff80036cb4>]: __lookup_hash+0x10b/0x12f
Nov 16 08:34:36 <user.notice> n16-1-2.local [<ffffffff8001b255>]: open_namei+0xf2/0x718
Nov 16 08:34:36 <user.notice> n16-1-2.local [<ffffffff8002771b>]: do_filp_open+0x1c/0x38
Nov 16 08:34:36 <user.notice> n16-1-2.local [<ffffffff8001a089>]: do_sys_open+0x44/0xbe
Nov 16 08:34:36 <user.notice> n16-1-2.local [<ffffffff8005d28d>]: tracesys+0xd5/0xe0
Nov 16 08:34:36 <user.notice> n16-1-2.local []: 
Nov 16 08:34:36 <user.notice> n16-1-2.local Kernel[]: panic - not syncing: LBUG
Nov 16 08:34:36 <user.notice> n16-1-2.local []: 
Nov 16 08:34:36 <kern.emerg> n16-1-2.local kernel[]: Dumping qib trace buffer from panic
Nov 16 08:34:36 <user.notice> n16-1-2.local Done[]: dumping qib trace buffer


Nov 16 09:12:04 <user.notice> n8-4-13.local 22471:0:(client.c:2089:__ptlrpc_free_req()) LBUG
Nov 16 09:12:04 <user.notice> n8-4-13.local Pid[]: 22471, comm: python
Nov 16 09:12:04 <user.notice> n8-4-13.local []: 
Nov 16 09:12:04 <user.notice> n8-4-13.local Call[]: Trace:
Nov 16 09:12:04 <user.notice> n8-4-13.local [<ffffffff883435f1>]: libcfs_debug_dumpstack+0x51/0x60 [libcfs]
Nov 16 09:12:04 <user.notice> n8-4-13.local [<ffffffff88343b2a>]: lbug_with_loc+0x7a/0xd0 [libcfs]
Nov 16 09:12:04 <user.notice> n8-4-13.local [<ffffffff88a1d32c>]: __ptlrpc_req_finished+0x44c/0x850 [ptlrpc]
Nov 16 09:12:04 <user.notice> n8-4-13.local [<ffffffff88b5fbbd>]: mdc_enqueue+0xd3d/0x1100 [mdc]
Nov 16 09:12:04 <user.notice> n8-4-13.local [<ffffffff889ecb1b>]: ldlm_res_hop_fid_hash+0x1cb/0x1f0 [ptlrpc]
Nov 16 09:12:04 <user.notice> n8-4-13.local [<ffffffff88351463>]: cfs_hash_bd_lookup_intent+0x83/0x130 [libcfs]
Nov 16 09:12:04 <user.notice> n8-4-13.local [<ffffffff889ecb1b>]: ldlm_res_hop_fid_hash+0x1cb/0x1f0 [ptlrpc]
Nov 16 09:12:04 <user.notice> n8-4-13.local [<ffffffff88b6102c>]: mdc_intent_lock+0x46c/0x610 [mdc]
Nov 16 09:12:04 <user.notice> n8-4-13.local [<ffffffff889ec5e9>]: ldlm_lock_match+0x729/0x870 [ptlrpc]
Nov 16 09:12:04 <user.notice> n8-4-13.local [<ffffffff88caf820>]: ll_md_blocking_ast+0x0/0x580 [lustre]
Nov 16 09:12:04 <user.notice> n8-4-13.local [<ffffffff88a06cd0>]: ldlm_completion_ast+0x0/0x6a0 [ptlrpc]
Nov 16 09:12:04 <user.notice> n8-4-13.local [<ffffffff88d41687>]: lmv_object_find+0x77/0xf0 [lmv]
Nov 16 09:12:04 <user.notice> n8-4-13.local [<ffffffff88d404c6>]: lmv_intent_open+0x9c6/0x12c0 [lmv]
Nov 16 09:12:04 <user.notice> n8-4-13.local [<ffffffff88caf820>]: ll_md_blocking_ast+0x0/0x580 [lustre]
Nov 16 09:12:04 <user.notice> n8-4-13.local [<ffffffff88447b11>]: cl_lock_counters+0x21/0x70 [obdclass]
Nov 16 09:12:04 <user.notice> n8-4-13.local [<ffffffff88d2b60f>]: lmv_lock_match+0x44f/0x5e0 [lmv]
Nov 16 09:12:04 <user.notice> n8-4-13.local [<ffffffff88d41084>]: lmv_intent_lock+0x2c4/0x390 [lmv]
Nov 16 09:12:04 <user.notice> n8-4-13.local [<ffffffff88caf820>]: ll_md_blocking_ast+0x0/0x580 [lustre]
Nov 16 09:12:04 <user.notice> n8-4-13.local [<ffffffff88c8c5d5>]: ll_prep_md_op_data+0x385/0x460 [lustre]
Nov 16 09:12:04 <user.notice> n8-4-13.local [<ffffffff889ecb1b>]: ldlm_res_hop_fid_hash+0x1cb/0x1f0 [ptlrpc]
Nov 16 09:12:04 <user.notice> n8-4-13.local [<ffffffff88c6738a>]: ll_revalidate_it+0xb6a/0x19c0 [lustre]
Nov 16 09:12:04 <user.notice> n8-4-13.local [<ffffffff88caf820>]: ll_md_blocking_ast+0x0/0x580 [lustre]
Nov 16 09:12:04 <user.notice> n8-4-13.local [<ffffffff8835167c>]: cfs_hash_bd_from_key+0x3c/0xc0 [libcfs]
Nov 16 09:12:04 <user.notice> n8-4-13.local [<ffffffff889ed87e>]: ldlm_resource_putref+0x12e/0x240 [ptlrpc]
Nov 16 09:12:04 <user.notice> n8-4-13.local [<ffffffff88cadb11>]: ll_convert_intent+0x121/0x1f0 [lustre]
Nov 16 09:12:04 <user.notice> n8-4-13.local [<ffffffff88c692f4>]: ll_revalidate_nd+0x194/0x39c [lustre]
Nov 16 09:12:04 <user.notice> n8-4-13.local [<ffffffff80009842>]: __d_lookup+0x98/0xff
Nov 16 09:12:04 <user.notice> n8-4-13.local [<ffffffff8000d1ab>]: do_lookup+0x18f/0x1e6
Nov 16 09:12:04 <user.notice> n8-4-13.local [<ffffffff8000a2e3>]: __link_path_walk+0xa3a/0xfd1
Nov 16 09:12:04 <user.notice> n8-4-13.local [<ffffffff8000eb88>]: link_path_walk+0x45/0xb8
Nov 16 09:12:04 <user.notice> n8-4-13.local [<ffffffff8000ce9c>]: do_path_lookup+0x294/0x310
Nov 16 09:12:04 <user.notice> n8-4-13.local [<ffffffff80023959>]: __path_lookup_intent_open+0x56/0x97
Nov 16 09:12:04 <user.notice> n8-4-13.local [<ffffffff8001b1d6>]: open_namei+0x73/0x718
Nov 16 09:12:04 <user.notice> n8-4-13.local [<ffffffff80067225>]: do_page_fault+0x4cc/0x842
Nov 16 09:12:04 <user.notice> n8-4-13.local [<ffffffff8002771b>]: do_filp_open+0x1c/0x38
Nov 16 09:12:04 <user.notice> n8-4-13.local [<ffffffff8001a089>]: do_sys_open+0x44/0xbe
Nov 16 09:12:04 <user.notice> n8-4-13.local [<ffffffff8005d28d>]: tracesys+0xd5/0xe0
Nov 16 09:12:04 <user.notice> n8-4-13.local []: 
Nov 16 09:12:04 <user.notice> n8-4-13.local Kernel[]: panic - not syncing: LBUG
Nov 16 09:12:04 <user.notice> n8-4-13.local []: 
Nov 16 09:12:04 <kern.emerg> n8-4-13.local kernel[]: Dumping qib trace buffer from panic
Nov 16 09:12:04 <user.notice> n8-4-13.local Done[]: dumping qib trace buffer