[LU-6799] getxattr failed: -2 triggers a Kernel BUG on MDS

Duplicate — Mon, 6 Jul 2015 17:46:39 +0000

The LustreError: (mdt_xattr.c:131:mdt_getxattr_one()) getxattr failed: -2 triggers the following kernel BUG on the MDS of one of our filesystem.

2015-06-30 13:40:01
2015-06-30 13:45:01 Lustre: DEBUG MARKER: Tue Jun 30 13:45:01 2015
2015-06-30 13:45:01
2015-06-30 13:50:01 Lustre: DEBUG MARKER: Tue Jun 30 13:50:01 2015
2015-06-30 13:50:01
2015-06-30 13:50:32 LustreError: 20168:0:(mdt_xattr.c:131:mdt_getxattr_one()) getxattr failed: -2
2015-06-30 13:50:32 BUG: unable to handle kernel NULL pointer dereference at (null)
2015-06-30 13:50:32 IP: [<ffffffff8129c452>] sg_next+0x2/0x30
2015-06-30 13:50:32 PGD 0
2015-06-30 13:50:32 Oops: 0000 [#1] SMP
2015-06-30 13:50:32 last sysfs file: /sys/devices/pci0000:80/0000:80:05.0/0000:85:00.0/host8/rport-8:0-0/target8:0:0/8:0:0:0/state
2015-06-30 13:50:32 CPU 23
2015-06-30 13:50:32 Modules linked in: osp(U) mdd(U) lfsck(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) lustre(U) lov(U) osc(U) mdc(U) lquota(U) fid(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) sha512_generic crc32c_intel libcfs(U) nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf bonding 8021q garp stp llc rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U) ib_cm(U) ipv6 ib_uverbs(U) ib_umad(U) mlx4_ib(U) ib_sa(U) ib_mad(U) ib_core(U) mlx4_core(U) dm_round_robin scsi_dh_rdac dm_multipath mic(U) uinput ipmi_devintf ipmi_si ipmi_msghandler sg lpc_ich mfd_core ioatdma lpfc scsi_transport_fc scsi_tgt igb dca i2c_algo_bit i2c_core ptp pps_core compat(U) ext4 jbd2 mbcache sd_mod crc_t10dif ahci dm_mirror dm_region_hash dm_log dm_mod megaraid_sas [last unloaded: scsi_wait_scan]
2015-06-30 13:50:32
2015-06-30 13:50:32 Pid: 20168, comm: mdt03_049 Not tainted 2.6.32-504.16.2.el6.Bull.74.x86_64 #1 BULL bullx super-node
2015-06-30 13:50:32 RIP: 0010:[<ffffffff8129c452>]  [<ffffffff8129c452>] sg_next+0x2/0x30
2015-06-30 13:50:32 RSP: 0018:ffff880847f518c8  EFLAGS: 00010246
2015-06-30 13:50:32 RAX: 0000000000000000 RBX: ffff88046e524000 RCX: 0000000000000000
2015-06-30 13:50:32 RDX: 0000000000000101 RSI: ffffc900175fb240 RDI: 0000000000000000
2015-06-30 13:50:32 RBP: ffff880847f51940 R08: ffffea003958cb98 R09: 0000000000000301
2015-06-30 13:50:32 R10: 0000000000001000 R11: 0000000000000000 R12: ffff880c78a28940
2015-06-30 13:50:32 R13: ffff88046e536000 R14: ffffc900175fb240 R15: ffff880c7980c090
2015-06-30 13:50:32 FS:  0000000000000000(0000) GS:ffff880c8e540000(0000) knlGS:0000000000000000
2015-06-30 13:50:32 CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
2015-06-30 13:50:32 CR2: 0000000000000000 CR3: 0000000001a85000 CR4: 00000000000007e0
2015-06-30 13:50:32 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
2015-06-30 13:50:32 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
2015-06-30 13:50:32 Process mdt03_049 (pid: 20168, threadinfo ffff880847f50000, task ffff88084585aab0)
2015-06-30 13:50:32 Stack:
2015-06-30 13:50:32  ffffffffa09cb46e ffff880847f51fd8 ffff88084585aab0 ffffffff00000101
2015-06-30 13:50:32 <d> ffffffff81a9ac80 ffff880c78a289c0 0000030100000001 ffff880847f51920
2015-06-30 13:50:32 <d> ffffffff8105e100 ffff880876a8d558 ffff880c796ffb80 ffffc900175fb240
2015-06-30 13:50:32 Call Trace:
2015-06-30 13:50:32  [<ffffffffa09cb46e>] ? kiblnd_map_tx+0x19e/0x540 [ko2iblnd]
2015-06-30 13:50:32  [<ffffffff8105e100>] ? __dequeue_entity+0x30/0x50
2015-06-30 13:50:32  [<ffffffffa09cbe0a>] kiblnd_setup_rd_iov+0x13a/0x2b0 [ko2iblnd]
2015-06-30 13:50:32  [<ffffffffa09d151a>] kiblnd_send+0x5da/0x9b0 [ko2iblnd]
2015-06-30 13:50:32  [<ffffffffa05e6d6b>] lnet_ni_send+0x4b/0xf0 [lnet]
2015-06-30 13:50:32  [<ffffffffa05eafa5>] lnet_send+0x655/0xb80 [lnet]
2015-06-30 13:50:32  [<ffffffffa05ec00a>] LNetPut+0x31a/0x860 [lnet]
2015-06-30 13:50:32  [<ffffffffa07fbc40>] ptl_send_buf+0x1e0/0x550 [ptlrpc]
2015-06-30 13:50:32  [<ffffffffa081b8b8>] ? at_measured+0x108/0x380 [ptlrpc]
2015-06-30 13:50:32  [<ffffffffa083c445>] ? null_authorize+0x75/0x100 [ptlrpc]
2015-06-30 13:50:32  [<ffffffffa07fc22b>] ptlrpc_send_reply+0x27b/0x7f0 [ptlrpc]
2015-06-30 13:50:32  [<ffffffffa07c7054>] target_send_reply_msg+0x54/0x190 [ptlrpc]
2015-06-30 13:50:32  [<ffffffffa07c7576>] target_send_reply+0x3e6/0x720 [ptlrpc]
2015-06-30 13:50:32  [<ffffffffa0ea2df9>] mdt_handle_common+0x5d9/0x1470 [mdt]
2015-06-30 13:50:32  [<ffffffffa0edf645>] mds_regular_handle+0x15/0x20 [mdt]
2015-06-30 13:50:32  [<ffffffffa0811ee5>] ptlrpc_server_handle_request+0x385/0xc00 [ptlrpc]
2015-06-30 13:50:32  [<ffffffffa053e4ce>] ? cfs_timer_arm+0xe/0x10 [libcfs]
2015-06-30 13:50:32  [<ffffffffa054f7d5>] ? lc_watchdog_touch+0x65/0x170 [libcfs]
2015-06-30 13:50:32  [<ffffffffa080a919>] ? ptlrpc_wait_event+0xa9/0x2d0 [ptlrpc]
2015-06-30 13:50:32  [<ffffffff81057819>] ? __wake_up_common+0x59/0x90
2015-06-30 13:50:32  [<ffffffffa081466d>] ptlrpc_main+0xaed/0x1770 [ptlrpc]
2015-06-30 13:50:32  [<ffffffffa0813b80>] ? ptlrpc_main+0x0/0x1770 [ptlrpc]
2015-06-30 13:50:32  [<ffffffff8109e71e>] kthread+0x9e/0xc0
2015-06-30 13:50:32  [<ffffffff8100c20a>] child_rip+0xa/0x20
2015-06-30 13:50:32  [<ffffffff8109e680>] ? kthread+0x0/0xc0
2015-06-30 13:50:32  [<ffffffff8100c200>] ? child_rip+0x0/0x20
2015-06-30 13:50:32 Code: 5c 41 5d 41 5e 41 5f c9 c3 55 48 c7 c2 30 cc 29 81 be 80 00 00 00 48 89 e5 e8 6b ff ff ff c9 c3 66 0f 1f 84 00 00 00 00 00 31 c0 <f6> 07 02 55 48 89 e5 75 0d 48 8b 57 20 48 8d 47 20 f6 c2 01 75
2015-06-30 13:50:32 RIP  [<ffffffff8129c452>] sg_next+0x2/0x30
2015-06-30 13:50:32  RSP <ffff880847f518c8>
2015-06-30 13:50:32 CR2: 0000000000000000

We hit this issue on our new filesystem only. This is a dedicated filesystem for our Lustre 2.7 clients/routers. We already had 4 occurrences since the beginning of this week.

This LustreError is also reported in the console of the MDS of a second filesystem, but there is no kernel BUG. The main difference is that the clients/routers of this second FS are running the same software stack than the servers (RHEL6.6/Lustre 2.5.3.90/OFED3.12).

Here are some traces from the crash (bt/bt -f):

      KERNEL: /usr/lib/debug/lib/modules/2.6.32-504.16.2.el6.Bull.74.x86_64/vmlinux
    DUMPFILE: vmcore  [PARTIAL DUMP]
        CPUS: 32
        DATE: Tue Jun 30 13:50:31 2015
      UPTIME: 19:57:11
LOAD AVERAGE: 0.10, 0.06, 0.10
       TASKS: 1685
    NODENAME: mds2
     RELEASE: 2.6.32-504.16.2.el6.Bull.74.x86_64
     VERSION: #1 SMP Tue Apr 28 01:43:42 CEST 2015
     MACHINE: x86_64  (2266 Mhz)
      MEMORY: 64 GB
       PANIC: "Oops: 0000 [#1] SMP " (check log for details)
         PID: 20168
     COMMAND: "mdt03_049"
        TASK: ffff88084585aab0  [THREAD_INFO: ffff880847f50000]
         CPU: 23
       STATE: TASK_RUNNING (PANIC)

crash> bt
PID: 20168  TASK: ffff88084585aab0  CPU: 23  COMMAND: "mdt03_049"
 #0 [ffff880847f514b0] machine_kexec at ffffffff8103b71b
 #1 [ffff880847f51510] crash_kexec at ffffffff810c9942
 #2 [ffff880847f515e0] oops_end at ffffffff8152f070
 #3 [ffff880847f51610] no_context at ffffffff8104c80b
 #4 [ffff880847f51660] __bad_area_nosemaphore at ffffffff8104ca95
 #5 [ffff880847f516b0] bad_area_nosemaphore at ffffffff8104cb63
 #6 [ffff880847f516c0] __do_page_fault at ffffffff8104d25c
 #7 [ffff880847f517e0] do_page_fault at ffffffff81530fbe
 #8 [ffff880847f51810] page_fault at ffffffff8152e375
    [exception RIP: sg_next+2]
    RIP: ffffffff8129c452  RSP: ffff880847f518c8  RFLAGS: 00010246
    RAX: 0000000000000000  RBX: ffff88046e524000  RCX: 0000000000000000
    RDX: 0000000000000101  RSI: ffffc900175fb240  RDI: 0000000000000000
    RBP: ffff880847f51940   R8: ffffea003958cb98   R9: 0000000000000301
    R10: 0000000000001000  R11: 0000000000000000  R12: ffff880c78a28940
    R13: ffff88046e536000  R14: ffffc900175fb240  R15: ffff880c7980c090
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
 #9 [ffff880847f518c8] kiblnd_map_tx at ffffffffa09cb46e [ko2iblnd]
#10 [ffff880847f51948] kiblnd_setup_rd_iov at ffffffffa09cbe0a [ko2iblnd]
#11 [ffff880847f519a8] kiblnd_send at ffffffffa09d151a [ko2iblnd]
#12 [ffff880847f51a48] lnet_ni_send at ffffffffa05e6d6b [lnet]
#13 [ffff880847f51a68] lnet_send at ffffffffa05eafa5 [lnet]
#14 [ffff880847f51ad8] LNetPut at ffffffffa05ec00a [lnet]
#15 [ffff880847f51b38] ptl_send_buf at ffffffffa07fbc40 [ptlrpc]
#16 [ffff880847f51be8] ptlrpc_send_reply at ffffffffa07fc22b [ptlrpc]
#17 [ffff880847f51c68] target_send_reply_msg at ffffffffa07c7054 [ptlrpc]
#18 [ffff880847f51c98] target_send_reply at ffffffffa07c7576 [ptlrpc]
#19 [ffff880847f51d08] mdt_handle_common at ffffffffa0ea2df9 [mdt]
#20 [ffff880847f51d58] mds_regular_handle at ffffffffa0edf645 [mdt]
#21 [ffff880847f51d68] ptlrpc_server_handle_request at ffffffffa0811ee5 [ptlrpc]
#22 [ffff880847f51e48] ptlrpc_main at ffffffffa081466d [ptlrpc]
#23 [ffff880847f51ee8] kthread at ffffffff8109e71e
#24 [ffff880847f51f48] kernel_thread at ffffffff8100c20a

crash> dis -rl ffffffffa09cb46e
0xffffffffa09cb2d0 <kiblnd_map_tx>:     push   %rbp
0xffffffffa09cb2d1 <kiblnd_map_tx+1>:   mov    %rsp,%rbp
0xffffffffa09cb2d4 <kiblnd_map_tx+4>:   push   %r15     
0xffffffffa09cb2d6 <kiblnd_map_tx+6>:   push   %r14     
0xffffffffa09cb2d8 <kiblnd_map_tx+8>:   push   %r13     
0xffffffffa09cb2da <kiblnd_map_tx+10>:  push   %r12     
0xffffffffa09cb2dc <kiblnd_map_tx+12>:  push   %rbx     
0xffffffffa09cb2dd <kiblnd_map_tx+13>:  sub    $0x48,%rsp
0xffffffffa09cb2e1 <kiblnd_map_tx+17>:  nopl   0x0(%rax,%rax,1)
0xffffffffa09cb2e6 <kiblnd_map_tx+22>:  mov    %ecx,-0x44(%rbp)
0xffffffffa09cb2e9 <kiblnd_map_tx+25>:  mov    0x10(%rsi),%rax 
0xffffffffa09cb2ed <kiblnd_map_tx+29>:  mov    %rdx,%rbx       
0xffffffffa09cb2f0 <kiblnd_map_tx+32>:  mov    0x50(%rdi),%rdi 
0xffffffffa09cb2f4 <kiblnd_map_tx+36>:  mov    %rsi,%r14       
0xffffffffa09cb2f7 <kiblnd_map_tx+39>:  mov    0x40(%rax),%r12 
0xffffffffa09cb2fb <kiblnd_map_tx+43>:  mov    %rdi,-0x50(%rbp)
0xffffffffa09cb2ff <kiblnd_map_tx+47>:  cmp    %rdx,0x80(%rsi) 
0xffffffffa09cb306 <kiblnd_map_tx+54>:  setne  %al             
0xffffffffa09cb309 <kiblnd_map_tx+57>:  movzbl %al,%edx        
0xffffffffa09cb30c <kiblnd_map_tx+60>:  add    $0x1,%edx       
0xffffffffa09cb30f <kiblnd_map_tx+63>:  mov    %edx,-0x48(%rbp)
0xffffffffa09cb312 <kiblnd_map_tx+66>:  mov    %edx,0xb0(%r14) 
0xffffffffa09cb319 <kiblnd_map_tx+73>:  mov    %ecx,0x88(%rsi) 
0xffffffffa09cb31f <kiblnd_map_tx+79>:  mov    0x8(%r12),%rdi  
0xffffffffa09cb324 <kiblnd_map_tx+84>:  mov    0x90(%rsi),%r13 
0xffffffffa09cb32b <kiblnd_map_tx+91>:  mov    0x2b8(%rdi),%rax
0xffffffffa09cb332 <kiblnd_map_tx+98>:  test   %rax,%rax       
0xffffffffa09cb335 <kiblnd_map_tx+101>: je     0xffffffffa09cb430 <kiblnd_map_tx+352>
0xffffffffa09cb33b <kiblnd_map_tx+107>: mov    %edx,%ecx
0xffffffffa09cb33d <kiblnd_map_tx+109>: mov    %r13,%rsi
0xffffffffa09cb340 <kiblnd_map_tx+112>: mov    -0x44(%rbp),%edx
0xffffffffa09cb343 <kiblnd_map_tx+115>: callq  *0x28(%rax)
0xffffffffa09cb346 <kiblnd_map_tx+118>: xor    %edx,%edx
0xffffffffa09cb348 <kiblnd_map_tx+120>: xor    %r15d,%r15d
0xffffffffa09cb34b <kiblnd_map_tx+123>: test   %eax,%eax
0xffffffffa09cb34d <kiblnd_map_tx+125>: mov    %eax,0x4(%rbx)
0xffffffffa09cb350 <kiblnd_map_tx+128>: jne    0xffffffffa09cb3b9 <kiblnd_map_tx+233>
0xffffffffa09cb352 <kiblnd_map_tx+130>: jmpq   0xffffffffa09cb3f0 <kiblnd_map_tx+288>
0xffffffffa09cb357 <kiblnd_map_tx+135>: nopw   0x0(%rax,%rax,1)
0xffffffffa09cb360 <kiblnd_map_tx+144>: mov    %edx,-0x60(%rbp)
0xffffffffa09cb363 <kiblnd_map_tx+147>: mov    %rcx,-0x68(%rbp)
0xffffffffa09cb367 <kiblnd_map_tx+151>: callq  *0x40(%rax)
0xffffffffa09cb36a <kiblnd_map_tx+154>: mov    -0x60(%rbp),%edx
0xffffffffa09cb36d <kiblnd_map_tx+157>: mov    -0x68(%rbp),%rcx
0xffffffffa09cb371 <kiblnd_map_tx+161>: lea    0x0(%r13,%r13,2),%rsi
0xffffffffa09cb376 <kiblnd_map_tx+166>: mov    %eax,0x8(%rbx,%rsi,4)
0xffffffffa09cb37a <kiblnd_map_tx+170>: mov    0x8(%r12),%rdi
0xffffffffa09cb37f <kiblnd_map_tx+175>: mov    %rcx,%rsi
0xffffffffa09cb382 <kiblnd_map_tx+178>: add    0x90(%r14),%rsi
0xffffffffa09cb389 <kiblnd_map_tx+185>: mov    0x2b8(%rdi),%rax
0xffffffffa09cb390 <kiblnd_map_tx+192>: test   %rax,%rax
0xffffffffa09cb393 <kiblnd_map_tx+195>: je     0xffffffffa09cb3e8 <kiblnd_map_tx+280>
0xffffffffa09cb395 <kiblnd_map_tx+197>: mov    %edx,-0x60(%rbp)
0xffffffffa09cb398 <kiblnd_map_tx+200>: callq  *0x38(%rax)
0xffffffffa09cb39b <kiblnd_map_tx+203>: mov    -0x60(%rbp),%edx
0xffffffffa09cb39e <kiblnd_map_tx+206>: lea    0x0(%r13,%r13,2),%rcx
0xffffffffa09cb3a3 <kiblnd_map_tx+211>: add    $0x1,%edx
0xffffffffa09cb3a6 <kiblnd_map_tx+214>: shl    $0x2,%rcx
0xffffffffa09cb3aa <kiblnd_map_tx+218>: mov    %rax,0xc(%rcx,%rbx,1)
0xffffffffa09cb3af <kiblnd_map_tx+223>: add    0x8(%rbx,%rcx,1),%r15d
0xffffffffa09cb3b4 <kiblnd_map_tx+228>: cmp    %edx,0x4(%rbx)
0xffffffffa09cb3b7 <kiblnd_map_tx+231>: jbe    0xffffffffa09cb3f0 <kiblnd_map_tx+288>
0xffffffffa09cb3b9 <kiblnd_map_tx+233>: mov    0x8(%r12),%rdi
0xffffffffa09cb3be <kiblnd_map_tx+238>: movslq %edx,%r13
0xffffffffa09cb3c1 <kiblnd_map_tx+241>: mov    %r13,%rcx
0xffffffffa09cb3c4 <kiblnd_map_tx+244>: shl    $0x5,%rcx
0xffffffffa09cb3c8 <kiblnd_map_tx+248>: mov    0x2b8(%rdi),%rax
0xffffffffa09cb3cf <kiblnd_map_tx+255>: mov    %rcx,%rsi
0xffffffffa09cb3d2 <kiblnd_map_tx+258>: add    0x90(%r14),%rsi
0xffffffffa09cb3d9 <kiblnd_map_tx+265>: test   %rax,%rax
0xffffffffa09cb3dc <kiblnd_map_tx+268>: jne    0xffffffffa09cb360 <kiblnd_map_tx+144>
0xffffffffa09cb3de <kiblnd_map_tx+270>: mov    0x18(%rsi),%eax
0xffffffffa09cb3e1 <kiblnd_map_tx+273>: jmp    0xffffffffa09cb371 <kiblnd_map_tx+161>
0xffffffffa09cb3e3 <kiblnd_map_tx+275>: nopl   0x0(%rax,%rax,1)
0xffffffffa09cb3e8 <kiblnd_map_tx+280>: mov    0x10(%rsi),%rax
0xffffffffa09cb3ec <kiblnd_map_tx+284>: jmp    0xffffffffa09cb39e <kiblnd_map_tx+206>
0xffffffffa09cb3ee <kiblnd_map_tx+286>: xchg   %ax,%ax
0xffffffffa09cb3f0 <kiblnd_map_tx+288>: mov    %rbx,%rsi
0xffffffffa09cb3f3 <kiblnd_map_tx+291>: mov    %r12,%rdi
0xffffffffa09cb3f6 <kiblnd_map_tx+294>: callq  0xffffffffa09be470 <kiblnd_find_rd_dma_mr>
0xffffffffa09cb3fb <kiblnd_map_tx+299>: test   %rax,%rax
0xffffffffa09cb3fe <kiblnd_map_tx+302>: je     0xffffffffa09cb492 <kiblnd_map_tx+450>
0xffffffffa09cb404 <kiblnd_map_tx+308>: cmp    %rbx,0x80(%r14)
0xffffffffa09cb40b <kiblnd_map_tx+315>: je     0xffffffffa09cb58e <kiblnd_map_tx+702>
0xffffffffa09cb411 <kiblnd_map_tx+321>: mov    0x1c(%rax),%eax
0xffffffffa09cb414 <kiblnd_map_tx+324>: mov    %eax,(%rbx)
0xffffffffa09cb416 <kiblnd_map_tx+326>: xor    %r8d,%r8d
0xffffffffa09cb419 <kiblnd_map_tx+329>: add    $0x48,%rsp
0xffffffffa09cb41d <kiblnd_map_tx+333>: mov    %r8d,%eax
0xffffffffa09cb420 <kiblnd_map_tx+336>: pop    %rbx
0xffffffffa09cb421 <kiblnd_map_tx+337>: pop    %r12
0xffffffffa09cb423 <kiblnd_map_tx+339>: pop    %r13
0xffffffffa09cb425 <kiblnd_map_tx+341>: pop    %r14
0xffffffffa09cb427 <kiblnd_map_tx+343>: pop    %r15
0xffffffffa09cb429 <kiblnd_map_tx+345>: leaveq
0xffffffffa09cb42a <kiblnd_map_tx+346>: retq
0xffffffffa09cb42b <kiblnd_map_tx+347>: nopl   0x0(%rax,%rax,1)
0xffffffffa09cb430 <kiblnd_map_tx+352>: mov    (%rdi),%r15
0xffffffffa09cb433 <kiblnd_map_tx+355>: test   %r15,%r15
0xffffffffa09cb436 <kiblnd_map_tx+358>: je     0xffffffffa09cb596 <kiblnd_map_tx+710>
0xffffffffa09cb43c <kiblnd_map_tx+364>: mov    0x1c0(%r15),%rcx
0xffffffffa09cb443 <kiblnd_map_tx+371>: test   %rcx,%rcx
0xffffffffa09cb446 <kiblnd_map_tx+374>: mov    %rcx,-0x58(%rbp)
0xffffffffa09cb44a <kiblnd_map_tx+378>: je     0xffffffffa09cb596 <kiblnd_map_tx+710>
0xffffffffa09cb450 <kiblnd_map_tx+384>: mov    -0x44(%rbp),%r9d
0xffffffffa09cb454 <kiblnd_map_tx+388>: test   %r9d,%r9d
0xffffffffa09cb457 <kiblnd_map_tx+391>: jle    0xffffffffa09cb476 <kiblnd_map_tx+422>
0xffffffffa09cb459 <kiblnd_map_tx+393>: mov    %r13,%rax
0xffffffffa09cb45c <kiblnd_map_tx+396>: xor    %edx,%edx
0xffffffffa09cb45e <kiblnd_map_tx+398>: xchg   %ax,%ax
0xffffffffa09cb460 <kiblnd_map_tx+400>: add    $0x1,%edx
0xffffffffa09cb463 <kiblnd_map_tx+403>: mov    %rax,%rdi
0xffffffffa09cb466 <kiblnd_map_tx+406>: mov    %edx,-0x60(%rbp)
0xffffffffa09cb469 <kiblnd_map_tx+409>: callq  0xffffffff8129c450 <sg_next>
0xffffffffa09cb46e <kiblnd_map_tx+414>: mov    -0x60(%rbp),%edx

crash> bt -f                                                                             
PID: 20168  TASK: ffff88084585aab0  CPU: 23  COMMAND: "mdt03_049"                        
 #0 [ffff880847f514b0] machine_kexec at ffffffff8103b71b
    ffff880847f514b8: 00000000030a1000 ffff8800030a1000
    ffff880847f514c8: 00000000030a0000 ffff880847f51818
    ffff880847f514d8: 8800000000000000 000000000000ffff
    ffff880847f514e8: ffff880847f51818 ffff880847f51518
    ffff880847f514f8: 0000000000000009 ffff88084585aab0
    ffff880847f51508: ffff880847f515d8 ffffffff810c9942                                  
 #1 [ffff880847f51510] crash_kexec at ffffffff810c9942
    ffff880847f51518: ffff880c7980c090 ffffc900175fb240
    ffff880847f51528: ffff88046e536000 ffff880c78a28940
    ffff880847f51538: ffff880847f51940 ffff88046e524000
    ffff880847f51548: 0000000000000000 0000000000001000
    ffff880847f51558: 0000000000000301 ffffea003958cb98
    ffff880847f51568: 0000000000000000 0000000000000000
    ffff880847f51578: 0000000000000101 ffffc900175fb240
    ffff880847f51588: 0000000000000000 ffffffffffffffff                                  
    ffff880847f51598: ffffffff8129c452 0000000000000010
    ffff880847f515a8: 0000000000010246 ffff880847f518c8
    ffff880847f515b8: 0000000000000018 ffff880847f51618
    ffff880847f515c8: 0000000000000246 ffff880847f51818
    ffff880847f515d8: ffff880847f51608 ffffffff8152f070                                  
 #2 [ffff880847f515e0] oops_end at ffffffff8152f070                                      
    ffff880847f515e8: 0000000000000000 ffff880847f51818
    ffff880847f515f8: 0000000000000000 0000000000000009
    ffff880847f51608: ffff880847f51658 ffffffff8104c80b                                  
 #3 [ffff880847f51610] no_context at ffffffff8104c80b                                    
    ffff880847f51618: ffffffff81531036 000000000000000a
    ffff880847f51628: ffff88047fe9be00 0000000000000000
    ffff880847f51638: 0000000000000000 ffff880847f51818
    ffff880847f51648: ffff88084585aab0 0000000000030001
    ffff880847f51658: ffff880847f516a8 ffffffff8104ca95                                  
 #4 [ffff880847f51660] __bad_area_nosemaphore at ffffffff8104ca95
    ffff880847f51668: ffffffff8134382e ffff88047fe9be00
    ffff880847f51678: 0000000000000000 0000000000000028
    ffff880847f51688: 0000000000000000 0000000000000000
    ffff880847f51698: ffffc900175fb240 ffff88084585aab0
    ffff880847f516a8: ffff880847f516b8 ffffffff8104cb63                                  
 #5 [ffff880847f516b0] bad_area_nosemaphore at ffffffff8104cb63
    ffff880847f516b8: ffff880847f517d8 ffffffff8104d25c                                  
 #6 [ffff880847f516c0] __do_page_fault at ffffffff8104d25c
    ffff880847f516c8: 000000000001da70 000000000000004e
    ffff880847f516d8: ffff880847f51818 0000000000000000
    ffff880847f516e8: 0000000000000068 0000000000000000
    ffff880847f516f8: ffffffff8100bb8e ffff880847f51810
    ffff880847f51708: 0000000000000000 0000000000000001
    ffff880847f51718: ffffffff816490a0 0000000000000000
    ffff880847f51728: 0000000000010500 0000000000001ba9
    ffff880847f51738: ffff880c8e540000 0000000000000046
    ffff880847f51748: 0000000000000246 ffffffffffffff10                                  
    ffff880847f51758: ffffffff81075d81 0000000000000010
    ffff880847f51768: 0000000000000246 ffff880847f51780
    ffff880847f51778: 0000000000000018 ffffc90017a2d032
    ffff880847f51788: 0000000000000246 ffffffff00000017
    ffff880847f51798: ffffffff8115c6dd 0000000000000010
    ffff880847f517a8: 0000000000000286 ffff880847f51818
    ffff880847f517b8: 0000000000000000 0000000000000000
    ffff880847f517c8: ffffc900175fb240 ffff880c7980c090
    ffff880847f517d8: ffff880847f51808 ffffffff81530fbe
 #7 [ffff880847f517e0] do_page_fault at ffffffff81530fbe
    ffff880847f517e8: 0000000000000001 ffff880c78a28940
    ffff880847f517f8: ffff88046e536000 ffffc900175fb240
    ffff880847f51808: ffff880847f51940 ffffffff8152e375
 #8 [ffff880847f51810] page_fault at ffffffff8152e375
    [exception RIP: sg_next+2]
    RIP: ffffffff8129c452  RSP: ffff880847f518c8  RFLAGS: 00010246
    RAX: 0000000000000000  RBX: ffff88046e524000  RCX: 0000000000000000
    RDX: 0000000000000101  RSI: ffffc900175fb240  RDI: 0000000000000000
    RBP: ffff880847f51940   R8: ffffea003958cb98   R9: 0000000000000301
    R10: 0000000000001000  R11: 0000000000000000  R12: ffff880c78a28940
    R13: ffff88046e536000  R14: ffffc900175fb240  R15: ffff880c7980c090
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018                                       
    ffff880847f51818: ffff880c7980c090 ffffc900175fb240
    ffff880847f51828: ffff88046e536000 ffff880c78a28940
    ffff880847f51838: ffff880847f51940 ffff88046e524000
    ffff880847f51848: 0000000000000000 0000000000001000
    ffff880847f51858: 0000000000000301 ffffea003958cb98
    ffff880847f51868: 0000000000000000 0000000000000000
    ffff880847f51878: 0000000000000101 ffffc900175fb240
    ffff880847f51888: 0000000000000000 ffffffffffffffff
    ffff880847f51898: ffffffff8129c452 0000000000000010
    ffff880847f518a8: 0000000000010246 ffff880847f518c8
    ffff880847f518b8: 0000000000000018 ffff880847f51940
    ffff880847f518c8: ffffffffa09cb46e
 #9 [ffff880847f518c8] kiblnd_map_tx at ffffffffa09cb46e [ko2iblnd]
    ffff880847f518d0: ffff880847f51fd8 ffff88084585aab0
    ffff880847f518e0: ffffffff00000101 ffffffff81a9ac80
    ffff880847f518f0: ffff880c78a289c0 0000030100000001
    ffff880847f51900: ffff880847f51920 ffffffff8105e100
    ffff880847f51910: ffff880876a8d558 ffff880c796ffb80                                  <=== HERE rbx: ffff880c796ffb80
    ffff880847f51920: ffffc900175fb240 ffff88046e524000
    ffff880847f51930: 0000000000000000 ffff8810639ee640
    ffff880847f51940: ffff880847f519a0 ffffffffa09cbe0a                                  
#10 [ffff880847f51948] kiblnd_setup_rd_iov at ffffffffa09cbe0a [ko2iblnd]                
    ffff880847f51950: ffff880800000378 ffff880c002ffef8                                  
    ffff880847f51960: ffff88046e53c000 ffffc90048a88000                                  
    ffff880847f51970: 0005000221056e78 ffff88105fff6200
    ffff880847f51980: 0000000000300270 ffffc900175fb240
    ffff880847f51990: 0005000221056e78 0000000000000001
    ffff880847f519a0: ffff880847f51a40 ffffffffa09d151a
#11 [ffff880847f519a8] kiblnd_send at ffffffffa09d151a [ko2iblnd]
    ffff880847f519b0: 0000000000300270 ffff8810638aa000
    ffff880847f519c0: ffffc90048788488 0000000000000000
    ffff880847f519d0: ffff881000000001 000000006714f9c0                                  
    ffff880847f519e0: ffff8810639ee630 ffff880c796ffb80                                  
    ffff880847f519f0: 000000000000ec08 ffff88105fff6200                                  
    ffff880847f51a00: 0005000221056e78 0000000000003039                                  
    ffff880847f51a10: ffff880847f51a60 ffff880c796ffb80                                  
    ffff880847f51a20: ffff88105fff6200 ffff88105fff6200                                  <=== HERE r12: ffff88105fff6200
    ffff880847f51a30: 0000000000000000 0000000000050003                                  
    ffff880847f51a40: ffff880847f51a60 ffffffffa05e6d6b                                  
#12 [ffff880847f51a48] lnet_ni_send at ffffffffa05e6d6b [lnet]                           
    ffff880847f51a50: ffff880c796ffb80 0000000000000000                                  
    ffff880847f51a60: ffff880847f51ad0 ffffffffa05eafa5                                  
#13 [ffff880847f51a68] lnet_send at ffffffffa05eafa5 [lnet]                              
    ffff880847f51a70: ffff880847f51ac0 0000000000000000                                  
    ffff880847f51a80: 000500030a64b972 0005000221056e03                                  
    ffff880847f51a90: ffff880847f51ad0 ffff8804780fd180                                  
    ffff880847f51aa0: 000500030a64b972 ffff88105fff6200                                  
    ffff880847f51ab0: ffff8810639ee5c0 0000000000000003                                  
    ffff880847f51ac0: 00000000002c3f4b 0000000000000001                                  
    ffff880847f51ad0: ffff880847f51b30 ffffffffa05ec00a                                  
#14 [ffff880847f51ad8] LNetPut at ffffffffa05ec00a [lnet]                                
    ffff880847f51ae0: 0000000147f51b30 0005000221056e03                                  
    ffff880847f51af0: 000500030a64b972 0000000000003039                                  
    ffff880847f51b00: ffff880847f51bd0 ffff880459fb2240                                  
    ffff880847f51b10: 0000000000000000 ffffc90048788108                                  
    ffff880847f51b20: 0000000000000023 0000000000000001                                  
    ffff880847f51b30: ffff880847f51be0 ffffffffa07fbc40                                  
#15 [ffff880847f51b38] ptl_send_buf at ffffffffa07fbc40 [ptlrpc]                         
    ffff880847f51b40: 00055926b072ae94 00000001000000c0                                  
    ffff880847f51b50: 0000000000000000 ffffc90048788000                                  
    ffff880847f51b60: 0000000000000023 ffffffffa081b8b8                                  
    ffff880847f51b70: 0030027047f51b88 ffffc90048788070                                  
    ffff880847f51b80: ffffc90048788108 0000000100300270
    ffff880847f51b90: 0000000048788108 ffffc90048788000
    ffff880847f51ba0: 0000000000000023 ffffffffa083c445
    ffff880847f51bb0: ffff88105ac1ec00 ffff88105ac1ec00
    ffff880847f51bc0: ffffc90048788000 ffff880459fb2240
    ffff880847f51bd0: 0000000000000000 0000000000000001
    ffff880847f51be0: ffff880847f51c60 ffffffffa07fc22b
#16 [ffff880847f51be8] ptlrpc_send_reply at ffffffffa07fc22b [ptlrpc]
    ffff880847f51bf0: ffff88100000000a 00055926b072ae94
    ffff880847f51c00: ffff8808000000c0 ffff8804573e5e40
    ffff880847f51c10: ffff8804573e5e58 ffff8804573e5e70
    ffff880847f51c20: ffff880847f51c70 ffff880850ce0a80
    ffff880847f51c30: ffff880847f51c30 ffff88105ac1ec00
    ffff880847f51c40: 0000000000000000 ffffffffa0f18580
    ffff880847f51c50: ffff88106aa76000 000000000000030c
    ffff880847f51c60: ffff880847f51c90 ffffffffa07c7054
#17 [ffff880847f51c68] target_send_reply_msg at ffffffffa07c7054 [ptlrpc]
    ffff880847f51c70: ffff880847f51ce0 ffffc90048788000
    ffff880847f51c80: ffff88105ac1ec00 ffffffffa0f18580
    ffff880847f51c90: ffff880847f51d00 ffffffffa07c7576
#18 [ffff880847f51c98] target_send_reply at ffffffffa07c7576 [ptlrpc]
    ffff880847f51ca0: ffff8810653a0880 0000000000001001
    ffff880847f51cb0: ffffc90048788108 0000000d0012ef0c
    ffff880847f51cc0: ffff880847f51ce0 00000000a0802b6c
    ffff880847f51cd0: ffffc9004027b9e8 ffff88105ac1ec00
    ffff880847f51ce0: ffff881062860000 ffffffffa0f18580
    ffff880847f51cf0: ffff88105ac1efa0 0000000000000000
    ffff880847f51d00: ffff880847f51d50 ffffffffa0ea2df9
#19 [ffff880847f51d08] mdt_handle_common at ffffffffa0ea2df9 [mdt]
    ffff880847f51d10: ffff880847f51d30 ffffffff00000002
    ffff880847f51d20: ffff880847f51d60 ffff88105ac1ec00
    ffff880847f51d30: ffff880850ce0a80 ffff88106aa76000
    ffff880847f51d40: ffff881067ecd940 ffff88105ac1ef40
    ffff880847f51d50: ffff880847f51d60 ffffffffa0edf645
#20 [ffff880847f51d58] mds_regular_handle at ffffffffa0edf645 [mdt]
    ffff880847f51d60: ffff880847f51e40 ffffffffa0811ee5
#21 [ffff880847f51d68] ptlrpc_server_handle_request at ffffffffa0811ee5 [ptlrpc]
    ffff880847f51d70: ffff880847f51d80 ffffffffa053e4ce
    ffff880847f51d80: ffff880847f51da0 ffffffffa054f7d5
    ffff880847f51d90: ffff881067ecd940 ffff88106aa76000
    ffff880847f51da0: ffff880847f51e40 ffffffffa080a919
    ffff880847f51db0: ffff880847f51e00 ffffffff81057819
    ffff880847f51dc0: ffff88106af7bdc0 0000000300000000
    ffff880847f51dd0: ffff88106af7bdc0 ffff88106aa76080
    ffff880847f51de0: 0000000000000282 0000000000000014
    ffff880847f51df0: 0000000000000001 0000000000000282
    ffff880847f51e00: 0000000055928287 00000000000884c1
    ffff880847f51e10: ffff880847f51e40 ffff881067ecd940
    ffff880847f51e20: ffff88106aa76000 0000000000000040
    ffff880847f51e30: 0000000000000005 ffff880850ce0a80
    ffff880847f51e40: ffff880847f51ee0 ffffffffa081466d
#22 [ffff880847f51e48] ptlrpc_main at ffffffffa081466d [ptlrpc]
    ffff880847f51e50: ffff880847f51e60 ffff88106aa76204
    ffff880847f51e60: ffff88106aa76080 0000000047f51fd8
    ffff880847f51e70: ffff880850ce0a80 ffff8810670b4800
    ffff880847f51e80: 000000004585aab0 ffff88106aa76068
    ffff880847f51e90: ffff88106aa76048 ffff881067ecd978
    ffff880847f51ea0: ffff8810670b4800 ffff88106aa76030
    ffff880847f51eb0: ffff880847f51ee0 ffff881062805780
    ffff880847f51ec0: ffff880847f51ef8 ffffffffa0813b80
    ffff880847f51ed0: ffff881067ecd940 ffff880479c40ab0
    ffff880847f51ee0: ffff880847f51f40 ffffffff8109e71e
#23 [ffff880847f51ee8] kthread at ffffffff8109e71e
    ffff880847f51ef0: ffffffff00000000 5a5a5a5a00000000
    ffff880847f51f00: 5a5a5a5a00000000 ffff880847f51f08
    ffff880847f51f10: ffff880847f51f08 0000000000000000
    ffff880847f51f20: ffff881062805780 ffffffff81ebb4c8
    ffff880847f51f30: ffff880479c40ab0 0000000000000000
    ffff880847f51f40: ffff880479c47f40 ffffffff8100c20a
#24 [ffff880847f51f48] kernel_thread at ffffffff8100c20a

Could you help me to troubleshoot this issue? Is there anything I should look into the crash dump?

Whamcloud Community JIRA

[LU-6799] getxattr failed: -2 triggers a Kernel BUG on MDS