Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-7353

Segfault on lctl commands 'lctl network peer_list' 'lctl network conn_list'

Details

    • Bug
    • Resolution: Fixed
    • Minor
    • Lustre 2.8.0
    • Lustre 2.8.0
    • None
    • CentOS 6.6, 2.6.32-504.el6.x86_64, Lustre 2.7.56 / ZFS
      (used Lustre 2.8 version as is closest to actual version)
    • 3
    • 9223372036854775807

    Description

      LFS performing normally. Admin issues lctl commands which result in segfaults.

      'lctl network peer_list'
      'lctl network conn_list'
      each result in segfaults.

      Attached core file resulting from 'lctl network peer_list' executed on an OSS node.

      Attachments

        Issue Links

          Activity

            [LU-7353] Segfault on lctl commands 'lctl network peer_list' 'lctl network conn_list'

            Landed for 2.8

            jgmitter Joseph Gmitter (Inactive) added a comment - Landed for 2.8

            Oleg Drokin (oleg.drokin@intel.com) merged in patch http://review.whamcloud.com/16980/
            Subject: LU-7353 utils: fix lctl usage messages
            Project: fs/lustre-release
            Branch: master
            Current Patch Set:
            Commit: fa54edb5af34c3bcbf878d7498ef337ef3a1df7a

            gerrit Gerrit Updater added a comment - Oleg Drokin (oleg.drokin@intel.com) merged in patch http://review.whamcloud.com/16980/ Subject: LU-7353 utils: fix lctl usage messages Project: fs/lustre-release Branch: master Current Patch Set: Commit: fa54edb5af34c3bcbf878d7498ef337ef3a1df7a

            Andreas Dilger (andreas.dilger@intel.com) uploaded a new patch: http://review.whamcloud.com/16980
            Subject: LU-7353 utils: fix lctl usage messages
            Project: fs/lustre-release
            Branch: master
            Current Patch Set: 1
            Commit: ae64bd87141af3187144f1ef1e233abd5eaf7a39

            gerrit Gerrit Updater added a comment - Andreas Dilger (andreas.dilger@intel.com) uploaded a new patch: http://review.whamcloud.com/16980 Subject: LU-7353 utils: fix lctl usage messages Project: fs/lustre-release Branch: master Current Patch Set: 1 Commit: ae64bd87141af3187144f1ef1e233abd5eaf7a39

            Hi Dmitry,
            Can you have a look at this issue?
            Thanks.
            Joe

            jgmitter Joseph Gmitter (Inactive) added a comment - Hi Dmitry, Can you have a look at this issue? Thanks. Joe

            Running 'lctl --net o2ib0 peer_list' appears to function normally. Executing the lctl command with improper arguments results in either a segfault or no error at all.

            'lctl network peer_list' results in a segfault

            'lctl network o2ib0 peer_list' results in no segfault, no stderr, exit 0

            'lctl --net o2ib0 peer_list' executes properly as expected.

            aeonjeffj Jeff Johnson (Inactive) added a comment - Running 'lctl --net o2ib0 peer_list' appears to function normally. Executing the lctl command with improper arguments results in either a segfault or no error at all. 'lctl network peer_list' results in a segfault 'lctl network o2ib0 peer_list' results in no segfault, no stderr, exit 0 'lctl --net o2ib0 peer_list' executes properly as expected.

            Trace:

            [root@ls15-mds-00.i ~]# lctl network peer_list

                • Segmentation fault
                  Register dump:

            RAX: 000000000063a620 RBX: 000000000063a620 RCX: 0000000000000000
            RDX: 000000000000000e RSI: 000000000043280b RDI: 0000000000000000
            RBP: 00007fffade0c00c R8 : 0000000000000000 R9 : 0000000000000001
            R10: 00000030d5329f00 R11: 0000000000000003 R12: 000000000000000f
            R13: 0000000000000000 R14: 0000000000000003 R15: 00007fffade0c7e6
            RSP: 00007fffade0bfa8

            RIP: 00000030d533372f EFLAGS: 00010246

            CS: 0033 FS: 0000 GS: 0000

            Trap: 0000000e Error: 00000004 OldMask: 00000000 CR2: 00000000

            FPUCW: 0000037f FPUSW: 00000000 TAG: 00000000
            RIP: 00000000 RDP: 00000000

            ST(0) 0000 0000000000000000 ST(1) 0000 0000000000000000
            ST(2) 0000 0000000000000000 ST(3) 0000 0000000000000000
            ST(4) 0000 0000000000000000 ST(5) 0000 0000000000000000
            ST(6) 0000 0000000000000000 ST(7) 0000 0000000000000000
            mxcsr: 1f80
            XMM0: 000000000000000000000000ff000000 XMM1: 000000000000000000000000ff000000
            XMM2: 000000000000000000000000ff000000 XMM3: 000000000000000000000000ff000000
            XMM4: 000000000000000000000000ff000000 XMM5: 000000000000000000000000ff000000
            XMM6: 000000000000000000000000ff000000 XMM7: 000000000000000000000000ff000000
            XMM8: 000000000000000000000000ff000000 XMM9: 000000000000000000000000ff000000
            XMM10: 000000000000000000000000ff000000 XMM11: 000000000000000000000000ff000000
            XMM12: 000000000000000000000000ff000000 XMM13: 000000000000000000000000ff000000
            XMM14: 000000000000000000000000ff000000 XMM15: 000000000000000000000000ff000000

            Backtrace:
            /lib64/libc.so.6[0x30d533372f]
            lctl[0x428a35]
            lctl[0x428b0e]
            lctl[0x422081]
            lctl[0x427698]
            lctl[0x416517]
            /lib64/libc.so.6(__libc_start_main+0xfd)[0x30d521ed5d]
            lctl[0x402f59]

            Memory map:

            00400000-00438000 r-xp 00000000 09:02 1201235 /usr/sbin/lctl
            00637000-0063b000 rw-p 00037000 09:02 1201235 /usr/sbin/lctl
            0063b000-00658000 rw-p 00000000 00:00 0
            0083a000-0083b000 rw-p 0003a000 09:02 1201235 /usr/sbin/lctl
            01dbd000-01dde000 rw-p 00000000 00:00 0 [heap]
            30d4e00000-30d4e20000 r-xp 00000000 09:02 131102 /lib64/ld-2.12.so
            30d501f000-30d5020000 r--p 0001f000 09:02 131102 /lib64/ld-2.12.so
            30d5020000-30d5021000 rw-p 00020000 09:02 131102 /lib64/ld-2.12.so
            30d5021000-30d5022000 rw-p 00000000 00:00 0
            30d5200000-30d538a000 r-xp 00000000 09:02 131128 /lib64/libc-2.12.so
            30d538a000-30d558a000 ---p 0018a000 09:02 131128 /lib64/libc-2.12.so
            30d558a000-30d558e000 r--p 0018a000 09:02 131128 /lib64/libc-2.12.so
            30d558e000-30d558f000 rw-p 0018e000 09:02 131128 /lib64/libc-2.12.so
            30d558f000-30d5594000 rw-p 00000000 00:00 0
            30d5600000-30d5602000 r-xp 00000000 09:02 131321 /lib64/libdl-2.12.so
            30d5602000-30d5802000 ---p 00002000 09:02 131321 /lib64/libdl-2.12.so
            30d5802000-30d5803000 r--p 00002000 09:02 131321 /lib64/libdl-2.12.so
            30d5803000-30d5804000 rw-p 00003000 09:02 131321 /lib64/libdl-2.12.so
            30d5a00000-30d5a17000 r-xp 00000000 09:02 131154 /lib64/libpthread-2.12.s
            o
            30d5a17000-30d5c17000 ---p 00017000 09:02 131154 /lib64/libpthread-2.12.s
            o
            30d5c17000-30d5c18000 r--p 00017000 09:02 131154 /lib64/libpthread-2.12.s
            o
            30d5c18000-30d5c19000 rw-p 00018000 09:02 131154 /lib64/libpthread-2.12.s
            o
            30d5c19000-30d5c1d000 rw-p 00000000 00:00 0
            30d7200000-30d7216000 r-xp 00000000 09:02 131557 /lib64/libgcc_s-4.4.7-20
            120601.so.1
            30d7216000-30d7415000 ---p 00016000 09:02 131557 /lib64/libgcc_s-4.4.7-20
            120601.so.1
            30d7415000-30d7416000 rw-p 00015000 09:02 131557 /lib64/libgcc_s-4.4.7-20
            120601.so.1
            30d7600000-30d7602000 r-xp 00000000 09:02 131500 /lib64/libkeyutils.so.1.
            3
            30d7602000-30d7801000 ---p 00002000 09:02 131500 /lib64/libkeyutils.so.1.
            3
            30d7801000-30d7802000 r--p 00001000 09:02 131500 /lib64/libkeyutils.so.1.
            3
            30d7802000-30d7803000 rw-p 00002000 09:02 131500 /lib64/libkeyutils.so.1.
            3
            7f6e22da2000-7f6e22db3000 rw-s 00000000 00:04 1736704 /SYSV00000000 (deleted)
            7f6e22db3000-7f6e22db7000 rw-p 00000000 00:00 0
            7f6e22db7000-7f6e22dbb000 r-xp 00000000 09:02 131082 /lib64/libSegFault.so
            7f6e22dbb000-7f6e22fba000 ---p 00004000 09:02 131082 /lib64/libSegFault.so
            7f6e22fba000-7f6e22fbb000 r--p 00003000 09:02 131082 /lib64/libSegFault.so
            7f6e22fbb000-7f6e22fbc000 rw-p 00004000 09:02 131082 /lib64/libSegFault.so
            7f6e22fc9000-7f6e22fca000 rw-p 00000000 00:00 0
            7fffaddf8000-7fffade0d000 rw-p 00000000 00:00 0 [stack]
            7fffade25000-7fffade26000 r-xp 00000000 00:00 0 [vdso]
            ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]

            aeonjeffj Jeff Johnson (Inactive) added a comment - Trace: [root@ls15-mds-00.i ~] # lctl network peer_list Segmentation fault Register dump: RAX: 000000000063a620 RBX: 000000000063a620 RCX: 0000000000000000 RDX: 000000000000000e RSI: 000000000043280b RDI: 0000000000000000 RBP: 00007fffade0c00c R8 : 0000000000000000 R9 : 0000000000000001 R10: 00000030d5329f00 R11: 0000000000000003 R12: 000000000000000f R13: 0000000000000000 R14: 0000000000000003 R15: 00007fffade0c7e6 RSP: 00007fffade0bfa8 RIP: 00000030d533372f EFLAGS: 00010246 CS: 0033 FS: 0000 GS: 0000 Trap: 0000000e Error: 00000004 OldMask: 00000000 CR2: 00000000 FPUCW: 0000037f FPUSW: 00000000 TAG: 00000000 RIP: 00000000 RDP: 00000000 ST(0) 0000 0000000000000000 ST(1) 0000 0000000000000000 ST(2) 0000 0000000000000000 ST(3) 0000 0000000000000000 ST(4) 0000 0000000000000000 ST(5) 0000 0000000000000000 ST(6) 0000 0000000000000000 ST(7) 0000 0000000000000000 mxcsr: 1f80 XMM0: 000000000000000000000000ff000000 XMM1: 000000000000000000000000ff000000 XMM2: 000000000000000000000000ff000000 XMM3: 000000000000000000000000ff000000 XMM4: 000000000000000000000000ff000000 XMM5: 000000000000000000000000ff000000 XMM6: 000000000000000000000000ff000000 XMM7: 000000000000000000000000ff000000 XMM8: 000000000000000000000000ff000000 XMM9: 000000000000000000000000ff000000 XMM10: 000000000000000000000000ff000000 XMM11: 000000000000000000000000ff000000 XMM12: 000000000000000000000000ff000000 XMM13: 000000000000000000000000ff000000 XMM14: 000000000000000000000000ff000000 XMM15: 000000000000000000000000ff000000 Backtrace: /lib64/libc.so.6 [0x30d533372f] lctl [0x428a35] lctl [0x428b0e] lctl [0x422081] lctl [0x427698] lctl [0x416517] /lib64/libc.so.6(__libc_start_main+0xfd) [0x30d521ed5d] lctl [0x402f59] Memory map: 00400000-00438000 r-xp 00000000 09:02 1201235 /usr/sbin/lctl 00637000-0063b000 rw-p 00037000 09:02 1201235 /usr/sbin/lctl 0063b000-00658000 rw-p 00000000 00:00 0 0083a000-0083b000 rw-p 0003a000 09:02 1201235 /usr/sbin/lctl 01dbd000-01dde000 rw-p 00000000 00:00 0 [heap] 30d4e00000-30d4e20000 r-xp 00000000 09:02 131102 /lib64/ld-2.12.so 30d501f000-30d5020000 r--p 0001f000 09:02 131102 /lib64/ld-2.12.so 30d5020000-30d5021000 rw-p 00020000 09:02 131102 /lib64/ld-2.12.so 30d5021000-30d5022000 rw-p 00000000 00:00 0 30d5200000-30d538a000 r-xp 00000000 09:02 131128 /lib64/libc-2.12.so 30d538a000-30d558a000 ---p 0018a000 09:02 131128 /lib64/libc-2.12.so 30d558a000-30d558e000 r--p 0018a000 09:02 131128 /lib64/libc-2.12.so 30d558e000-30d558f000 rw-p 0018e000 09:02 131128 /lib64/libc-2.12.so 30d558f000-30d5594000 rw-p 00000000 00:00 0 30d5600000-30d5602000 r-xp 00000000 09:02 131321 /lib64/libdl-2.12.so 30d5602000-30d5802000 ---p 00002000 09:02 131321 /lib64/libdl-2.12.so 30d5802000-30d5803000 r--p 00002000 09:02 131321 /lib64/libdl-2.12.so 30d5803000-30d5804000 rw-p 00003000 09:02 131321 /lib64/libdl-2.12.so 30d5a00000-30d5a17000 r-xp 00000000 09:02 131154 /lib64/libpthread-2.12.s o 30d5a17000-30d5c17000 ---p 00017000 09:02 131154 /lib64/libpthread-2.12.s o 30d5c17000-30d5c18000 r--p 00017000 09:02 131154 /lib64/libpthread-2.12.s o 30d5c18000-30d5c19000 rw-p 00018000 09:02 131154 /lib64/libpthread-2.12.s o 30d5c19000-30d5c1d000 rw-p 00000000 00:00 0 30d7200000-30d7216000 r-xp 00000000 09:02 131557 /lib64/libgcc_s-4.4.7-20 120601.so.1 30d7216000-30d7415000 ---p 00016000 09:02 131557 /lib64/libgcc_s-4.4.7-20 120601.so.1 30d7415000-30d7416000 rw-p 00015000 09:02 131557 /lib64/libgcc_s-4.4.7-20 120601.so.1 30d7600000-30d7602000 r-xp 00000000 09:02 131500 /lib64/libkeyutils.so.1. 3 30d7602000-30d7801000 ---p 00002000 09:02 131500 /lib64/libkeyutils.so.1. 3 30d7801000-30d7802000 r--p 00001000 09:02 131500 /lib64/libkeyutils.so.1. 3 30d7802000-30d7803000 rw-p 00002000 09:02 131500 /lib64/libkeyutils.so.1. 3 7f6e22da2000-7f6e22db3000 rw-s 00000000 00:04 1736704 /SYSV00000000 (deleted) 7f6e22db3000-7f6e22db7000 rw-p 00000000 00:00 0 7f6e22db7000-7f6e22dbb000 r-xp 00000000 09:02 131082 /lib64/libSegFault.so 7f6e22dbb000-7f6e22fba000 ---p 00004000 09:02 131082 /lib64/libSegFault.so 7f6e22fba000-7f6e22fbb000 r--p 00003000 09:02 131082 /lib64/libSegFault.so 7f6e22fbb000-7f6e22fbc000 rw-p 00004000 09:02 131082 /lib64/libSegFault.so 7f6e22fc9000-7f6e22fca000 rw-p 00000000 00:00 0 7fffaddf8000-7fffade0d000 rw-p 00000000 00:00 0 [stack] 7fffade25000-7fffade26000 r-xp 00000000 00:00 0 [vdso] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]

            People

              dmiter Dmitry Eremin (Inactive)
              aeonjeffj Jeff Johnson (Inactive)
              Votes:
              1 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: