Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-5605

Interop 2.5.2<->2.7 lustre-rsync-test test_6: soft lockup on statahead

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Minor
    • None
    • Lustre 2.7.0, Lustre 2.5.4
    • None
    • server: lustre-master build # 2639
      client: 2.5.2
    • 3
    • 15678

    Description

      This issue was created by maloo for sarah <sarah@whamcloud.com>

      This issue relates to the following test suite run: https://testing.hpdd.intel.com/test_sets/7e94d1f8-32e1-11e4-9c61-5254006e85c2.

      The sub-test test_6 failed with the following error:

      test failed to respond and timed out

      test log

      22:57:14:Lustre: DEBUG MARKER: == lustre-rsync-test test 6: lustre_rsync large no of hard links == 22:55:14 (1409637314)
22:57:14:BUG: soft lockup - CPU#0 stuck for 67s! [ll_sa_27678:27679]
22:57:14:Modules linked in: lustre(U) obdecho(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) sha512_generic sha256_generic nfs fscache nfsd lockd nfs_acl auth_rpcgss sunrpc exportfs autofs4 ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs]
22:57:14:BUG: soft lockup - CPU#1 stuck for 67s! [ptlrpcd_0:1399]
22:57:14:Modules linked in: lustre(U) obdecho(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) sha512_generic sha256_generic nfs fscache nfsd lockd nfs_acl auth_rpcgss sunrpc exportfs autofs4 ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs]
22:57:14:CPU 1 
22:57:14:Modules linked in: lustre(U) obdecho(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) sha512_generic sha256_generic nfs fscache nfsd lockd nfs_acl auth_rpcgss sunrpc exportfs autofs4 ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs]
22:57:14:
22:57:14:Pid: 1399, comm: ptlrpcd_0 Not tainted 2.6.32-431.17.1.el6.x86_64 #1 Red Hat KVM
22:57:14:RIP: 0010:[<ffffffff8152a5ee>]  [<ffffffff8152a5ee>] _spin_lock+0x1e/0x30
22:57:14:RSP: 0018:ffff88007c48dc20  EFLAGS: 00000206
22:57:14:RAX: 0000000000000002 RBX: ffff88007c48dc20 RCX: 5a5a5a5a5a5a5a5a
22:57:14:RDX: 0000000000000000 RSI: 0000000000000004 RDI: ffff88005302f440
22:57:14:RBP: ffffffff8100bb8e R08: 5a5a5a5a5a5a5a5a R09: 5a5a5a5a5a5a5a5a
22:57:14:R10: 0000000000000000 R11: 0000000000000078 R12: 0000000000000000
22:57:14:R13: ffff88007cbe12d8 R14: 0000000000000000 R15: ffffffff00000000
22:57:14:FS:  0000000000000000(0000) GS:ffff880002300000(0000) knlGS:0000000000000000
22:57:14:CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
22:57:14:CR2: 0000000001eb81c0 CR3: 0000000001a85000 CR4: 00000000000006e0
22:57:14:DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
22:57:14:DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
22:57:14:Process ptlrpcd_0 (pid: 1399, threadinfo ffff88007c48c000, task ffff88007a29d540)
22:57:14:Stack:
22:57:14: ffff88007c48dc80 ffffffffa1083c5b 0000000000000000 ffff880049195f48
22:57:14:<d> ffff88005302f440 ffff880060ac6800 ffff880060ac6800 ffff88007adc0480
22:57:14:<d> ffff880060ac6800 ffff880049195e00 0000000000000000 ffff88007c641400
22:57:14:Call Trace:
22:57:14: [<ffffffffa1083c5b>] ? ll_statahead_interpret+0x7b/0x560 [lustre]
22:57:14: [<ffffffffa0efa736>] ? mdc_intent_getattr_async_interpret+0x1f6/0x540 [mdc]
22:57:14: [<ffffffffa0d3caec>] ? ptlrpc_check_set+0x2bc/0x1b50 [ptlrpc]
22:57:14: [<ffffffffa0d6804b>] ? ptlrpcd_check+0x53b/0x560 [ptlrpc]
22:57:14: [<ffffffffa0d6856b>] ? ptlrpcd+0x20b/0x370 [ptlrpc]
22:57:14: [<ffffffff81061d00>] ? default_wake_function+0x0/0x20
22:57:14: [<ffffffffa0d68360>] ? ptlrpcd+0x0/0x370 [ptlrpc]
22:57:14: [<ffffffff8109ab56>] ? kthread+0x96/0xa0
22:57:14: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
22:57:14: [<ffffffff8109aac0>] ? kthread+0x0/0xa0
22:57:14: [<ffffffff8100c200>] ? child_rip+0x0/0x20
22:57:14:Code: 00 00 00 01 74 05 e8 22 40 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
22:57:14:Call Trace:
22:57:14: [<ffffffffa1083c5b>] ? ll_statahead_interpret+0x7b/0x560 [lustre]
22:57:14: [<ffffffffa0efa736>] ? mdc_intent_getattr_async_interpret+0x1f6/0x540 [mdc]
22:57:14: [<ffffffffa0d3caec>] ? ptlrpc_check_set+0x2bc/0x1b50 [ptlrpc]
22:57:14: [<ffffffffa0d6804b>] ? ptlrpcd_check+0x53b/0x560 [ptlrpc]
22:57:14: [<ffffffffa0d6856b>] ? ptlrpcd+0x20b/0x370 [ptlrpc]
22:57:14: [<ffffffff81061d00>] ? default_wake_function+0x0/0x20
22:57:14: [<ffffffffa0d68360>] ? ptlrpcd+0x0/0x370 [ptlrpc]
22:57:14: [<ffffffff8109ab56>] ? kthread+0x96/0xa0
22:57:14: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
22:57:14: [<ffffffff8109aac0>] ? kthread+0x0/0xa0
22:57:14: [<ffffffff8100c200>] ? child_rip+0x0/0x20
22:57:14:CPU 0 
22:58:35:Modules linked in: lustre(U) obdecho(U) mgc(U) lov(U) osc(U) mdc(U) lmv(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ksocklnd(U) lnet(U) libcfs(U) sha512_generic sha256_generic nfs fscache nfsd lockd nfs_acl auth_rpcgss sunrpc exportfs autofs4 ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa ib_mad ib_core microcode virtio_balloon 8139too 8139cp mii i2c_piix4 i2c_core ext3 jbd mbcache virtio_blk virtio_pci virtio_ring virtio pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: libcfs]
22:58:35:
22:58:35:Pid: 27679, comm: ll_sa_27678 Not tainted 2.6.32-431.17.1.el6.x86_64 #1 Red Hat KVM
22:58:35:RIP: 0010:[<ffffffff8152a5ee>]  [<ffffffff8152a5ee>] _spin_lock+0x1e/0x30
22:58:35:RSP: 0018:ffff880068113d50  EFLAGS: 00000206
22:58:35:RAX: 0000000000000001 RBX: ffff880068113d50 RCX: 0000000000000000
22:58:35:RDX: 0000000000000000 RSI: ffff8800531765c0 RDI: ffff88005302f440
22:58:35:RBP: ffffffff8100bb8e R08: 0000000000000000 R09: 00000000fffffffe
22:58:35:R10: 0000000000000000 R11: 0000000000000001 R12: ffff880068113d30
22:58:35:R13: ffff8800228a4400 R14: 0000000000001000 R15: 0000000000000000
22:58:35:FS:  0000000000000000(0000) GS:ffff880002200000(0000) knlGS:0000000000000000
22:58:35:CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
22:58:35:CR2: 00000000019ea0f8 CR3: 00000000253dd000 CR4: 00000000000006f0
22:58:35:DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
22:58:35:DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
22:58:35:Process ll_sa_27678 (pid: 27679, threadinfo ffff880068112000, task ffff88002879aaa0)
22:58:35:Stack:
22:58:35: ffff880068113dc0 ffffffffa1084cc0 ffff88005302f178 ffff880033595670
22:58:35:<d> 0000000000000000 ffff88005302f080 ffff88005dc395c0 ffff88002513f600
22:58:35:<d> ffff880068113dc0 ffff880028433000 ffff880028433170 ffff88005302f400
22:58:35:Call Trace:
22:58:35: [<ffffffffa1084cc0>] ? ll_post_statahead+0x50/0xa20 [lustre]
22:58:35: [<ffffffffa1088ce8>] ? ll_statahead_thread+0x228/0xfb0 [lustre]
22:58:35: [<ffffffff81061d00>] ? default_wake_function+0x0/0x20
22:58:35: [<ffffffffa1088ac0>] ? ll_statahead_thread+0x0/0xfb0 [lustre]
22:58:35: [<ffffffff8109ab56>] ? kthread+0x96/0xa0
22:58:35: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
22:58:35: [<ffffffff8109aac0>] ? kthread+0x0/0xa0
22:58:35: [<ffffffff8100c200>] ? child_rip+0x0/0x20
22:58:35:Code: 00 00 00 01 74 05 e8 22 40 d6 ff c9 c3 55 48 89 e5 0f 1f 44 00 00 b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8 10 39 c2 74 0e f3 90 <0f> 1f 44 00 00 83 3f 00 75 f4 eb df c9 c3 0f 1f 40 00 55 48 89 
22:58:35:Call Trace:
22:58:35: [<ffffffffa1084cc0>] ? ll_post_statahead+0x50/0xa20 [lustre]
22:58:35: [<ffffffffa1088ce8>] ? ll_statahead_thread+0x228/0xfb0 [lustre]
22:58:35: [<ffffffff81061d00>] ? default_wake_function+0x0/0x20
22:58:35: [<ffffffffa1088ac0>] ? ll_statahead_thread+0x0/0xfb0 [lustre]
22:58:35: [<ffffffff8109ab56>] ? kthread+0x96/0xa0
22:58:35: [<ffffffff8100c20a>] ? child_rip+0xa/0x20
22:58:35: [<ffffffff8109aac0>] ? kthread+0x0/0xa0
22:58:35: [<ffffffff8100c200>] ? child_rip+0x0/0x20

      Info required for matching: lustre-rsync-test 6

      Attachments

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. LU-4410 sanityn test 40a: BUG: soft lockup - CPU#0 stuck for 67s! [ptlrpcd_0:2892]

          • Blocker - Blocks development and/or testing work, production could not run.
          • Resolved

          Activity

            People

              laisiyao Lai Siyao
              maloo Maloo
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: