Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-1897

Failure on test suite replay-single, test_70b: dbench not found on some of the test nodes

Details

    • Bug
    • Resolution: Cannot Reproduce
    • Blocker
    • Lustre 2.4.0
    • Lustre 2.4.0, Lustre 2.1.6, Lustre 2.8.0
    • 3
    • 5743

    Description

      This issue was created by maloo for jay <jay@whamcloud.com>

      This issue relates to the following test suite run: https://maloo.whamcloud.com/test_sets/c4620b6c-fc31-11e1-a4a6-52540035b04c.

      The sub-test test_70b failed with the following error:

      dbench not found on some of client-28vm1,client-28vm2.lab.whamcloud.com !

      Info required for matching: replay-single 70b

      Attachments

        Issue Links

          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. LU-2120 Test failure on test suite replay-single, subtest test_70b

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          mentioned in

          [Whamcloud Community Wiki] Page Page No Confluence page found with the given URL.

          [Whamcloud Community Wiki] Page Page No Confluence page found with the given URL.

          [Whamcloud Community Wiki] Page Page No Confluence page found with the given URL.

          [Whamcloud Community Wiki] Page Page No Confluence page found with the given URL.

          Activity

            [LU-1897] Failure on test suite replay-single, test_70b: dbench not found on some of the test nodes
            yujian Jian Yu added a comment -

            Hi Andreas,

            Is there a different ticket tracking the "no space left on device" failure?

            Yes, it's LU-4846. And the original issue in this ticket is not related to "No space left on device", so we do not need link this ticket to LU-4846.

            yujian Jian Yu added a comment - Hi Andreas, Is there a different ticket tracking the "no space left on device" failure? Yes, it's LU-4846 . And the original issue in this ticket is not related to "No space left on device", so we do not need link this ticket to LU-4846 .
            adilger Andreas Dilger added a comment -

            Sarah,
            Is there a different ticket tracking the "no space left on device" failure? If this problem is still happening, but this bug is closed, then either this bug needs to be linked to an existing ticket or a new ticket opened up to track these failures.

            adilger Andreas Dilger added a comment - Sarah, Is there a different ticket tracking the "no space left on device" failure? If this problem is still happening, but this bug is closed, then either this bug needs to be linked to an existing ticket or a new ticket opened up to track these failures.
            sarah Sarah Liu added a comment -

            close this ticket as the recent instances on master (tag-2.7.63 and tag-2.7.64) are caused by the "no space left on device"

            shadow-52vm6: [769] open ./clients/client0/~dmtmp/PWRPNT/PPTC112.TMP failed for handle 10033 (No space left on device)
shadow-52vm6: (770) ERROR: handle 10033 was not found
shadow-52vm6: Child failed with status 1
shadow-52vm6: touch: missing file operand
shadow-52vm6: Try 'touch --help' for more information.
shadow-52vm6: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 21 sec
shadow-52vm1:    1       733     0.17 MB/sec  warmup 124 sec  latency 53560.001 ms
shadow-52vm5:    1       791     0.17 MB/sec  warmup 124 sec  latency 54097.359 ms
shadow-52vm1:    1       733     0.17 MB/sec  warmup 125 sec  latency 1340.522 ms
shadow-52vm5:    1       794     0.17 MB/sec  warmup 125 sec  latency 54380.844 ms
CMD: shadow-52vm1.shadow.whamcloud.com,shadow-52vm5,shadow-52vm6 killall -0 dbench
shadow-52vm6: dbench: no process found
 replay-single test_70b: @@@@@@ FAIL: dbench stopped on some of shadow-52vm1.shadow.whamcloud.com,shadow-52vm5,shadow-52vm6! 
shadow-52vm1:    1       733     0.17 MB/sec  warmup 126 sec  latency 2340.713 ms
shadow-52vm5:    1       794     0.17 MB/sec  warmup 126 sec  latency 1715.056 ms
            sarah Sarah Liu added a comment - close this ticket as the recent instances on master (tag-2.7.63 and tag-2.7.64) are caused by the "no space left on device" shadow-52vm6: [769] open ./clients/client0/~dmtmp/PWRPNT/PPTC112.TMP failed for handle 10033 (No space left on device) shadow-52vm6: (770) ERROR: handle 10033 was not found shadow-52vm6: Child failed with status 1 shadow-52vm6: touch: missing file operand shadow-52vm6: Try 'touch --help' for more information. shadow-52vm6: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 21 sec shadow-52vm1: 1 733 0.17 MB/sec warmup 124 sec latency 53560.001 ms shadow-52vm5: 1 791 0.17 MB/sec warmup 124 sec latency 54097.359 ms shadow-52vm1: 1 733 0.17 MB/sec warmup 125 sec latency 1340.522 ms shadow-52vm5: 1 794 0.17 MB/sec warmup 125 sec latency 54380.844 ms CMD: shadow-52vm1.shadow.whamcloud.com,shadow-52vm5,shadow-52vm6 killall -0 dbench shadow-52vm6: dbench: no process found replay-single test_70b: @@@@@@ FAIL: dbench stopped on some of shadow-52vm1.shadow.whamcloud.com,shadow-52vm5,shadow-52vm6! shadow-52vm1: 1 733 0.17 MB/sec warmup 126 sec latency 2340.713 ms shadow-52vm5: 1 794 0.17 MB/sec warmup 126 sec latency 1715.056 ms
            standan Saurabh Tandan (Inactive) added a comment -

            master, build# 3266, 2.7.64 tag
            Hard Failover:EL7 Server/SLES11 SP3 Client
            https://testing.hpdd.intel.com/test_sets/a8b3fb9e-a077-11e5-8d69-5254006e85c2

            standan Saurabh Tandan (Inactive) added a comment - master, build# 3266, 2.7.64 tag Hard Failover:EL7 Server/SLES11 SP3 Client https://testing.hpdd.intel.com/test_sets/a8b3fb9e-a077-11e5-8d69-5254006e85c2
            standan Saurabh Tandan (Inactive) added a comment -

            master, build# 3264, 2.7.64 tag
            Hard Failover: EL7 Server/Client - ZFS
            https://testing.hpdd.intel.com/test_sets/5ba6d7bc-9e20-11e5-91b0-5254006e85c2

            standan Saurabh Tandan (Inactive) added a comment - master, build# 3264, 2.7.64 tag Hard Failover: EL7 Server/Client - ZFS https://testing.hpdd.intel.com/test_sets/5ba6d7bc-9e20-11e5-91b0-5254006e85c2
            standan Saurabh Tandan (Inactive) added a comment -

            Instance found in recent tags 2.7.63, 2.7.64

            standan Saurabh Tandan (Inactive) added a comment - Instance found in recent tags 2.7.63, 2.7.64
            standan Saurabh Tandan (Inactive) added a comment -

            master, build# 3264, 2.7.64 tag
            Hard Failover: EL6.7 Server/Client
            https://testing.hpdd.intel.com/test_sets/80a20678-9edd-11e5-87a9-5254006e85c2

            standan Saurabh Tandan (Inactive) added a comment - master, build# 3264, 2.7.64 tag Hard Failover: EL6.7 Server/Client https://testing.hpdd.intel.com/test_sets/80a20678-9edd-11e5-87a9-5254006e85c2
            yujian Jian Yu added a comment -

            The issue also exists on Lustre b2_1 branch:
            https://maloo.whamcloud.com/test_sets/a0806aa0-c5ce-11e3-9255-52540035b04c

            yujian Jian Yu added a comment - The issue also exists on Lustre b2_1 branch: https://maloo.whamcloud.com/test_sets/a0806aa0-c5ce-11e3-9255-52540035b04c
            pjones Peter Jones added a comment -

            Landed for 2.4

            pjones Peter Jones added a comment - Landed for 2.4
            keith Keith Mannthey (Inactive) added a comment -

            http://review.whamcloud.com/5761 version 1 of the patch has been submitted.

            keith Keith Mannthey (Inactive) added a comment - http://review.whamcloud.com/5761 version 1 of the patch has been submitted.

            People

              keith Keith Mannthey (Inactive)
              maloo Maloo
              Votes:
              1 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: