Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-10753

sanity test 300c fails with 'create 5k files failed'

    Details

    • Type: Bug
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: Lustre 2.10.1, Lustre 2.11.0, Lustre 2.12.0, Lustre 2.13.0, Lustre 2.14.0
    • Fix Version/s: None
    • Labels:
    • Severity:
      3
    • Rank (Obsolete):
      9223372036854775807

      Description

      sanity test_300c fails to create 5,000 files for Lustre file systems with DNE configured and ZFS server targets. Actually, it can’t even create one file. The client test log has the following output

      == sanity test 300c: chown && check ls under striped directory ======================================= 07:16:38 (1508915798)
      CMD: trevis-49vm7 /usr/sbin/lctl get_param -n version 2>/dev/null ||
      				/usr/sbin/lctl lustre_build_version 2>/dev/null ||
      				/usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2
      running as uid/gid/euid/egid 500/500/500/500, groups:
       [createmany] [-o] [/mnt/lustre/d300c.sanity/striped_dir/f] [5000]
      open(/mnt/lustre/d300c.sanity/striped_dir/f0) error: Permission denied
      total: 0 open/close in 0.01 seconds: 0.00 ops/second
       sanity test_300c: @@@@@@ FAIL: create 5k files failed
      

      There’s nothing interesting in the console and dmesg logs on any of the nodes.

      This test started failing with the ‘create 5k files failed’ error on 2017-09-29 for master tag 2.10.53. We’ve also seen this fail once for full test session DNE/ZFS testing on 2.10.1.

      Logs for test sessions with this failure are at:
      https://testing.hpdd.intel.com/test_sets/7ce33f5a-ba7b-11e7-9abd-52540065bddc
      https://testing.hpdd.intel.com/test_sets/336e7196-12b9-11e8-a10a-52540065bddc
      https://testing.hpdd.intel.com/test_sets/d61ba008-167e-11e8-bd00-52540065bddc

      We see the other sanity 300* tests fail in similar ways and can open separate tickets if we think the following have distinct root cause for the failures.

      sanity test 300d fails with

      == sanity test 300d: check default stripe under striped directory ==================================== 06:31:06 (1516775466)
      CMD: trevis-13vm4 /usr/sbin/lctl get_param -n version 2>/dev/null ||
      				/usr/sbin/lctl lustre_build_version 2>/dev/null ||
      				/usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2
      open(/mnt/lustre/d300d.sanity/striped_dir/f1) error: Permission denied
      total: 1 open/close in 0.01 seconds: 189.21 ops/second
       sanity test_300d: @@@@@@ FAIL: create 10 files failed 
      

      with logs at
      https://testing.hpdd.intel.com/test_sets/46afbcfe-00e6-11e8-a10a-52540065bddc

      sanity test 300e fails with

      == sanity test 300e: check rename under striped directory ============================================ 01:22:13 (1519521733)
      CMD: trevis-66vm8 /usr/sbin/lctl get_param -n version 2>/dev/null ||
      				/usr/sbin/lctl lustre_build_version 2>/dev/null ||
      				/usr/sbin/lctl --version 2>/dev/null | cut -d' ' -f2
      touch: cannot touch '/mnt/lustre/d300e.sanity/striped_dir/a': Permission denied
      touch: cannot touch '/mnt/lustre/d300e.sanity/striped_dir/c': Permission denied
      mkdir: cannot create directory '/mnt/lustre/d300e.sanity/striped_dir/dir_a': Permission denied
      mkdir: cannot create directory '/mnt/lustre/d300e.sanity/striped_dir/dir_c': Permission denied
      rename returned -1: No such file or directory
       sanity test_300e: @@@@@@ FAIL: rename dir under striped dir fails 
      

      with logs at:
      https://testing.hpdd.intel.com/test_sets/cceb366a-19df-11e8-a6ad-52540065bddc
      https://testing.hpdd.intel.com/test_sets/46afbcfe-00e6-11e8-a10a-52540065bddc
      https://testing.hpdd.intel.com/test_sets/4b90ec14-ffb7-11e7-bd00-52540065bddc

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                wc-triage WC Triage
                Reporter:
                jamesanunez James Nunez
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated: