Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-20941

tests: conf-sanity reformat fails on dm-flakey devices left by an earlier run

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Medium Medium
    • None
    • None
    • None
    • 3
    • 9223372036854775807

      conf-sanity fails every test at its first reformat, before any subtest runs, when the dm-flakey devices of an earlier run on the same nodes are still in place:

      Formatting mgs, mds, osts
      Format mds1: /dev/vdb
      co5-mds: mkfs.lustre FATAL: Unable to build fs /dev/vdb (256)
      co5-mds: mkfs.lustre FATAL: mkfs failed 256
      

      dmsetup ls on the MDS still shows mds1_flakey and mds2_flakey as linear maps over /dev/vdb and /dev/vdc, with nothing mounted, so mkfs on the raw disk fails. Removing them by hand (dmsetup remove) lets the run proceed. Seen on master (2.17.59) with an ltvm 3-node cluster (separate MDS, OSS, client, real block devices, no failover hosts), running conf-sanity twice in a row, or after llmount.sh, without llmountcleanup.sh in between.

      Cause

      • stop() only calls dm_cleanup_dev() when the facet has a distinct failover host, CLEANUP_DM_DEV is true, or the facet is fs*. init_test_env sets CLEANUP_DM_DEV=false, so in a non-failover setup the mapper normally outlives stop() and is reused by the next mount_facet(), which is intended.
      • check_and_setup_lustre() and setupall() reformat with CLEANUP_DM_DEV=true formatall, which removes those mappers first. But reformat() in conf-sanity-framework.sh calls plain formatall, so a conf-sanity started on nodes where a previous run (any suite, or llmount.sh) left its mappers runs mkfs on the raw device underneath them. A new test process also starts with ${facet}_dev pointing at the raw device, so it never learns that the mapper exists.

      LU-10893 fixed the same mkfs failure in 2018; this path still bypasses it.

      Related: dm_cleanup_dev() leaves the facet device exported when the mapper is gone

      dm_cleanup_dev() {
      	...
      	is_dm_dev $facet $dm_dev || return 0
      	...
      	unexport_dm_dev $facet
      

      When the mapper was already removed (by another process, or by a stop() in a subshell, whose unexport_dm_dev() only changes that subshell), dm_cleanup_dev() returns before unexport_dm_dev(), so ${facet}_dev keeps the /dev/mapper path and the next start mounts a device that no longer exists. With CLEANUP_DM_DEV=true set in cfg/local.sh, ltvm saw sanity test_160h and test_278 fail this way on every run (ltvm commit b8d9abd121b1).

      Suggested fix

      • conf-sanity-framework.sh reformat(): CLEANUP_DM_DEV=true formatall, as check_and_setup_lustre() does, or have formatall() remove any existing ${facet}_flakey map of the facets it formats before running mkfs.
      • dm_cleanup_dev(): call unexport_dm_dev() also when is_dm_dev finds no mapper.

      Workaround: run llmountcleanup.sh (cleanupall, which uses CLEANUP_DM_DEV=true) before starting conf-sanity, as ltvm's suite run does.

            wc-triage WC Triage
            adilger Andreas Dilger
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: