-
Bug
-
Resolution: Unresolved
-
Medium
-
None
-
None
-
None
-
3
-
9223372036854775807
conf-sanity fails every test at its first reformat, before any subtest runs, when the dm-flakey devices of an earlier run on the same nodes are still in place:
Formatting mgs, mds, osts Format mds1: /dev/vdb co5-mds: mkfs.lustre FATAL: Unable to build fs /dev/vdb (256) co5-mds: mkfs.lustre FATAL: mkfs failed 256
dmsetup ls on the MDS still shows mds1_flakey and mds2_flakey as linear maps over /dev/vdb and /dev/vdc, with nothing mounted, so mkfs on the raw disk fails. Removing them by hand (dmsetup remove) lets the run proceed. Seen on master (2.17.59) with an ltvm 3-node cluster (separate MDS, OSS, client, real block devices, no failover hosts), running conf-sanity twice in a row, or after llmount.sh, without llmountcleanup.sh in between.
Cause
- stop() only calls dm_cleanup_dev() when the facet has a distinct failover host, CLEANUP_DM_DEV is true, or the facet is fs*. init_test_env sets CLEANUP_DM_DEV=false, so in a non-failover setup the mapper normally outlives stop() and is reused by the next mount_facet(), which is intended.
- check_and_setup_lustre() and setupall() reformat with CLEANUP_DM_DEV=true formatall, which removes those mappers first. But reformat() in conf-sanity-framework.sh calls plain formatall, so a conf-sanity started on nodes where a previous run (any suite, or llmount.sh) left its mappers runs mkfs on the raw device underneath them. A new test process also starts with ${facet}_dev pointing at the raw device, so it never learns that the mapper exists.
LU-10893 fixed the same mkfs failure in 2018; this path still bypasses it.
Related: dm_cleanup_dev() leaves the facet device exported when the mapper is gone
dm_cleanup_dev() {
...
is_dm_dev $facet $dm_dev || return 0
...
unexport_dm_dev $facet
When the mapper was already removed (by another process, or by a stop() in a subshell, whose unexport_dm_dev() only changes that subshell), dm_cleanup_dev() returns before unexport_dm_dev(), so ${facet}_dev keeps the /dev/mapper path and the next start mounts a device that no longer exists. With CLEANUP_DM_DEV=true set in cfg/local.sh, ltvm saw sanity test_160h and test_278 fail this way on every run (ltvm commit b8d9abd121b1).
Suggested fix
- conf-sanity-framework.sh reformat(): CLEANUP_DM_DEV=true formatall, as check_and_setup_lustre() does, or have formatall() remove any existing ${facet}_flakey map of the facets it formats before running mkfs.
- dm_cleanup_dev(): call unexport_dm_dev() also when is_dm_dev finds no mapper.
Workaround: run llmountcleanup.sh (cleanupall, which uses CLEANUP_DM_DEV=true) before starting conf-sanity, as ltvm's suite run does.