[LU-3624] mds-survey has several bugs

Tue, 31 Dec 2013 15:50:58 +0000

I faced several bugs while running mds-survey tests.

1)
An error message is displayed by mds-survey at directory creation step (lctl test_mkdir).
"ostid_set_id() Bad 18446744073709551615 to set 0:0"

Here are the step by step commands executed by mds-survey.

# lctl dl
  0 UP osd-ldiskfs fs2-MDT0000-osd fs2-MDT0000-osd_UUID 11
  1 UP mgc MGC30.1.0.95@o2ib 21007e3d-b2b2-494c-8c9e-a536f037ee6d 5
  2 UP mds MDS MDS_uuid 3
  3 UP lod fs2-MDT0000-mdtlov fs2-MDT0000-mdtlov_UUID 4
  4 UP mdt fs2-MDT0000 fs2-MDT0000_UUID 15
  5 UP mdd fs2-MDD0000 fs2-MDD0000_UUID 4
  6 UP qmt fs2-QMT0000 fs2-QMT0000_UUID 4
  7 UP lwp fs2-MDT0000-lwp-MDT0000 fs2-MDT0000-lwp-MDT0000_UUID 5
  8 UP osp fs2-OST0003-osc-MDT0000 fs2-MDT0000-mdtlov_UUID 5
  9 UP osp fs2-OST0002-osc-MDT0000 fs2-MDT0000-mdtlov_UUID 5
 10 UP osp fs2-OST0001-osc-MDT0000 fs2-MDT0000-mdtlov_UUID 5
 11 UP osp fs2-OST0000-osc-MDT0000 fs2-MDT0000-mdtlov_UUID 5
# modprobe obdecho
# lctl << EOF
> attach echo_client fs2-MDT0000_ecc fs2-MDT0000_ecc_UUID
> setup fs2-MDT0000 mdd
> EOF
# lctl --device 12 test_mkdir /test0
(/homes/pichong/SB/AE4_kernel38/obj/x86_64_bullxlinux6.3/topdir/BUILD/lustre-2.4.0/lustre/include/lustre/lustre_idl.h:683:ostid_set_id()) Bad 18446744073709551615 to set 0:0

2)
The get_global_stats() subroutine of mds-survey does not correctly aggregate results from each MDT.

get_global_stats () {
    local rfile=$1
    awk < $rfile                                               \
    'BEGIN {n = 0;}                                            \
    {    n++;                                                  \
         if (n == 1) { err = $1; ave = $2; min = $3; max = $4} \
         else                                                  \
         { if ($1 < err) err = $1;                             \
           if ($2 < min) min = $2;                             \
           if ($3 > max) max = $3;                             \
         }                                                     \
    }                                                          \
    END { if (n == 0) err = 0;                                 \
          printf "%d %f %f %f\n", err, ave, min, max}'
}

should be

get_global_stats () {
    local rfile=$1
    awk < $rfile                                               \
    'BEGIN {n = 0;}                                            \
    {    n++;                                                  \
         if (n == 1) { err = $1; ave = $2; min = $3; max = $4} \
         else                                                  \
         { if ($1 < err) err = $1;                             \
           ave += $2;                                          \
           if ($3 < min) min = $3;                             \
           if ($4 > max) max = $4;                             \
         }                                                     \
    }                                                          \
    END { if (n == 0) err = 0;                                 \
          printf "%d %f %f %f\n", err, ave/n, min, max}'
}

I am going to provide a patch.

3)
When stripe_count is positive, the mds-survey "destroy" action does not free the objects that were created on the OSTs. As a result, after several runs of mds-survey, the OSTs return ENOSPC at file creation and "lctl test_create" hangs with the following stack.

[<ffffffffa0171731>] cfs_waitq_timedwait+0x11/0x20 [libcfs]
[<ffffffffa0e74154>] osp_precreate_reserve+0x5c4/0x1ee0 [osp]
[<ffffffffa0e6dc55>] osp_declare_object_create+0x155/0x4f0 [osp]
[<ffffffffa0b7438d>] lod_qos_declare_object_on+0xed/0x480 [lod]
[<ffffffffa0b75f0f>] lod_alloc_rr.clone.2+0x66f/0xde0 [lod]
[<ffffffffa0b77b69>] lod_qos_prep_create+0xfa9/0x1b14 [lod]
[<ffffffffa0b71cab>] lod_declare_striped_object+0x14b/0x880 [lod]
[<ffffffffa0b72df3>] lod_declare_xattr_set+0x273/0x410 [lod]
[<ffffffffa0547700>] mdo_declare_xattr_set.clone.4+0x40/0xe0 [mdd]
[<ffffffffa054a470>] mdd_declare_create+0x4b0/0x860 [mdd]
[<ffffffffa054afb1>] mdd_create+0x791/0x1740 [mdd]
[<ffffffffa0ea8fef>] echo_md_create_internal+0x1cf/0x640 [obdecho]
[<ffffffffa0eb2b43>] echo_md_handler+0x1333/0x1ac0 [obdecho]
[<ffffffffa0eb7257>] echo_client_iocontrol+0x2dc7/0x3b40 [obdecho]
[<ffffffffa060849f>] class_handle_ioctl+0x12ff/0x1ec0 [obdclass]
[<ffffffffa05f02ab>] obd_class_ioctl+0x4b/0x190 [obdclass]
[<ffffffff81181372>] vfs_ioctl+0x22/0xa0
[<ffffffff81181514>] do_vfs_ioctl+0x84/0x580
[<ffffffff81181a91>] sys_ioctl+0x81/0xa0
[<ffffffff81003072>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff

Whamcloud Community JIRA

[LU-3624] mds-survey has several bugs