[LU-6747] Intermittent rc=-EROFS from lod_statfs_and_check

Related — Sat, 13 Feb 2016 14:25:03 +0000

Running mdtest I get failures like this:

06/18/2015 16:22:12: Process 26(zwicky29): FAILED in create_remove_items_helper, unable to create file file.mdtest.26.0 (cwd=/p/lburn/faaland1/zfs-crada/mdtest/2/#test-dir.0/mdtest_tree.26.0): No space left on device
06/18/2015 16:22:12: Process 19(zwicky22): FAILED in create_remove_items_helper, unable to create file file.mdtest.19.0 (cwd=/p/lburn/faaland1/zfs-crada/mdtest/2/#test-dir.0/mdtest_tree.19.0): No space left on device

The three servers involved are using the zfs backend and their pools have lots of free space; all are <1% full.

In the lustre debug log on the MDS, I'm seeing
lod_qos.c:238:lod_statfs_and_check() return -30 (-LUSTRE_EROFS)
lod_qos.c:1016:lod_alloc_rr() return -28 (-ENOSPC)

Many other functions report exiting with -28 as well:

lod_object.c:2104:lod_declare_xattr_set()
lod_object.c:3352:lod_declare_striped_object()
lod_object.c:3384:lod_declare_striped_object()
lod_object.c:3463:lod_declare_object_create()
lod_qos.c:1913:lod_qos_prep_create()
mdd_dir.c:1786:mdd_create_data()
mdd_dir.c:1807:mdd_create_data()
mdd_dir.c:1983:mdd_declare_object_create()
mdd_dir.c:2054:mdd_declare_create()
mdd_dir.c:2354:mdd_create()
mdd_object.c:352:mdd_declare_object_create_internal()
mdt_open.c:1105:mdt_open_by_fid_lock()
mdt_open.c:1255:mdt_reint_open()
mdt_open.c:1374:mdt_reint_open()
mdt_open.c:138:mdt_create_data()
mdt_open.c:347:mdt_mfd_open()
mdt_open.c:607:mdt_finish_open()
mdt_reint.c:1997:mdt_reint_rec()

I've attached a few thousand lines of debug output from the mds with both debug and debug_subsys set to -1. I can reproduce easily, so I can get debug output with specific subsystems turned off or on.

Whamcloud Community JIRA

[LU-6747] Intermittent rc=-EROFS from lod_statfs_and_check