[LU-16623] lod_statfs_and_check() does not skip unusable OSTs Created: 07/Mar/23 Updated: 23/Jan/24 Resolved: 14/Jun/23 |
|
| Status: | Resolved |
| Project: | Lustre |
| Component/s: | None |
| Affects Version/s: | Lustre 2.16.0 |
| Fix Version/s: | Lustre 2.16.0 |
| Type: | Bug | Priority: | Major |
| Reporter: | Andreas Dilger | Assignee: | Andreas Dilger |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Issue Links: |
|
||||||||||||||||||||||||||||||||
| Severity: | 3 | ||||||||||||||||||||||||||||||||
| Rank (Obsolete): | 9223372036854775807 | ||||||||||||||||||||||||||||||||
| Description |
|
The LOV_USES_DEFAULT_STRIPE flag is never set during OST object allocation, so min_stripe_count() does not allow reducing the stripe count set on a file. This can result in the MDS trying to allocate objects on too many OSTs, especially when the space is imbalanced, or OSTs are deactivated by "osp.*.max_create_count=0". It also appears that lod_statfs_and_check() is expecting lod_statfs_check() to only return -ENOSPC, so it isn't skipping an OST when it is mounted read-only or when max_create_count=0. It would probably be better to change lod_statfs_and_check() to skip the OST if lod_statfs_check() returns any error, since all of them are reasons to skip an OST. Also, it appears that the merged MDT and OST QOS allocation does skip MDTs marked with OS_STATFS_ENOINO. |
| Comments |
| Comment by Andreas Dilger [ 07/Mar/23 ] |
|
I think I see a larger problem with the way that the lod_statfs_check() is being handled. Not only did lod_statfs_and_check() only check for -ENOSPC being returned (which wasn't true when OS_STATFS_NOPRECREATE was returned when max_create_count=0 is set), but for any other return code, the OST would be marked inactive (ltd_active = 0) in the later part of this function, which means that OST_DESTROY RPCs would not be sent and in general the MDS would stop sending any RPCs to this OST: (lod_qos.c:200:lod_statfs_and_check()) Process entered (osp_dev.c:779:osp_statfs()) testfs-OST0000-osc-MDT0000: 78276 blocks, 76593 free, 69736 avail, 100000 files, 98890 free files (lod_qos.c:236:lod_statfs_and_check()) testfs-OST0000-osc-MDT0000: turns inactive ******* (lod_qos.c:263:lod_statfs_and_check()) Process leaving (rc=18446744073709551511 : -105 : ffffffffffffff97) |
| Comment by Andreas Dilger [ 08/Mar/23 ] |
|
OK, I may have confused ltd_active=0 with deactivating the whole device with "lctl set_param osp.*.active=0". Setting max_create_count=0 does set ltd_active=0 and reduce the value reported by lod.*.activeobd once some file tries to be created on that MDT, which is also done by setting "lctl set_param osp.*.active=0. One open question is why that is done for OS_STATE_NOPRECREATE but not consistently OS_STATE_ENOSPC or OS_STATE_ENOINO? Also, the ldt_active field is used for other things in the LOD code, for example controlling whether lod_sync() will send a sync RPC to the OST/MDT, and controls llog config and set_info calls. |
| Comment by Gerrit Updater [ 10/Mar/23 ] |
|
"Andreas Dilger <adilger@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/50250 |
| Comment by Gerrit Updater [ 14/Jun/23 ] |
|
"Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/50250/ |
| Comment by Peter Jones [ 14/Jun/23 ] |
|
Landed for 2.16 |
| Comment by Gerrit Updater [ 09/Jan/24 ] |
|
"Andreas Dilger <adilger@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/53628 |
| Comment by Gerrit Updater [ 23/Jan/24 ] |
|
"Jian Yu <yujian@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/53774 |