Thanks Stephane
Just upgraded the servers to 2.10.2 and we'll see how it goes (right now I can't reproduce the issue).
I'm also noticing the following lquota log messages (also reported in -LU-9790
00040000:02020000:27.0F:1513208262.246875:0:257335:0:(qsd_config.c:202:qsd_process_config()) 0-0: oak-OST003c: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already 00040000:02020000:27.0:1513208262.272624:0:257335:0:(qsd_config.c:202:qsd_process_config()) 0-0: oak-OST0038: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already 00040000:02020000:27.0:1513208262.272627:0:257335:0:(qsd_config.c:202:qsd_process_config()) 0-0: oak-OST0034: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already 00040000:02020000:27.0:1513208262.272628:0:257335:0:(qsd_config.c:202:qsd_process_config()) 0-0: oak-OST0042: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already 00040000:02020000:27.0:1513208262.272629:0:257335:0:(qsd_config.c:202:qsd_process_config()) 0-0: oak-OST0046: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already 00040000:02020000:27.0:1513208262.272630:0:257335:0:(qsd_config.c:202:qsd_process_config()) 0-0: oak-OST003a: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already 00040000:02020000:27.0:1513208262.272631:0:257335:0:(qsd_config.c:202:qsd_process_config()) 0-0: oak-OST004c: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already 00040000:02020000:27.0:1513208262.272632:0:257335:0:(qsd_config.c:202:qsd_process_config()) 0-0: oak-OST004a: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
Our setup is: space acct on ug (no project quota, ie. no ldiskfs flag) and quota only enabled for groups (g):
target name: oak-OST004c pool ID: 0 type: dt quota enabled: g conn to master: setup space acct: ug user uptodate: glb[0],slv[0],reint[0] group uptodate: glb[1],slv[1],reint[0] project uptodate: glb[0],slv[0],reint[0]
Best,
Stephane
https://testing.hpdd.intel.com/test_sets/9488b58c-f1b6-11e7-a169-52540065bddc
see similar message 'can't enable quota enforcement'
Hongchao
Can you please investigate?
Thanks
Peter
the bug in https://testing.hpdd.intel.com/test_sets/9488b58c-f1b6-11e7-a169-52540065bddc is not the same issue,
it's caused by the delayed response from MDT and client reconnected to MDT, which cause the MDS_QUOTACTL
request in "mdc_quotactl" to fail for it is marked as "no resend".
static int mdc_quotactl(struct obd_device *unused, struct obd_export *exp,
struct obd_quotactl *oqctl)
{
struct ptlrpc_request *req;
struct obd_quotactl *oqc;
int rc;
ENTRY;
req = ptlrpc_request_alloc_pack(class_exp2cliimp(exp),
&RQF_MDS_QUOTACTL, LUSTRE_MDS_VERSION,
MDS_QUOTACTL);
if (req == NULL)
RETURN(-ENOMEM);
oqc = req_capsule_client_get(&req->rq_pill, &RMF_OBD_QUOTACTL);
*oqc = *oqctl;
ptlrpc_request_set_replen(req);
ptlrpc_at_set_req_timeout(req);
req->rq_no_resend = 1; <--- here
rc = ptlrpc_queue_wait(req);
if (rc)
CERROR("ptlrpc_queue_wait failed, rc: %d\n", rc);
if (req->rq_repmsg &&
(oqc = req_capsule_server_get(&req->rq_pill, &RMF_OBD_QUOTACTL))) {
*oqctl = *oqc;
} else if (!rc) {
CERROR ("Can't unpack obd_quotactl\n");
rc = -EPROTO;
}
ptlrpc_req_finished(req);
RETURN(rc);
}
the corresponding logs are
[ 8535.569851] Lustre: DEBUG MARKER: == sanity-quota test 8: Run dbench with quota enabled ================================================ 22:08:14 (1515103694) [ 9410.140055] Lustre: 19972:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1515103967/real 1515103967] req@ffff88006518cc00 x1588697858744704/t0(0) o48->lustre-MDT0000-mdc-ffff88006b606800@10.9.4.248@tcp:12/10 lens 336/336 e 21 to 1 dl 1515104568 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [ 9410.148998] Lustre: lustre-MDT0000-mdc-ffff88006b606800: Connection to lustre-MDT0000 (at 10.9.4.248@tcp) was lost; in progress operations using this service will wait for recovery to complete [ 9410.278389] Lustre: lustre-MDT0000-mdc-ffff88006b606800: Connection restored to 10.9.4.248@tcp (at 10.9.4.248@tcp) [ 9410.281474] LustreError: 19972:0:(mdc_request.c:1840:mdc_quotactl()) ptlrpc_queue_wait failed, rc: -107 [ 9410.462846] Lustre: DEBUG MARKER: /usr/sbin/lctl mark sanity-quota test_8: @@@@@@ FAIL: clear quota for [type:-u name:quota_usr] failed
Hongchao Zhang (hongchao.zhang@intel.com) uploaded a new patch: https://review.whamcloud.com/31773
Subject: LU-10368 mdc: resend quotactl if needed
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 30466952d14f2e891f3a1bdd29103ae578f00413
Oleg Drokin (oleg.drokin@intel.com) merged in patch https://review.whamcloud.com/31773/
Subject: LU-10368 mdc: resend quotactl if needed
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: d511918e8eb725abba2561cc493e30651a89ac27
Stephane
I'm not sure whether it is easy for you to test out this fix and confirm whether it resolves the issue, but if you do have a way then that would be much appreciated. For now I'll mark this ticket as resolved - "innocent until proven guilty" 
- and we'll queue it up for a future 2.10.x release.
Peter
Minh Diep (minh.diep@intel.com) uploaded a new patch: https://review.whamcloud.com/32075
Subject: LU-10368 mdc: resend quotactl if needed
Project: fs/lustre-release
Branch: b2_10
Current Patch Set: 1
Commit: e90b1a190b38db9573a5284ca53efbff612d0972
Hi Peter,
I'll try the patch at the first opportunity after LUG.
Please also note that we haven't noticed any new occurence of this issue in 2.10.3 so far.
Thanks!
Stephane
John L. Hammond (john.hammond@intel.com) merged in patch https://review.whamcloud.com/32075/
Subject: LU-10368 mdc: resend quotactl if needed
Project: fs/lustre-release
Branch: b2_10
Current Patch Set:
Commit: c2c8a3f6dec17144f317aab409f48b862d9aa1b1