[3072117.458377] LustreError: 35144:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 34 previous similar messages [3072723.005751] LustreError: 36687:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3072723.035775] LustreError: 36687:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 32 previous similar messages [3073343.101989] LustreError: 35145:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107814850800 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3073343.131823] LustreError: 35145:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 34 previous similar messages [3073962.255198] LustreError: 35174:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3073962.285218] LustreError: 35174:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 33 previous similar messages [3074579.340664] LustreError: 35148:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3074579.370340] LustreError: 35148:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 33 previous similar messages [3075188.498893] LustreError: 35203:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3075188.528744] LustreError: 35203:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 33 previous similar messages [3075807.337325] LustreError: 35192:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107814850800 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3075807.367003] LustreError: 35192:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 31 previous similar messages [3076416.124256] LustreError: 35182:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107814850800 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3076416.153933] LustreError: 35182:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 32 previous similar messages [3076632.941989] Lustre: 31360:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1713340068/real 1713340068] req@ffff903d222fc380 x1793446854266304/t0(0) o6->oak-OST0033-osc-MDT0000@10.0.2.106@o2ib5:28/4 lens 544/432 e 20 to 1 dl 1713340743 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [3076632.971227] Lustre: oak-OST0033-osc-MDT0000: Connection to oak-OST0033 (at 10.0.2.106@o2ib5) was lost; in progress operations using this service will wait for recovery to complete [3076632.997928] Lustre: oak-OST0033-osc-MDT0000: Connection restored to 10.0.2.106@o2ib5 (at 10.0.2.106@o2ib5) [3077028.992826] LustreError: 35225:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3077029.023017] LustreError: 35225:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 31 previous similar messages [3077642.228084] LustreError: 36657:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3077642.257761] LustreError: 36657:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 34 previous similar messages [3078249.768889] LustreError: 35206:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3078249.798736] LustreError: 35206:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 30 previous similar messages [3078865.972486] LustreError: 35184:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3078866.002509] LustreError: 35184:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 35 previous similar messages [3079485.105847] LustreError: 35179:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3079485.135525] LustreError: 35179:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 33 previous similar messages [3080107.629325] LustreError: 35227:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3080107.659344] LustreError: 35227:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 33 previous similar messages [3080712.699694] LustreError: 36661:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3080712.729552] LustreError: 36661:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 35 previous similar messages [3081320.605384] LustreError: 36685:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3081320.635402] LustreError: 36685:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 35 previous similar messages [3081924.046206] LustreError: 35183:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3081924.076051] LustreError: 35183:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 36 previous similar messages [3082537.494370] LustreError: 35152:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3082537.524224] LustreError: 35152:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 35 previous similar messages [3083155.287286] LustreError: 36659:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3083155.317135] LustreError: 36659:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 36 previous similar messages [3083765.851831] LustreError: 36669:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3083765.881506] LustreError: 36669:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 32 previous similar messages [3084375.790899] LustreError: 36683:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3084375.821045] LustreError: 36683:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 32 previous similar messages [3084977.781949] LustreError: 36670:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3084977.811799] LustreError: 36670:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 33 previous similar messages [3085578.031894] LustreError: 36668:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3085578.061742] LustreError: 36668:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 31 previous similar messages [3086183.700601] LustreError: 35199:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3086183.730273] LustreError: 35199:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 34 previous similar messages [3086796.048036] LustreError: 35153:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3086796.077998] LustreError: 35153:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 35 previous similar messages [3087400.334376] LustreError: 35162:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3087400.364397] LustreError: 35162:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 35 previous similar messages [3088003.199643] LustreError: 36654:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3088003.229496] LustreError: 36654:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 34 previous similar messages [3088607.564552] LustreError: 36667:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3088607.594412] LustreError: 36667:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 35 previous similar messages [3089214.633223] LustreError: 35154:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3089214.663066] LustreError: 35154:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 33 previous similar messages [3089816.866209] LustreError: 36662:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3089816.896060] LustreError: 36662:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 35 previous similar messages [3090416.877477] LustreError: 36664:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3090416.907326] LustreError: 36664:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 39 previous similar messages [3091021.673106] LustreError: 36682:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3091021.703126] LustreError: 36682:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 34 previous similar messages [3091631.143922] LustreError: 36688:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3091631.173946] LustreError: 36688:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 36 previous similar messages [3092231.189231] LustreError: 36700:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3092231.219248] LustreError: 36700:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 36 previous similar messages [3092855.329742] LustreError: 35157:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3092855.359418] LustreError: 35157:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 36 previous similar messages [3093458.546463] LustreError: 35218:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3093458.576475] LustreError: 35218:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 36 previous similar messages [3094069.494579] LustreError: 35144:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107814850800 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3094069.524259] LustreError: 35144:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 33 previous similar messages [3094710.123455] LustreError: 35229:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3094710.153469] LustreError: 35229:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 38 previous similar messages [3095313.063957] LustreError: 35179:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3095313.093809] LustreError: 35179:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 37 previous similar messages [3095924.451490] LustreError: 35147:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107814850800 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3095924.481167] LustreError: 35147:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 37 previous similar messages [3096537.131009] LustreError: 36659:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3096537.160689] LustreError: 36659:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 33 previous similar messages [3097152.985303] LustreError: 36672:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107814850800 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3097153.014978] LustreError: 36672:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 36 previous similar messages [3097757.146837] LustreError: 35225:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3097757.176860] LustreError: 35225:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 34 previous similar messages [3098357.225808] LustreError: 36685:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3098357.255822] LustreError: 36685:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 36 previous similar messages [3098965.900839] LustreError: 35151:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107814850800 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3098965.930510] LustreError: 35151:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 34 previous similar messages [3099598.943627] LustreError: 35177:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107814850800 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3099598.973301] LustreError: 35177:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 35 previous similar messages [3100203.053671] LustreError: 35149:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3100203.083340] LustreError: 35149:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 35 previous similar messages [3100806.710792] LustreError: 36671:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3100806.740472] LustreError: 36671:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 34 previous similar messages [3100934.878744] Lustre: oak-MDT0003: haven't heard from client 8344e394-ca93-49e3-9360-8aaf8e163f80 (at 10.51.0.13@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9002a7d3a800, cur 1713365154 expire 1713365004 last 1713364927 [3100934.900790] Lustre: Skipped 21 previous similar messages [3100951.881919] Lustre: oak-MDT0000: haven't heard from client 8344e394-ca93-49e3-9360-8aaf8e163f80 (at 10.51.0.13@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9005fcbe0000, cur 1713365171 expire 1713365021 last 1713364944 [3101359.867713] Lustre: oak-MDT0003: haven't heard from client cfb6e198-f0dc-481d-afd0-151f05cadde1 (at 10.51.8.5@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9018fb764000, cur 1713365579 expire 1713365429 last 1713365352 [3101373.861305] Lustre: oak-MDT0000: haven't heard from client cfb6e198-f0dc-481d-afd0-151f05cadde1 (at 10.51.8.5@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9006f70b3000, cur 1713365593 expire 1713365443 last 1713365366 [3101373.883094] Lustre: Skipped 4 previous similar messages [3101417.084005] LustreError: 36687:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3101417.114025] LustreError: 36687:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 39 previous similar messages [3101807.848885] Lustre: oak-MDT0003: haven't heard from client 340f2902-e09b-4874-afac-b6ebd0e979e9 (at 10.50.5.45@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903a297e6c00, cur 1713366027 expire 1713365877 last 1713365800 [3101807.870762] Lustre: Skipped 4 previous similar messages [3102017.130287] LustreError: 36691:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3102017.160308] LustreError: 36691:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 39 previous similar messages [3102634.204431] Lustre: oak-MDT0000: Client 1c80ca26-dc6a-4d90-a309-490369dc21c3 (at 10.51.1.7@o2ib3) reconnecting [3102634.749251] Lustre: oak-MDT0003: Client f66b8b92-e1dd-4523-b0e0-c4010755e866 (at 10.51.1.10@o2ib3) reconnecting [3102636.335553] Lustre: oak-MDT0003: Client 08db67f8-ecd9-4fe4-831d-c9734cdfc2a8 (at 10.51.1.18@o2ib3) reconnecting [3102636.345810] Lustre: Skipped 2 previous similar messages [3102638.425614] Lustre: oak-MDT0003: Client 86473494-233b-4864-975a-bb4b49494cc8 (at 10.51.1.8@o2ib3) reconnecting [3102638.435787] Lustre: Skipped 6 previous similar messages [3102641.696820] LustreError: 35145:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3102641.726501] LustreError: 35145:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 38 previous similar messages [3102645.274612] Lustre: oak-MDT0000: Client 833236bf-5538-4632-9566-a6a6ce99fbf7 (at 10.51.0.15@o2ib3) reconnecting [3102645.284877] Lustre: Skipped 3 previous similar messages [3102688.932837] LNet: Aborting message for 12345-10.0.2.216@o2ib5: LNetM[DE]Unlink() already called on the MD/ME. [3102688.942951] LNetError: 4164:0:(lib-lnet.h:1243:lnet_set_route_aliveness()) route to o2ib3 through 10.0.2.216@o2ib5 has gone from up to down [3102688.967288] LNetError: 4212:0:(lib-lnet.h:1243:lnet_set_route_aliveness()) route to o2ib3 through 10.0.2.216@o2ib5 has gone from down to up [3102689.333064] Lustre: oak-MDT0000: Client d6c3fefd-46a7-4b1e-911d-fb828a8b9f4d (at 10.51.3.40@o2ib3) reconnecting [3102689.343322] Lustre: Skipped 2 previous similar messages [3102691.069400] LustreError: 137-5: oak-MDT0002_UUID: not available for connect from 10.51.4.1@o2ib3 (no target). If you are running an HA pair check that the target is mounted on the other server. [3102693.488236] LustreError: 137-5: oak-MDT0001_UUID: not available for connect from 10.51.5.17@o2ib3 (no target). If you are running an HA pair check that the target is mounted on the other server. [3102693.505691] LustreError: Skipped 1 previous similar message [3102693.945844] LNet: Dropping PUT from 12345-10.51.6.17@o2ib3 portal 26 match 1793373991571904 offset 0 length 520: 4 [3102694.756727] LustreError: 137-5: oak-MDT0001_UUID: not available for connect from 10.51.1.10@o2ib3 (no target). If you are running an HA pair check that the target is mounted on the other server. [3102694.774180] LustreError: Skipped 3 previous similar messages [3102695.488042] LNet: Dropping PUT from 12345-10.51.10.65@o2ib3 portal 26 match 1793373401210304 offset 0 length 520: 4 [3102695.498644] LNet: Skipped 1 previous similar message [3102696.769090] LustreError: 137-5: oak-MDT0001_UUID: not available for connect from 10.51.1.22@o2ib3 (no target). If you are running an HA pair check that the target is mounted on the other server. [3102696.786533] LustreError: Skipped 10 previous similar messages [3102696.982707] LNet: Dropping PUT from 12345-10.51.6.1@o2ib3 portal 26 match 1793377586466560 offset 0 length 520: 4 [3102696.993136] LNet: Skipped 1 previous similar message [3102699.268933] LNet: Dropping PUT from 12345-10.51.17.24@o2ib3 portal 26 match 1793379808770752 offset 0 length 520: 4 [3102701.303441] LustreError: 137-5: oak-MDT0002_UUID: not available for connect from 10.51.14.17@o2ib3 (no target). If you are running an HA pair check that the target is mounted on the other server. [3102701.320975] LustreError: Skipped 22 previous similar messages [3102703.851852] LNet: Dropping PUT from 12345-10.51.9.65@o2ib3 portal 26 match 1793456389496192 offset 0 length 520: 4 [3102703.862380] LNet: Skipped 6 previous similar messages [3102705.475021] Lustre: oak-MDT0000: Client 478ca69d-9c99-456d-a196-71ded4a8cfc1 (at 10.51.3.9@o2ib3) reconnecting [3102705.475228] Lustre: oak-MDT0000: Export ffff90073d8cb400 already connecting from 10.51.3.9@o2ib3 [3102705.475230] Lustre: Skipped 2 previous similar messages [3102705.499550] Lustre: Skipped 56 previous similar messages [3102710.704005] LustreError: 137-5: oak-MDT0001_UUID: not available for connect from 10.51.10.9@o2ib3 (no target). If you are running an HA pair check that the target is mounted on the other server. [3102710.721454] LustreError: Skipped 14 previous similar messages [3102712.906894] LNet: Dropping PUT from 12345-10.51.4.1@o2ib3 portal 26 match 1793359587998848 offset 0 length 520: 4 [3102712.917330] LNet: Skipped 3 previous similar messages [3102740.186512] Lustre: oak-MDT0003: Client 834f3ebe-f34a-4023-aaa3-ca4b05558655 (at 10.51.2.41@o2ib3) reconnecting [3102740.196778] Lustre: Skipped 41 previous similar messages [3103220.028678] Lustre: oak-MDT0003: Client d446bae4-81c8-41a9-8d2c-3c99e10ac85b (at 10.51.0.12@o2ib3) reconnecting [3103220.039265] Lustre: Skipped 22 previous similar messages [3103241.857279] LustreError: 36669:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3103241.886953] LustreError: 36669:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 39 previous similar messages [3103251.276333] LNet: Dropping PUT from 12345-10.51.6.42@o2ib3 portal 26 match 1793385727195648 offset 0 length 520: 4 [3103253.355100] LustreError: 137-5: oak-MDT0001_UUID: not available for connect from 10.51.5.2@o2ib3 (no target). If you are running an HA pair check that the target is mounted on the other server. [3103253.372466] LustreError: Skipped 5 previous similar messages [3103841.912877] LustreError: 36661:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3103841.942555] LustreError: 36661:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 37 previous similar messages [3104061.734408] Lustre: oak-MDT0003: haven't heard from client 46c1f137-79e9-45f4-bbe7-53e818679f0e (at 10.51.0.62@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9029e200f400, cur 1713368281 expire 1713368131 last 1713368054 [3104061.756280] Lustre: Skipped 1 previous similar message [3104444.905872] LustreError: 36650:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3104444.935550] LustreError: 36650:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 36 previous similar messages [3105044.950933] LustreError: 35155:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3105044.980723] LustreError: 35155:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 38 previous similar messages [3105556.665093] Lustre: oak-MDT0003: haven't heard from client d6088369-a3fa-451f-b470-402417700529 (at 10.50.1.70@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901bea82c400, cur 1713369776 expire 1713369626 last 1713369549 [3105556.686978] Lustre: Skipped 1 previous similar message [3105567.664826] Lustre: oak-MDT0000: haven't heard from client 2cd8a056-da08-4b52-8472-7da0991b5601 (at 10.50.1.69@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90381cb74c00, cur 1713369787 expire 1713369637 last 1713369560 [3105567.686699] Lustre: Skipped 11 previous similar messages [3105632.669207] Lustre: oak-MDT0003: haven't heard from client 63d82dd4-2053-4f29-9632-50df48800bcd (at 10.50.2.34@o2ib2) in 221 seconds. I think it's dead, and I am evicting it. exp ffff903b0621f000, cur 1713369852 expire 1713369702 last 1713369631 [3105632.691475] Lustre: Skipped 11 previous similar messages [3105643.662771] Lustre: oak-MDT0000: haven't heard from client 63d82dd4-2053-4f29-9632-50df48800bcd (at 10.50.2.34@o2ib2) in 219 seconds. I think it's dead, and I am evicting it. exp ffff90353578f400, cur 1713369863 expire 1713369713 last 1713369644 [3105643.684758] Lustre: Skipped 6 previous similar messages [3105644.996101] LustreError: 35207:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3105645.025782] LustreError: 35207:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 36 previous similar messages [3105708.658196] Lustre: oak-MDT0003: haven't heard from client 5ad5e6f0-f696-41c6-8020-83e704594ee4 (at 10.50.2.51@o2ib2) in 226 seconds. I think it's dead, and I am evicting it. exp ffff902078929c00, cur 1713369928 expire 1713369778 last 1713369702 [3105708.680073] Lustre: Skipped 7 previous similar messages [3105719.657536] Lustre: oak-MDT0000: haven't heard from client 72403a9d-20e6-4cb9-898a-534f22f79fd3 (at 10.50.2.66@o2ib2) in 207 seconds. I think it's dead, and I am evicting it. exp ffff903c581d4800, cur 1713369939 expire 1713369789 last 1713369732 [3105719.679419] Lustre: Skipped 4 previous similar messages [3105784.657603] Lustre: oak-MDT0003: haven't heard from client 88d51f04-1230-4967-b75c-793c6faa30d9 (at 10.50.3.42@o2ib2) in 218 seconds. I think it's dead, and I am evicting it. exp ffff900e3780d400, cur 1713370004 expire 1713369854 last 1713369786 [3105784.679486] Lustre: Skipped 5 previous similar messages [3105860.663461] Lustre: oak-MDT0003: haven't heard from client a7de2230-c038-4440-8bf5-703625b59bf8 (at 10.50.5.6@o2ib2) in 194 seconds. I think it's dead, and I am evicting it. exp ffff9007196f1000, cur 1713370080 expire 1713369930 last 1713369886 [3105860.685257] Lustre: Skipped 13 previous similar messages [3105936.653646] Lustre: oak-MDT0003: haven't heard from client fb676cc2-bf51-4bc3-b8c6-f1cbff0ad051 (at 10.50.5.12@o2ib2) in 215 seconds. I think it's dead, and I am evicting it. exp ffff902092d66000, cur 1713370156 expire 1713370006 last 1713369941 [3105936.675531] Lustre: Skipped 18 previous similar messages [3106095.640734] Lustre: oak-MDT0003: haven't heard from client 8a105754-68c4-4750-a372-05c3f73d8180 (at 10.50.6.27@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900b4b32b800, cur 1713370315 expire 1713370165 last 1713370088 [3106095.662609] Lustre: Skipped 24 previous similar messages [3106246.127692] LustreError: 35203:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3106246.158033] LustreError: 35203:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 39 previous similar messages [3106387.631257] Lustre: oak-MDT0000: haven't heard from client 48ed78bb-23da-4db4-ada8-b8a802a96c86 (at 10.50.8.71@o2ib2) in 226 seconds. I think it's dead, and I am evicting it. exp ffff901e41bff800, cur 1713370607 expire 1713370457 last 1713370381 [3106387.653138] Lustre: Skipped 78 previous similar messages [3106846.285215] LustreError: 36650:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3106846.315062] LustreError: 36650:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 38 previous similar messages [3106925.600263] Lustre: oak-MDT0003: haven't heard from client b31fe0ab-09a3-4ac8-afe3-5abfa30b56b7 (at 10.51.4.29@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90147ee7a400, cur 1713371145 expire 1713370995 last 1713370918 [3106925.622144] Lustre: Skipped 176 previous similar messages [3107446.333339] LustreError: 36662:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3107446.363187] LustreError: 36662:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 39 previous similar messages [3107718.566682] Lustre: oak-MDT0000: haven't heard from client f45064b2-6e6b-4905-a64d-598b0d5d379a (at 10.50.0.71@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901926f37000, cur 1713371938 expire 1713371788 last 1713371711 [3107718.588552] Lustre: Skipped 349 previous similar messages [3108046.344738] LustreError: 36658:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3108046.374584] LustreError: 36658:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 37 previous similar messages [3108677.925431] LustreError: 36661:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3108677.955106] LustreError: 36661:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 37 previous similar messages [3109277.971526] LustreError: 36658:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3109278.001197] LustreError: 36658:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 37 previous similar messages [3109880.035481] LustreError: 35192:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3109880.065443] LustreError: 35192:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 35 previous similar messages [3110504.572176] LustreError: 35143:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3110504.602218] LustreError: 35143:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 36 previous similar messages [3111043.110493] LustreError: 33277:0:(qmt_lock.c:425:qmt_lvbo_update()) $$$ failed to release quota space on glimpse 0!=4299454872 : rc = -22 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:370710889696 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3111108.361180] LustreError: 35154:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3111108.391029] LustreError: 35154:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 38 previous similar messages [3111710.262766] LustreError: 35208:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069399941360 granted:559613176, total:368983867536 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:368983867536 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3111710.293223] LustreError: 35208:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 394 previous similar messages [3112310.411287] LustreError: 31510:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069389868352 granted:559613176, total:368989597540 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:368989597540 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3112310.441743] LustreError: 31510:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 2725 previous similar messages [3112910.449156] LustreError: 36654:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069379736008 granted:559613176, total:369167418280 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:369167418280 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3112910.479609] LustreError: 36654:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 2605 previous similar messages [3113510.630109] LustreError: 35158:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069370011540 granted:559613176, total:369281776100 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:369281776100 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3113510.660563] LustreError: 35158:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 2512 previous similar messages [3114110.796718] LustreError: 35202:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069359536048 granted:559613176, total:369368328252 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:369368328252 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3114110.827205] LustreError: 35202:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 2683 previous similar messages [3114710.888607] LustreError: 36667:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069349642952 granted:559613176, total:369415561476 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:369415561476 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3114710.919256] LustreError: 36667:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 2545 previous similar messages [3114936.226767] Lustre: oak-MDT0000: haven't heard from client 6d211768-f23e-4304-8b61-325b14cd9c55 (at 10.50.2.47@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900aaf983000, cur 1713379156 expire 1713379006 last 1713378929 [3114936.248640] Lustre: Skipped 155 previous similar messages [3114958.884876] LustreError: 44928:0:(ldlm_lockd.c:721:ldlm_handle_ast_error()) ### client (nid 10.50.1.63@o2ib2) returned error from blocking AST (req@ffff901731616c00 x1793447792492224 status -107 rc -107), evict it ns: mdt-oak-MDT0000_UUID lock: ffff902a4365bf00/0x1e62cadd64f11b8e lrc: 4/0,0 mode: PR/PR res: [0x200067689:0x15c:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x60200400000020 nid: 10.50.1.63@o2ib2 remote: 0x5384d6ac4bcfff12 expref: 1502 pid: 44788 timeout: 3115250 lvb_type: 0 [3114958.928770] LustreError: 138-a: oak-MDT0000: A client on nid 10.50.1.63@o2ib2 was evicted due to a lock blocking callback time out: rc -107 [3114958.941489] LustreError: 31496:0:(ldlm_lockd.c:261:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 10.50.1.63@o2ib2 ns: mdt-oak-MDT0000_UUID lock: ffff902a4365bf00/0x1e62cadd64f11b8e lrc: 3/0,0 mode: PR/PR res: [0x200067689:0x15c:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x60200400000020 nid: 10.50.1.63@o2ib2 remote: 0x5384d6ac4bcfff12 expref: 1503 pid: 44788 timeout: 0 lvb_type: 0 [3115012.221735] Lustre: oak-MDT0000: haven't heard from client 49130ab8-b9aa-4577-a059-024fe578d064 (at 10.50.2.33@o2ib2) in 205 seconds. I think it's dead, and I am evicting it. exp ffff9010e853d000, cur 1713379232 expire 1713379082 last 1713379027 [3115012.243789] Lustre: Skipped 148 previous similar messages [3115172.214629] Lustre: oak-MDT0003: haven't heard from client 0e289032-2e4e-4508-a3d7-21e96fff7739 (at 10.51.0.63@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9004b62abc00, cur 1713379392 expire 1713379242 last 1713379165 [3115172.236510] Lustre: Skipped 1 previous similar message [3115313.568246] LustreError: 35152:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069349224396 granted:559613176, total:369463295728 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:369463295728 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3115313.598857] LustreError: 35152:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 331 previous similar messages [3115916.936332] LustreError: 35180:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069349224032 granted:559613176, total:369499496212 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:369499496212 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3115916.966919] LustreError: 35180:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 66 previous similar messages [3116346.157010] Lustre: oak-MDT0000: haven't heard from client d563ef06-622a-41bf-be76-f67a8a8421eb (at 10.50.2.33@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9017c27b9c00, cur 1713380566 expire 1713380416 last 1713380339 [3116346.178885] Lustre: Skipped 1 previous similar message [3116507.148881] Lustre: oak-MDT0000: haven't heard from client 56f01528-2e4a-42f4-9694-424e03532bb5 (at 10.50.7.57@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903d97dcb000, cur 1713380727 expire 1713380577 last 1713380500 [3116507.170760] Lustre: Skipped 1 previous similar message [3116516.910326] LustreError: 35145:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3116516.940169] LustreError: 35145:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 79 previous similar messages [3116583.145127] Lustre: oak-MDT0000: haven't heard from client c8cad442-161b-4d90-8b8d-fc5727df426b (at 10.51.10.65@o2ib3) in 226 seconds. I think it's dead, and I am evicting it. exp ffff901c4172f800, cur 1713380803 expire 1713380653 last 1713380577 [3116583.167103] Lustre: Skipped 73 previous similar messages [3116984.131480] Lustre: oak-MDT0000: haven't heard from client cbbc9177-74ff-4054-9d48-5e591abd1f32 (at 10.50.5.48@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900cc431d400, cur 1713381204 expire 1713381054 last 1713380977 [3116984.153738] Lustre: Skipped 631 previous similar messages [3117138.090979] LustreError: 36653:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069349218460 granted:559613176, total:369543046948 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:369543046948 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3117138.121440] LustreError: 36653:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 52 previous similar messages [3117744.377680] LustreError: 35202:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069349218476 granted:559613176, total:369565074300 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:369565074300 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3117744.408132] LustreError: 35202:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 43 previous similar messages [3118138.073979] Lustre: oak-MDT0000: haven't heard from client e90c2ccc-3c8f-4982-9a5d-a4708715c6d1 (at 10.50.17.31@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903cec782800, cur 1713382358 expire 1713382208 last 1713382131 [3118138.095946] Lustre: Skipped 87 previous similar messages [3118364.756700] LustreError: 35178:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069349219440 granted:559613176, total:369586586056 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:369586586056 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3118364.787160] LustreError: 35178:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 44 previous similar messages [3118977.019686] LustreError: 35164:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3118977.049705] LustreError: 35164:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 43 previous similar messages [3118991.031137] Lustre: oak-MDT0000: haven't heard from client 179834c7-e569-4486-9074-f6b16bead5ff (at 10.51.8.1@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90073d99e800, cur 1713383211 expire 1713383061 last 1713382984 [3118991.052933] Lustre: Skipped 5 previous similar messages [3119598.588672] LustreError: 36692:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3119598.619030] LustreError: 36692:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 40 previous similar messages [3120219.034791] LustreError: 35156:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069349219440 granted:559613176, total:369652684304 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:369652684304 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3120219.065244] LustreError: 35156:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 44 previous similar messages [3120819.271489] LustreError: 35183:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107814850800 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3120819.301168] LustreError: 35183:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 71 previous similar messages [3121438.468188] LustreError: 35204:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107814850800 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3121438.497861] LustreError: 35204:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 64 previous similar messages [3122039.615370] LustreError: 36670:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3122039.645045] LustreError: 36670:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 41 previous similar messages [3122645.224350] LustreError: 35176:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107814850800 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3122645.254042] LustreError: 35176:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 52 previous similar messages [3123267.448616] LustreError: 36664:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3123267.478467] LustreError: 36664:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 41 previous similar messages [3123390.813451] Lustre: oak-MDT0000: haven't heard from client daf77900-4959-4f17-ac02-455c684660b7 (at 10.51.8.4@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902c726d1c00, cur 1713387611 expire 1713387461 last 1713387384 [3123390.835237] Lustre: Skipped 1 previous similar message [3123881.312165] LustreError: 36670:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069349091912 granted:559613176, total:369660028028 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:369660028028 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3123881.342625] LustreError: 36670:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 44 previous similar messages [3124455.071178] LustreError: 32594:0:(mdt_open.c:1217:mdt_cross_open()) oak-MDT0003: [0x32c00202bf:0xf561:0x0] doesn't exist!: rc = -14 [3124494.300725] LustreError: 36659:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3124494.330579] LustreError: 36659:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 49 previous similar messages [3124694.643989] LustreError: 44902:0:(mdt_open.c:1217:mdt_cross_open()) oak-MDT0003: [0x32c00207ee:0x2d01:0x0] doesn't exist!: rc = -14 [3124990.361615] LustreError: 44971:0:(mdt_open.c:1217:mdt_cross_open()) oak-MDT0003: [0x32c00204ce:0x2ab9:0x0] doesn't exist!: rc = -14 [3125094.344821] LustreError: 36665:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3125094.374669] LustreError: 36665:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 48 previous similar messages [3125570.703147] Lustre: oak-MDT0000: haven't heard from client 4bdf79c5-695e-41c8-aaf1-45ff9edaf92a (at 10.51.17.20@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901f29da6800, cur 1713389791 expire 1713389641 last 1713389564 [3125570.725109] Lustre: Skipped 1 previous similar message [3125700.306745] LustreError: 35177:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069349120536 granted:559613176, total:369663698784 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:369663698784 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3125700.337195] LustreError: 35177:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 43 previous similar messages [3125779.919724] Lustre: 31359:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1713389252/real 1713389252] req@ffff90244f257980 x1793448187090688/t0(0) o6->oak-OST0035-osc-MDT0000@10.0.2.106@o2ib5:28/4 lens 544/432 e 20 to 1 dl 1713389927 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [3125779.948966] Lustre: oak-OST0035-osc-MDT0000: Connection to oak-OST0035 (at 10.0.2.106@o2ib5) was lost; in progress operations using this service will wait for recovery to complete [3125779.976400] Lustre: oak-OST0035-osc-MDT0000: Connection restored to 10.0.2.106@o2ib5 (at 10.0.2.106@o2ib5) [3125833.691442] Lustre: oak-MDT0003: haven't heard from client 1a5584bf-f029-48a3-aa63-9b8ce5cf7764 (at 10.50.6.58@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90335abfa800, cur 1713390054 expire 1713389904 last 1713389827 [3125833.713319] Lustre: Skipped 7 previous similar messages [3125909.686299] Lustre: oak-MDT0003: haven't heard from client 18aaba42-e435-4b22-8ca4-d576ee06304c (at 10.50.7.32@o2ib2) in 195 seconds. I think it's dead, and I am evicting it. exp ffff90146ce0a400, cur 1713390130 expire 1713389980 last 1713389935 [3125909.708183] Lustre: Skipped 1 previous similar message [3126173.674451] Lustre: oak-MDT0000: haven't heard from client fda94298-cb00-49fc-9cd7-6c06133b6b9c (at 10.51.9.59@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90245affc000, cur 1713390394 expire 1713390244 last 1713390167 [3126173.696352] Lustre: Skipped 27 previous similar messages [3126322.958036] LustreError: 35196:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3126322.987715] LustreError: 35196:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 48 previous similar messages [3126346.665760] Lustre: oak-MDT0003: haven't heard from client fc2b3f57-b2db-4589-9860-fae8fca33ad2 (at 10.50.5.72@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900cb71ef400, cur 1713390567 expire 1713390417 last 1713390340 [3126346.687640] Lustre: Skipped 1 previous similar message [3126422.662150] Lustre: oak-MDT0003: haven't heard from client 142f3c1d-f709-495c-a009-07daccf4814c (at 10.50.0.61@o2ib2) in 208 seconds. I think it's dead, and I am evicting it. exp ffff901c557d5000, cur 1713390643 expire 1713390493 last 1713390435 [3126422.684032] Lustre: Skipped 5 previous similar messages [3126581.048757] Lustre: 31359:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1713390000/real 1713390000] req@ffff90244f257980 x1793448187090688/t0(0) o6->oak-OST0035-osc-MDT0000@10.0.2.106@o2ib5:28/4 lens 544/432 e 20 to 1 dl 1713390675 ref 1 fl Rpc:XQr/2/ffffffff rc 0/-1 job:'' [3126581.077835] Lustre: oak-OST0035-osc-MDT0000: Connection to oak-OST0035 (at 10.0.2.106@o2ib5) was lost; in progress operations using this service will wait for recovery to complete [3126581.104536] Lustre: oak-OST0035-osc-MDT0000: Connection restored to 10.0.2.106@o2ib5 (at 10.0.2.106@o2ib5) [3126923.003123] LustreError: 36667:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3126923.032802] LustreError: 36667:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 47 previous similar messages [3127538.764488] LustreError: 35172:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3127538.794518] LustreError: 35172:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 47 previous similar messages [3128031.587264] Lustre: oak-MDT0000: haven't heard from client c3fc3a6f-8ddc-4104-9f2e-1a334af0f558 (at 10.50.2.63@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900fd506bc00, cur 1713392252 expire 1713392102 last 1713392025 [3128031.609491] Lustre: Skipped 11 previous similar messages [3128138.808699] LustreError: 36680:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3128138.838718] LustreError: 36680:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 60 previous similar messages [3128271.572611] Lustre: oak-MDT0000: haven't heard from client 3ad35d1d-264a-40e4-85e7-41c84f37d509 (at 10.50.7.45@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90385f564800, cur 1713392492 expire 1713392342 last 1713392265 [3128271.594492] Lustre: Skipped 1 previous similar message [3128659.555694] Lustre: oak-MDT0003: haven't heard from client 57b1c58b-9c5d-4ec4-8a4a-17e538a330cc (at 10.50.5.72@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901e0f9fd800, cur 1713392880 expire 1713392730 last 1713392653 [3128659.577575] Lustre: Skipped 1 previous similar message [3128739.246893] LustreError: 36679:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3128739.277053] LustreError: 36679:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 46 previous similar messages [3129339.428512] LustreError: 36670:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107814850800 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3129339.458306] LustreError: 36670:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 118 previous similar messages [3129370.517899] Lustre: oak-MDT0003: haven't heard from client 97d839f8-0ce9-47be-8ade-4efc736390b6 (at 10.50.0.62@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903bf0503400, cur 1713393591 expire 1713393441 last 1713393364 [3129370.539774] Lustre: Skipped 9 previous similar messages [3129446.514172] Lustre: oak-MDT0003: haven't heard from client d0c4a534-95da-487d-81c1-0a1bd3869277 (at 10.51.2.31@o2ib3) in 157 seconds. I think it's dead, and I am evicting it. exp ffff90144ef5e400, cur 1713393667 expire 1713393517 last 1713393510 [3129446.536044] Lustre: Skipped 1 previous similar message [3129522.510399] Lustre: oak-MDT0003: haven't heard from client 057c61e4-bdf2-4982-ba7f-e479b5c620af (at 10.51.2.48@o2ib3) in 225 seconds. I think it's dead, and I am evicting it. exp ffff9022f2828800, cur 1713393743 expire 1713393593 last 1713393518 [3129522.532448] Lustre: Skipped 6 previous similar messages [3129771.498082] Lustre: oak-MDT0000: haven't heard from client a083c361-b16b-4760-8624-15e4f735d721 (at 10.50.2.28@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9022c4995800, cur 1713393992 expire 1713393842 last 1713393765 [3129771.520145] Lustre: Skipped 34 previous similar messages [3129929.490927] Lustre: oak-MDT0003: haven't heard from client 2c288ac7-a2c6-4785-8c38-c6b6323083bf (at 10.51.5.72@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90139bb84400, cur 1713394150 expire 1713394000 last 1713393923 [3129929.512808] Lustre: Skipped 11 previous similar messages [3129942.038792] LustreError: 36668:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069395920228 granted:559613176, total:368967795848 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:368967795848 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3129942.069250] LustreError: 36668:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 116 previous similar messages [3130157.478996] Lustre: oak-MDT0003: haven't heard from client c57213e9-db1c-4474-9b29-8b7d517b0608 (at 10.51.9.4@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90208f49c000, cur 1713394378 expire 1713394228 last 1713394151 [3130157.501253] Lustre: Skipped 7 previous similar messages [3130476.463203] Lustre: oak-MDT0003: haven't heard from client f0439ea7-b3a6-436b-9557-2347dc396439 (at 10.51.2.68@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90402ba76800, cur 1713394697 expire 1713394547 last 1713394470 [3130476.485087] Lustre: Skipped 3 previous similar messages [3130543.944865] LustreError: 35177:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069395898928 granted:559613176, total:368986674996 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:368986674996 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3130543.975326] LustreError: 35177:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 240 previous similar messages [3130552.459332] Lustre: oak-MDT0003: haven't heard from client 3d1d5f30-0501-439f-8e24-b28d00d85f8c (at 10.50.4.25@o2ib2) in 164 seconds. I think it's dead, and I am evicting it. exp ffff900b73e01400, cur 1713394773 expire 1713394623 last 1713394609 [3130552.481256] Lustre: Skipped 1 previous similar message [3131098.432467] Lustre: oak-MDT0000: haven't heard from client c2d9987e-5d01-413c-b993-4c5389e76c6d (at 10.51.2.64@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903505fe6800, cur 1713395319 expire 1713395169 last 1713395092 [3131098.454345] Lustre: Skipped 11 previous similar messages [3131145.500052] LustreError: 35145:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069395880720 granted:559613176, total:369005037856 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:369005037856 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3131145.530506] LustreError: 35145:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 256 previous similar messages [3131634.406583] Lustre: oak-MDT0000: haven't heard from client 36bcb71b-13ac-4682-9c77-638c5d0b16b7 (at 10.50.8.25@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901ccc9b6000, cur 1713395855 expire 1713395705 last 1713395628 [3131634.428457] Lustre: Skipped 5 previous similar messages [3131745.864178] LustreError: 35206:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069395865768 granted:559613176, total:369016052716 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:369016052716 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3131745.894641] LustreError: 35206:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 252 previous similar messages [3132215.377063] Lustre: oak-MDT0000: haven't heard from client c06b3611-ccee-48ab-92e9-120d8b2bb8d3 (at 10.51.5.55@o2ib3) in 169 seconds. I think it's dead, and I am evicting it. exp ffff903c73946800, cur 1713396436 expire 1713396286 last 1713396267 [3132215.398942] Lustre: Skipped 15 previous similar messages [3132347.792172] LustreError: 36653:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069395830368 granted:559613176, total:369027067824 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:369027067824 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3132347.822628] LustreError: 36653:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 232 previous similar messages [3132399.425814] LustreError: 31496:0:(ldlm_lockd.c:261:expired_lock_main()) ### lock callback timer expired after 150s: evicting client at 10.50.1.57@o2ib2 ns: mdt-oak-MDT0000_UUID lock: ffff9031a9176300/0x1e62cade114cecf0 lrc: 4/0,0 mode: PR/PR res: [0x200064b51:0xdbef:0x0].0x0 bits 0x13/0x0 rrc: 6 type: IBT gid 0 flags: 0x60200400000020 nid: 10.50.1.57@o2ib2 remote: 0x22e7dc9805155101 expref: 3004 pid: 44884 timeout: 3132541 lvb_type: 0 [3132950.398868] LustreError: 31509:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069395782412 granted:559613176, total:369026543536 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:369026543536 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3132950.429329] LustreError: 31509:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 236 previous similar messages [3133440.316285] Lustre: oak-MDT0003: haven't heard from client 81a1eb23-c74b-455e-884e-2b0f0a330aa8 (at 10.51.16.15@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900343ba3800, cur 1713397661 expire 1713397511 last 1713397434 [3133440.338244] Lustre: Skipped 22 previous similar messages [3133516.325303] Lustre: oak-MDT0003: haven't heard from client 1371bd82-478a-4e07-94de-ddca6b65d09f (at 10.50.7.56@o2ib2) in 158 seconds. I think it's dead, and I am evicting it. exp ffff90122db4b400, cur 1713397737 expire 1713397587 last 1713397579 [3133516.347188] Lustre: Skipped 3 previous similar messages [3133552.186177] LustreError: 35146:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069395745724 granted:559613176, total:369026543536 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:369026543536 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3133552.216635] LustreError: 35146:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 272 previous similar messages [3134153.282446] LustreError: 35234:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3134153.312678] LustreError: 35234:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 262 previous similar messages [3134244.275398] Lustre: oak-MDT0000: haven't heard from client 9f0fc44f-832e-4706-bd09-28096af41e34 (at 10.50.1.58@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902078914c00, cur 1713398465 expire 1713398315 last 1713398238 [3134244.297397] Lustre: Skipped 1 previous similar message [3134606.780261] Lustre: 31312:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1713398057/real 1713398057] req@ffff901e43c17500 x1793448560793024/t0(0) o6->oak-OST0058-osc-MDT0000@10.0.2.105@o2ib5:28/4 lens 544/432 e 20 to 1 dl 1713398732 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [3134606.809333] Lustre: oak-OST0058-osc-MDT0000: Connection to oak-OST0058 (at 10.0.2.105@o2ib5) was lost; in progress operations using this service will wait for recovery to complete [3134606.852407] Lustre: oak-OST0058-osc-MDT0000: Connection restored to 10.0.2.105@o2ib5 (at 10.0.2.105@o2ib5) [3134755.217069] LustreError: 35148:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069395702536 granted:559613176, total:369026543536 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:369026543536 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3134755.247532] LustreError: 35148:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 266 previous similar messages [3134938.242003] Lustre: oak-MDT0000: haven't heard from client d333f33b-95ae-4b4f-93b9-8807b3f78436 (at 10.51.9.56@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901a42d4d800, cur 1713399159 expire 1713399009 last 1713398932 [3134938.263880] Lustre: Skipped 3 previous similar messages [3135358.217096] LustreError: 35147:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069395678088 granted:559613176, total:369026543536 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:369026543536 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3135358.247550] LustreError: 35147:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 228 previous similar messages [3135572.211485] Lustre: oak-MDT0000: haven't heard from client 8bd3d35d-585b-484e-a987-21bbffadadcd (at 10.51.5.45@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903e94184800, cur 1713399793 expire 1713399643 last 1713399566 [3135572.233541] Lustre: Skipped 1 previous similar message [3135961.355873] LustreError: 36650:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069395640408 granted:559613176, total:369026543536 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:369026543536 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3135961.386327] LustreError: 36650:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 228 previous similar messages [3136192.179724] Lustre: oak-MDT0000: haven't heard from client 2f841c19-fd16-43ab-a6ed-08ef53b72432 (at 10.51.15.6@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901f795ba800, cur 1713400413 expire 1713400263 last 1713400186 [3136192.201948] Lustre: Skipped 5 previous similar messages [3136562.426982] LustreError: 35180:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069395627800 granted:559613176, total:369026543536 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:369026543536 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3136562.457434] LustreError: 35180:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 233 previous similar messages [3137162.572488] LustreError: 35157:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069395592240 granted:559613176, total:369026543536 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:369026543536 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3137162.602946] LustreError: 35157:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 237 previous similar messages [3137183.131091] Lustre: oak-MDT0003: haven't heard from client 721439a8-7a6e-4436-a495-0f80315021b9 (at 10.51.18.13@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90078371fc00, cur 1713401404 expire 1713401254 last 1713401177 [3137183.153049] Lustre: Skipped 13 previous similar messages [3137764.591315] LustreError: 35206:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069395317644 granted:559613176, total:369030215132 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:369030215132 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3137764.621768] LustreError: 35206:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 303 previous similar messages [3137863.097782] Lustre: oak-MDT0000: haven't heard from client 0adcec27-3966-4554-9eeb-aca4e91bc110 (at 10.51.9.44@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900463945800, cur 1713402084 expire 1713401934 last 1713401857 [3137863.119660] Lustre: Skipped 19 previous similar messages [3138365.285826] LustreError: 35149:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069395012784 granted:559613176, total:369037559292 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:369037559292 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3138365.316781] LustreError: 35149:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 356 previous similar messages [3138803.051298] Lustre: oak-MDT0000: haven't heard from client bef3b6df-5ffb-42a5-99b2-348befc5e757 (at 10.51.16.15@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900e329e3800, cur 1713403024 expire 1713402874 last 1713402797 [3138803.073265] Lustre: Skipped 53 previous similar messages [3138966.055600] LustreError: 36667:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069394757120 granted:559613176, total:369037559292 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:369037559292 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3138966.086060] LustreError: 36667:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 412 previous similar messages [3139418.020983] Lustre: oak-MDT0003: haven't heard from client 717928a9-1ed7-4a69-a10e-ab1a0c038a0c (at 10.50.1.57@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902bb575a400, cur 1713403639 expire 1713403489 last 1713403412 [3139418.043085] Lustre: Skipped 11 previous similar messages [3139568.271613] LustreError: 36670:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069394691768 granted:559613176, total:369041230316 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:369041230316 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3139568.302065] LustreError: 36670:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 422 previous similar messages [3140168.334060] LustreError: 35180:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069394640756 granted:559613176, total:369040706028 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:369040706028 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3140168.364514] LustreError: 35180:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 206 previous similar messages [3140475.968997] Lustre: oak-MDT0000: haven't heard from client 811fbe48-68a0-4328-bdcd-6c228edb9980 (at 10.51.14.11@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9012f7a87400, cur 1713404697 expire 1713404547 last 1713404470 [3140475.990958] Lustre: Skipped 15 previous similar messages [3140768.346535] LustreError: 35204:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069394617432 granted:559613176, total:369044376184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:369044376184 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3140768.376988] LustreError: 35204:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 270 previous similar messages [3141369.146635] LustreError: 36666:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069394576144 granted:559613176, total:369044376184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:369044376184 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3141369.177091] LustreError: 36666:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 261 previous similar messages [3141975.910168] LustreError: 35198:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069394540036 granted:559613176, total:369044376184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:369044376184 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3141975.940623] LustreError: 35198:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 285 previous similar messages [3142579.184935] LustreError: 35203:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069394525036 granted:559613176, total:369044376184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:369044376184 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3142579.215395] LustreError: 35203:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 215 previous similar messages [3142934.849576] Lustre: oak-MDT0000: haven't heard from client 44bef2f8-143e-4654-9d02-33fe9791e656 (at 10.51.11.8@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90116efd9c00, cur 1713407156 expire 1713407006 last 1713406929 [3142934.871452] Lustre: Skipped 3 previous similar messages [3143010.851870] Lustre: oak-MDT0003: haven't heard from client fb8c34f4-f79b-464d-bcfb-41171114bc9f (at 10.51.3.8@o2ib3) in 165 seconds. I think it's dead, and I am evicting it. exp ffff901d579a0000, cur 1713407232 expire 1713407082 last 1713407067 [3143010.873675] Lustre: Skipped 1 previous similar message [3143180.113164] LustreError: 35149:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069394492768 granted:559613176, total:369048046384 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:369048046384 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3143180.143761] LustreError: 35149:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 726 previous similar messages [3143783.913315] LustreError: 35154:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069394447044 granted:559613176, total:369048046384 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:369048046384 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3143783.943765] LustreError: 35154:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 260 previous similar messages [3144384.502073] LustreError: 35200:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3144384.531753] LustreError: 35200:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 238 previous similar messages [3144742.763334] Lustre: oak-MDT0000: haven't heard from client 7e89f29f-4e64-4976-9860-7408b7ee3d9b (at 10.50.2.33@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900d4ddc8400, cur 1713408964 expire 1713408814 last 1713408737 [3144742.785206] Lustre: Skipped 5 previous similar messages [3144984.789934] LustreError: 36669:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069394364576 granted:559613176, total:369048046384 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:369048046384 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3144984.820387] LustreError: 36669:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 281 previous similar messages [3145212.740081] Lustre: oak-MDT0000: haven't heard from client c0ea0771-3d57-4f9d-9ead-34f278d48574 (at 10.50.14.11@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901ccc86c800, cur 1713409434 expire 1713409284 last 1713409207 [3145212.762049] Lustre: Skipped 5 previous similar messages [3145586.755099] LustreError: 31509:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069394327196 granted:559613176, total:369048046384 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:369048046384 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3145586.785551] LustreError: 31509:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 290 previous similar messages [3145715.716262] Lustre: oak-MDT0000: haven't heard from client 3514fe48-883a-4978-866c-3851e57612a2 (at 10.50.7.8@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900c14105800, cur 1713409937 expire 1713409787 last 1713409710 [3145715.738495] Lustre: Skipped 1 previous similar message [3145885.709922] Lustre: oak-MDT0003: haven't heard from client 26d5c03c-08b1-4b30-863b-e6d4a9a95ea6 (at 10.50.1.57@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902fec98c400, cur 1713410107 expire 1713409957 last 1713409880 [3145885.731794] Lustre: Skipped 3 previous similar messages [3146097.705526] Lustre: oak-MDT0003: haven't heard from client 3e601386-0036-4ce4-84e1-7f6321321df7 (at 10.51.9.44@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901b4788bc00, cur 1713410319 expire 1713410169 last 1713410092 [3146097.727403] Lustre: Skipped 5 previous similar messages [3146188.638253] LustreError: 35144:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069394673408 granted:559613176, total:369048046384 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:369048046384 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3146188.668709] LustreError: 35144:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 233 previous similar messages [3146791.918162] LustreError: 35154:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069394688068 granted:559613176, total:369048046384 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:369048046384 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3146791.948620] LustreError: 35154:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 119 previous similar messages [3147032.655027] Lustre: oak-MDT0000: haven't heard from client c1d4bd1b-2e76-41cf-bee4-8a528a3b66bd (at 10.51.8.36@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903ce0ce0800, cur 1713411254 expire 1713411104 last 1713411027 [3147032.676917] Lustre: Skipped 9 previous similar messages [3147395.702802] LustreError: 36657:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3147395.732653] LustreError: 36657:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 71 previous similar messages [3147686.633082] Lustre: oak-MDT0000: haven't heard from client 76e266d6-8c6c-43c5-9df8-09c0a493e651 (at 10.51.4.36@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9036e54c6800, cur 1713411908 expire 1713411758 last 1713411681 [3147686.654955] Lustre: Skipped 1 previous similar message [3147998.720514] LustreError: 35169:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3147998.750529] LustreError: 35169:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 44 previous similar messages [3148598.766628] LustreError: 35172:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3148598.796643] LustreError: 35172:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 50 previous similar messages [3149200.091224] LustreError: 35195:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069395096508 granted:559613176, total:369046997808 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:369046997808 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3149200.122001] LustreError: 35195:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 361 previous similar messages [3149329.551729] Lustre: oak-MDT0003: haven't heard from client 56abc87a-0171-46fd-9260-bf02be495b42 (at 10.50.5.24@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902078921c00, cur 1713413551 expire 1713413401 last 1713413324 [3149329.573608] Lustre: Skipped 3 previous similar messages [3149800.127017] LustreError: 36665:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069395181820 granted:559613176, total:369046997808 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:369046997808 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3149800.157481] LustreError: 36665:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 306 previous similar messages [3150400.138783] LustreError: 36655:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069395122440 granted:559613176, total:369046997808 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:369046997808 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3150400.169357] LustreError: 36655:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 143 previous similar messages [3151002.225470] LustreError: 35148:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069395043060 granted:559613176, total:369049622828 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:369049622828 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3151002.256082] LustreError: 35148:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 104 previous similar messages [3151185.800407] LustreError: 44921:0:(mdt_open.c:1217:mdt_cross_open()) oak-MDT0003: [0x32c0020464:0xd026:0x0] doesn't exist!: rc = -14 [3151602.237382] LustreError: 36670:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069394471176 granted:559613176, total:369049098540 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:369049098540 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3151602.267840] LustreError: 36670:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 279 previous similar messages [3152100.418293] Lustre: oak-MDT0003: haven't heard from client 2b1ae450-88cd-4c4e-abdb-3dfd4211ff8e (at 10.51.5.64@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9018cf36b400, cur 1713416322 expire 1713416172 last 1713416095 [3152100.440173] Lustre: Skipped 1 previous similar message [3152205.738275] LustreError: 35182:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069394471252 granted:559613176, total:369049098540 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:369049098540 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3152205.768917] LustreError: 35182:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 52 previous similar messages [3152833.406720] LustreError: 36667:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3152833.436397] LustreError: 36667:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 54 previous similar messages [3153443.263132] LustreError: 35183:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107814850800 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3153443.293081] LustreError: 35183:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 49 previous similar messages [3154051.563612] LustreError: 35152:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069394471128 granted:559613176, total:369048574252 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:369048574252 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3154051.594066] LustreError: 35152:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 46 previous similar messages [3154652.002795] LustreError: 36671:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3154652.032641] LustreError: 36671:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 49 previous similar messages [3155156.277183] Lustre: oak-MDT0000: haven't heard from client ae791419-ab3d-4a84-bbb0-647da38d4441 (at 10.50.7.49@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903f9f646400, cur 1713419378 expire 1713419228 last 1713419151 [3155156.299067] Lustre: Skipped 1 previous similar message [3155232.271724] Lustre: oak-MDT0003: haven't heard from client 41576151-1848-4754-aea5-23f7cc713012 (at 10.50.10.59@o2ib2) in 164 seconds. I think it's dead, and I am evicting it. exp ffff9003d07eb400, cur 1713419454 expire 1713419304 last 1713419290 [3155232.293689] Lustre: Skipped 1 previous similar message [3155252.048973] LustreError: 36667:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3155252.078824] LustreError: 36667:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 69 previous similar messages [3155868.606130] LustreError: 36661:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3155868.635807] LustreError: 36661:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 49 previous similar messages [3156481.848409] LustreError: 36655:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107814850800 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3156481.878087] LustreError: 36655:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 53 previous similar messages [3157087.292069] LustreError: 35158:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069394470304 granted:559613176, total:369048574252 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:369048574252 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3157087.322530] LustreError: 35158:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 56 previous similar messages [3157105.196702] Lustre: oak-MDT0003: haven't heard from client 204e716c-0eb6-4c9a-8be1-0ae00bd3e257 (at 10.51.3.21@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9004e1c0a800, cur 1713421327 expire 1713421177 last 1713421100 [3157105.218585] Lustre: Skipped 1 previous similar message [3157108.200450] Lustre: oak-MDT0000: haven't heard from client 1ee779c0-70a0-48c3-b66c-1163f6100e12 (at 10.51.3.13@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903e0e4a4800, cur 1713421330 expire 1713421180 last 1713421103 [3157108.222318] Lustre: Skipped 3 previous similar messages [3157571.163087] Lustre: oak-MDT0003: haven't heard from client 364fd5ec-de9e-49e2-b9e0-7480efd5a623 (at 10.51.13.8@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9020788a8000, cur 1713421793 expire 1713421643 last 1713421566 [3157571.184969] Lustre: Skipped 3 previous similar messages [3157697.973506] LustreError: 35157:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3157698.003358] LustreError: 35157:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 58 previous similar messages [3158303.122677] LustreError: 35175:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3158303.152694] LustreError: 35175:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 55 previous similar messages [3158429.122494] Lustre: oak-MDT0000: haven't heard from client 62b6532f-3347-4365-9a68-f449e75b0cf7 (at 10.51.4.45@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90073d8ef400, cur 1713422651 expire 1713422501 last 1713422424 [3158429.144370] Lustre: Skipped 1 previous similar message [3158905.102428] LustreError: 35145:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107814850800 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3158905.132100] LustreError: 35145:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 53 previous similar messages [3159506.880355] LustreError: 36665:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3159506.910027] LustreError: 36665:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 68 previous similar messages [3160106.891542] LustreError: 36658:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3160106.921215] LustreError: 36658:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 58 previous similar messages [3160716.640825] LustreError: 35227:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3160716.670841] LustreError: 35227:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 80 previous similar messages [3161316.685688] LustreError: 36690:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3161316.715712] LustreError: 36690:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 62 previous similar messages [3161932.611308] LustreError: 35181:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3161932.641156] LustreError: 35181:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 47 previous similar messages [3162476.955174] Lustre: oak-MDT0003: haven't heard from client 6221aa65-7fb6-4e99-bf24-03b90a4823cd (at 10.51.15.8@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902092d11000, cur 1713426699 expire 1713426549 last 1713426472 [3162476.977053] Lustre: Skipped 1 previous similar message [3162477.935339] Lustre: oak-MDT0000: haven't heard from client 6221aa65-7fb6-4e99-bf24-03b90a4823cd (at 10.51.15.8@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901a42da3800, cur 1713426700 expire 1713426550 last 1713426473 [3162534.756085] LustreError: 35145:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069394415708 granted:559613176, total:369059585500 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:369059585500 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3162534.786540] LustreError: 35145:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 63 previous similar messages [3163146.976475] LustreError: 36660:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069394406832 granted:559613176, total:369059585500 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:369059585500 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3163147.006931] LustreError: 36660:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 62 previous similar messages [3163749.419404] LustreError: 35200:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069394405472 granted:559613176, total:369059585500 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:369059585500 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3163749.449864] LustreError: 35200:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 85 previous similar messages [3164360.341762] LustreError: 36660:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069394405472 granted:559613176, total:369059585500 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:369059585500 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3164360.372221] LustreError: 36660:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 46 previous similar messages [3164973.347342] LustreError: 36656:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069394404688 granted:559613176, total:369059585500 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:369059585500 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3164973.377800] LustreError: 36656:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 81 previous similar messages [3165574.233393] LustreError: 35156:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069394403624 granted:559613176, total:369059585500 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:369059585500 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3165574.264005] LustreError: 35156:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 906 previous similar messages [3166178.170446] LustreError: 35180:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069394402028 granted:559613176, total:368897440344 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:368897440344 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3166178.200900] LustreError: 35180:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 1339 previous similar messages [3166778.690916] LustreError: 35179:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069394399336 granted:559613176, total:368897440344 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:368897440344 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3166778.721370] LustreError: 35179:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 2079 previous similar messages [3167382.185916] LustreError: 35157:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069394396428 granted:559613176, total:368807787096 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:368807787096 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3167382.216690] LustreError: 35157:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 2219 previous similar messages [3167982.213445] LustreError: 36657:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069394393672 granted:559613176, total:368807262808 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:368807262808 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3167982.243903] LustreError: 36657:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 2232 previous similar messages [3168590.937427] LustreError: 35152:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069394390712 granted:559613176, total:368565489896 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:368565489896 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3168590.967887] LustreError: 35152:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 1999 previous similar messages [3169192.600007] LustreError: 35204:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069394389712 granted:559613176, total:368564965608 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:368564965608 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3169192.630463] LustreError: 35204:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 954 previous similar messages [3169793.576487] LustreError: 35178:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069394388196 granted:559613176, total:368563392744 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:368563392744 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3169793.606941] LustreError: 35178:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 314 previous similar messages [3170394.617636] LustreError: 36657:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069394387176 granted:559613176, total:368471095704 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:368471095704 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3170394.648093] LustreError: 36657:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 355 previous similar messages [3170994.724890] LustreError: 35179:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069393745012 granted:559613176, total:368544187600 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:368544187600 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3170994.755350] LustreError: 35179:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 1169 previous similar messages [3171595.212995] LustreError: 36662:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069393500144 granted:559613176, total:368545083224 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:368545083224 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3171595.243626] LustreError: 36662:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 1110 previous similar messages [3172195.781891] LustreError: 35192:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069393914720 granted:559613176, total:368451668424 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:368451668424 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3172195.812359] LustreError: 35192:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 1129 previous similar messages [3172799.148654] LustreError: 35158:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069393754428 granted:559613176, total:368470110688 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:368470110688 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3172799.179107] LustreError: 35158:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 349 previous similar messages [3173410.881953] LustreError: 36665:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3173410.911804] LustreError: 36665:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 226 previous similar messages [3173682.416431] Lustre: oak-MDT0000: haven't heard from client 8e4e5225-b544-4a92-9a12-1fdd06217dfa (at 10.51.9.43@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9020c5da5c00, cur 1713437905 expire 1713437755 last 1713437678 [3174010.926924] LustreError: 36667:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3174010.956770] LustreError: 36667:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 82 previous similar messages [3174613.383848] LustreError: 36654:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3174613.413530] LustreError: 36654:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 179 previous similar messages [3175213.429121] LustreError: 36672:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3175213.458794] LustreError: 36672:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 189 previous similar messages [3175832.503365] LustreError: 35182:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3175832.533039] LustreError: 35182:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 99 previous similar messages [3176449.206798] LustreError: 36653:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3176449.236474] LustreError: 36653:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 53 previous similar messages [3177058.913469] LustreError: 35206:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107814850800 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3177058.943152] LustreError: 35206:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 47 previous similar messages [3177246.245936] Lustre: oak-MDT0000: haven't heard from client c09df7c1-b9df-4ba0-ab8a-976111b59b07 (at 10.51.9.32@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902060185800, cur 1713441469 expire 1713441319 last 1713441242 [3177246.268002] Lustre: Skipped 1 previous similar message [3177660.365602] LustreError: 31510:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069394184116 granted:559613176, total:367310380328 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:367310380328 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3177660.396064] LustreError: 31510:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 45 previous similar messages [3178264.801489] LustreError: 35146:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069394184116 granted:559613176, total:367314052080 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:367314052080 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3178264.831944] LustreError: 35146:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 43 previous similar messages [3178875.742763] LustreError: 36684:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3178875.772783] LustreError: 36684:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 42 previous similar messages [3179012.394186] Lustre: oak-MDT0000: OI scrub prep, flags = 0x56 [3179012.400030] Lustre: oak-MDT0000: reset OI scrub file, old flags = 0x0, add flags = 0x0 [3179012.408334] Lustre: oak-MDT0000: store scrub file: rc = 0 [3179012.413913] Lustre: oak-MDT0000: OI scrub start, flags = 0x56, pos = 12 [3179012.413920] Lustre: oak-MDT0000: trigger partial OI scrub for RPC inconsistency, checking FID [0x2000526c2:0x5c:0x0]/899780669: rc = 0 [3179030.320300] Lustre: oak-MDT0000: reset OI scrub file, old flags = 0x5, add flags = 0x2 [3179030.328403] Lustre: oak-MDT0000: store scrub file: rc = 0 [3179030.333986] Lustre: oak-MDT0000: joined in the OI scrub with flag 12: rc = 0 [3179089.555176] Lustre: oak-MDT0000: OI scrub checkpoint at pos 411382890 [3179089.561813] Lustre: oak-MDT0000: store scrub file: rc = 0 [3179149.552384] Lustre: oak-MDT0000: OI scrub checkpoint at pos 437606519 [3179149.559017] Lustre: oak-MDT0000: store scrub file: rc = 0 [3179209.549657] Lustre: oak-MDT0000: OI scrub checkpoint at pos 464388423 [3179209.556298] Lustre: oak-MDT0000: store scrub file: rc = 0 [3179269.546887] Lustre: oak-MDT0000: OI scrub checkpoint at pos 490600453 [3179269.553592] Lustre: oak-MDT0000: store scrub file: rc = 0 [3179329.543971] Lustre: oak-MDT0000: OI scrub checkpoint at pos 521530430 [3179329.551650] Lustre: oak-MDT0000: store scrub file: rc = 0 [3179389.541172] Lustre: oak-MDT0000: OI scrub checkpoint at pos 556060161 [3179389.547815] Lustre: oak-MDT0000: store scrub file: rc = 0 [3179449.538588] Lustre: oak-MDT0000: OI scrub checkpoint at pos 581766773 [3179449.545234] Lustre: oak-MDT0000: store scrub file: rc = 0 [3179475.788709] LustreError: 36698:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3179475.818724] LustreError: 36698:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 47 previous similar messages [3179509.535400] Lustre: oak-MDT0000: OI scrub checkpoint at pos 599658921 [3179509.542127] Lustre: oak-MDT0000: store scrub file: rc = 0 [3179569.532625] Lustre: oak-MDT0000: OI scrub checkpoint at pos 623089487 [3179569.539260] Lustre: oak-MDT0000: store scrub file: rc = 0 [3179629.529846] Lustre: oak-MDT0000: OI scrub checkpoint at pos 637329029 [3179629.536501] Lustre: oak-MDT0000: store scrub file: rc = 0 [3179689.526861] Lustre: oak-MDT0000: OI scrub checkpoint at pos 650383492 [3179689.533496] Lustre: oak-MDT0000: store scrub file: rc = 0 [3179749.524177] Lustre: oak-MDT0000: OI scrub checkpoint at pos 667698009 [3179749.530875] Lustre: oak-MDT0000: store scrub file: rc = 0 [3179809.521255] Lustre: oak-MDT0000: OI scrub checkpoint at pos 692339095 [3179809.527887] Lustre: oak-MDT0000: store scrub file: rc = 0 [3179869.518475] Lustre: oak-MDT0000: OI scrub checkpoint at pos 704381887 [3179869.525118] Lustre: oak-MDT0000: store scrub file: rc = 0 [3179901.491922] Lustre: 31332:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1713443233/real 1713443233] req@ffff90203e7bf080 x1793449829169024/t0(0) o6->oak-OST0054-osc-MDT0000@10.0.2.105@o2ib5:28/4 lens 544/432 e 20 to 1 dl 1713443908 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [3179901.521158] Lustre: oak-OST0054-osc-MDT0000: Connection to oak-OST0054 (at 10.0.2.105@o2ib5) was lost; in progress operations using this service will wait for recovery to complete [3179901.547960] Lustre: oak-OST0054-osc-MDT0000: Connection restored to 10.0.2.105@o2ib5 (at 10.0.2.105@o2ib5) [3179929.515591] Lustre: oak-MDT0000: OI scrub checkpoint at pos 720102278 [3179929.522250] Lustre: oak-MDT0000: store scrub file: rc = 0 [3179989.512890] Lustre: oak-MDT0000: OI scrub checkpoint at pos 743202450 [3179989.519532] Lustre: oak-MDT0000: store scrub file: rc = 0 [3180049.510085] Lustre: oak-MDT0000: OI scrub checkpoint at pos 770937863 [3180049.516721] Lustre: oak-MDT0000: store scrub file: rc = 0 [3180097.710043] LustreError: 35183:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069394184116 granted:559613176, total:367313527792 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:367313527792 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3180097.740496] LustreError: 35183:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 43 previous similar messages [3180109.507232] Lustre: oak-MDT0000: OI scrub checkpoint at pos 795111271 [3180109.513873] Lustre: oak-MDT0000: store scrub file: rc = 0 [3180169.504281] Lustre: oak-MDT0000: OI scrub checkpoint at pos 806735866 [3180169.510921] Lustre: oak-MDT0000: store scrub file: rc = 0 [3180229.501561] Lustre: oak-MDT0000: OI scrub checkpoint at pos 817655957 [3180229.508200] Lustre: oak-MDT0000: store scrub file: rc = 0 [3180289.498553] Lustre: oak-MDT0000: OI scrub checkpoint at pos 832174344 [3180289.505211] Lustre: oak-MDT0000: store scrub file: rc = 0 [3180349.495692] Lustre: oak-MDT0000: OI scrub checkpoint at pos 846985873 [3180349.502331] Lustre: oak-MDT0000: store scrub file: rc = 0 [3180409.492933] Lustre: oak-MDT0000: OI scrub checkpoint at pos 859960856 [3180409.499570] Lustre: oak-MDT0000: store scrub file: rc = 0 [3180469.489979] Lustre: oak-MDT0000: OI scrub checkpoint at pos 871146007 [3180469.496623] Lustre: oak-MDT0000: store scrub file: rc = 0 [3180529.487081] Lustre: oak-MDT0000: OI scrub checkpoint at pos 883111346 [3180529.493716] Lustre: oak-MDT0000: store scrub file: rc = 0 [3180589.484259] Lustre: oak-MDT0000: OI scrub checkpoint at pos 894028293 [3180589.490893] Lustre: oak-MDT0000: store scrub file: rc = 0 [3180649.481346] Lustre: oak-MDT0000: OI scrub checkpoint at pos 908781025 [3180649.487979] Lustre: oak-MDT0000: store scrub file: rc = 0 [3180698.036070] LustreError: 35149:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069393919840 granted:559613176, total:367282354236 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:367282354236 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3180698.066703] LustreError: 35149:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 377 previous similar messages [3180709.478651] Lustre: oak-MDT0000: OI scrub checkpoint at pos 918209481 [3180709.485359] Lustre: oak-MDT0000: store scrub file: rc = 0 [3180769.475622] Lustre: oak-MDT0000: OI scrub checkpoint at pos 932857998 [3180769.482279] Lustre: oak-MDT0000: store scrub file: rc = 0 [3180829.473007] Lustre: oak-MDT0000: OI scrub checkpoint at pos 953287941 [3180829.479640] Lustre: oak-MDT0000: store scrub file: rc = 0 [3180889.470061] Lustre: oak-MDT0000: OI scrub checkpoint at pos 965554183 [3180889.476689] Lustre: oak-MDT0000: store scrub file: rc = 0 [3180949.467306] Lustre: oak-MDT0000: OI scrub checkpoint at pos 978007173 [3180949.474012] Lustre: oak-MDT0000: store scrub file: rc = 0 [3181009.464788] Lustre: oak-MDT0000: OI scrub checkpoint at pos 988922389 [3181009.471429] Lustre: oak-MDT0000: store scrub file: rc = 0 [3181069.461535] Lustre: oak-MDT0000: OI scrub checkpoint at pos 1000491269 [3181069.468249] Lustre: oak-MDT0000: store scrub file: rc = 0 [3181129.458697] Lustre: oak-MDT0000: OI scrub checkpoint at pos 1023052850 [3181129.465423] Lustre: oak-MDT0000: store scrub file: rc = 0 [3181189.456155] Lustre: oak-MDT0000: OI scrub checkpoint at pos 1042974341 [3181189.462896] Lustre: oak-MDT0000: store scrub file: rc = 0 [3181249.453078] Lustre: oak-MDT0000: OI scrub checkpoint at pos 1061457665 [3181249.459809] Lustre: oak-MDT0000: store scrub file: rc = 0 [3181298.082777] LustreError: 35204:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069393646344 granted:559613176, total:367443931156 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:367443931156 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3181298.113247] LustreError: 35204:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 458 previous similar messages [3181309.450188] Lustre: oak-MDT0000: OI scrub checkpoint at pos 1076376555 [3181309.456913] Lustre: oak-MDT0000: store scrub file: rc = 0 [3181369.447343] Lustre: oak-MDT0000: OI scrub checkpoint at pos 1088619165 [3181369.454143] Lustre: oak-MDT0000: store scrub file: rc = 0 [3181429.444611] Lustre: oak-MDT0000: OI scrub checkpoint at pos 1101619253 [3181429.451343] Lustre: oak-MDT0000: store scrub file: rc = 0 [3181489.441692] Lustre: oak-MDT0000: OI scrub checkpoint at pos 1116330188 [3181489.448431] Lustre: oak-MDT0000: store scrub file: rc = 0 [3181549.438855] Lustre: oak-MDT0000: OI scrub checkpoint at pos 1128884532 [3181549.445659] Lustre: oak-MDT0000: store scrub file: rc = 0 [3181609.436189] Lustre: oak-MDT0000: OI scrub checkpoint at pos 1140306841 [3181609.442916] Lustre: oak-MDT0000: store scrub file: rc = 0 [3181669.433325] Lustre: oak-MDT0000: OI scrub checkpoint at pos 1160402565 [3181669.440052] Lustre: oak-MDT0000: store scrub file: rc = 0 [3181729.430399] Lustre: oak-MDT0000: OI scrub checkpoint at pos 1178133419 [3181729.437126] Lustre: oak-MDT0000: store scrub file: rc = 0 [3181789.427691] Lustre: oak-MDT0000: OI scrub checkpoint at pos 1196488881 [3181789.434417] Lustre: oak-MDT0000: store scrub file: rc = 0 [3181849.425456] Lustre: oak-MDT0000: OI scrub checkpoint at pos 1217537856 [3181849.432186] Lustre: oak-MDT0000: store scrub file: rc = 0 [3181900.902421] LustreError: 36672:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107814850800 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3181900.932095] LustreError: 36672:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 328 previous similar messages [3181909.421904] Lustre: oak-MDT0000: OI scrub checkpoint at pos 1235189677 [3181909.428628] Lustre: oak-MDT0000: store scrub file: rc = 0 [3181969.419229] Lustre: oak-MDT0000: OI scrub checkpoint at pos 1250616837 [3181969.426077] Lustre: oak-MDT0000: store scrub file: rc = 0 [3182029.416323] Lustre: oak-MDT0000: OI scrub checkpoint at pos 1265744901 [3182029.423049] Lustre: oak-MDT0000: store scrub file: rc = 0 [3182089.416507] Lustre: oak-MDT0000: OI scrub checkpoint at pos 1288276239 [3182089.423239] Lustre: oak-MDT0000: store scrub file: rc = 0 [3182149.410516] Lustre: oak-MDT0000: OI scrub checkpoint at pos 1311689788 [3182149.417244] Lustre: oak-MDT0000: store scrub file: rc = 0 [3182209.407658] Lustre: oak-MDT0000: OI scrub checkpoint at pos 1332762072 [3182209.414766] Lustre: oak-MDT0000: store scrub file: rc = 0 [3182269.404869] Lustre: oak-MDT0000: OI scrub checkpoint at pos 1345749248 [3182269.411597] Lustre: oak-MDT0000: store scrub file: rc = 0 [3182329.402169] Lustre: oak-MDT0000: OI scrub checkpoint at pos 1364129157 [3182329.408893] Lustre: oak-MDT0000: store scrub file: rc = 0 [3182389.399137] Lustre: oak-MDT0000: OI scrub checkpoint at pos 1385572451 [3182389.406036] Lustre: oak-MDT0000: store scrub file: rc = 0 [3182449.396292] Lustre: oak-MDT0000: OI scrub checkpoint at pos 1404568345 [3182449.403015] Lustre: oak-MDT0000: store scrub file: rc = 0 [3182501.752955] LustreError: 35206:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069394103716 granted:559613176, total:366681532180 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:366681532180 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3182501.783413] LustreError: 35206:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 483 previous similar messages [3182509.456992] Lustre: oak-MDT0000: OI scrub checkpoint at pos 1422762898 [3182509.463780] Lustre: oak-MDT0000: store scrub file: rc = 0 [3182569.511782] Lustre: oak-MDT0000: OI scrub checkpoint at pos 1433976357 [3182569.518537] Lustre: oak-MDT0000: store scrub file: rc = 0 [3182629.387986] Lustre: oak-MDT0000: OI scrub checkpoint at pos 1450550021 [3182629.394717] Lustre: oak-MDT0000: store scrub file: rc = 0 [3182689.384900] Lustre: oak-MDT0000: OI scrub checkpoint at pos 1472078270 [3182689.391625] Lustre: oak-MDT0000: store scrub file: rc = 0 [3182749.382052] Lustre: oak-MDT0000: OI scrub checkpoint at pos 1488349137 [3182749.388769] Lustre: oak-MDT0000: store scrub file: rc = 0 [3182809.379320] Lustre: oak-MDT0000: OI scrub checkpoint at pos 1509342725 [3182809.386126] Lustre: oak-MDT0000: store scrub file: rc = 0 [3182869.376306] Lustre: oak-MDT0000: OI scrub checkpoint at pos 1526810808 [3182869.383017] Lustre: oak-MDT0000: store scrub file: rc = 0 [3182929.373436] Lustre: oak-MDT0000: OI scrub checkpoint at pos 1540004168 [3182929.380154] Lustre: oak-MDT0000: store scrub file: rc = 0 [3182989.370902] Lustre: oak-MDT0000: OI scrub checkpoint at pos 1558108421 [3182989.377634] Lustre: oak-MDT0000: store scrub file: rc = 0 [3183049.367773] Lustre: oak-MDT0000: OI scrub checkpoint at pos 1571469293 [3183049.374646] Lustre: oak-MDT0000: store scrub file: rc = 0 [3183109.364970] Lustre: oak-MDT0000: OI scrub checkpoint at pos 1589459665 [3183109.371696] Lustre: oak-MDT0000: store scrub file: rc = 0 [3183115.968243] LustreError: 35204:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069394222776 granted:559613176, total:366476262124 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:366476262124 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3183115.998697] LustreError: 35204:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 790 previous similar messages [3183169.362090] Lustre: oak-MDT0000: OI scrub checkpoint at pos 1610395071 [3183169.368814] Lustre: oak-MDT0000: store scrub file: rc = 0 [3183229.359392] Lustre: oak-MDT0000: OI scrub checkpoint at pos 1626232022 [3183229.366120] Lustre: oak-MDT0000: store scrub file: rc = 0 [3183289.356453] Lustre: oak-MDT0000: OI scrub checkpoint at pos 1637134274 [3183289.363293] Lustre: oak-MDT0000: store scrub file: rc = 0 [3183349.353551] Lustre: oak-MDT0000: OI scrub checkpoint at pos 1650716102 [3183349.360274] Lustre: oak-MDT0000: store scrub file: rc = 0 [3183409.350978] Lustre: oak-MDT0000: OI scrub checkpoint at pos 1670707333 [3183409.357705] Lustre: oak-MDT0000: store scrub file: rc = 0 [3183469.347815] Lustre: oak-MDT0000: OI scrub checkpoint at pos 1686927688 [3183469.354528] Lustre: oak-MDT0000: store scrub file: rc = 0 [3183529.344950] Lustre: oak-MDT0000: OI scrub checkpoint at pos 1706826501 [3183529.351681] Lustre: oak-MDT0000: store scrub file: rc = 0 [3183589.342062] Lustre: oak-MDT0000: OI scrub checkpoint at pos 1722042793 [3183589.348777] Lustre: oak-MDT0000: store scrub file: rc = 0 [3183649.339263] Lustre: oak-MDT0000: OI scrub checkpoint at pos 1744628442 [3183649.345990] Lustre: oak-MDT0000: store scrub file: rc = 0 [3183709.336329] Lustre: oak-MDT0000: OI scrub checkpoint at pos 1769687637 [3183709.343220] Lustre: oak-MDT0000: store scrub file: rc = 0 [3183726.897312] LustreError: 35201:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3183726.926993] LustreError: 35201:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 77 previous similar messages [3183769.333684] Lustre: oak-MDT0000: OI scrub checkpoint at pos 1797450885 [3183769.340412] Lustre: oak-MDT0000: store scrub file: rc = 0 [3183829.330609] Lustre: oak-MDT0000: OI scrub checkpoint at pos 1824784790 [3183829.337339] Lustre: oak-MDT0000: store scrub file: rc = 0 [3183889.327879] Lustre: oak-MDT0000: OI scrub checkpoint at pos 1845152370 [3183889.334671] Lustre: oak-MDT0000: store scrub file: rc = 0 [3183949.324877] Lustre: oak-MDT0000: OI scrub checkpoint at pos 1865234799 [3183949.331605] Lustre: oak-MDT0000: store scrub file: rc = 0 [3184009.322008] Lustre: oak-MDT0000: OI scrub checkpoint at pos 1886085600 [3184009.328731] Lustre: oak-MDT0000: store scrub file: rc = 0 [3184069.319217] Lustre: oak-MDT0000: OI scrub checkpoint at pos 1911675269 [3184069.326164] Lustre: oak-MDT0000: store scrub file: rc = 0 [3184129.316232] Lustre: oak-MDT0000: OI scrub checkpoint at pos 1934775718 [3184129.322954] Lustre: oak-MDT0000: store scrub file: rc = 0 [3184189.585320] Lustre: oak-MDT0000: OI scrub checkpoint at pos 1947851672 [3184189.592035] Lustre: oak-MDT0000: store scrub file: rc = 0 [3184249.310523] Lustre: oak-MDT0000: OI scrub checkpoint at pos 1966183008 [3184249.317314] Lustre: oak-MDT0000: store scrub file: rc = 0 [3184309.307726] Lustre: oak-MDT0000: OI scrub checkpoint at pos 1986192367 [3184309.314449] Lustre: oak-MDT0000: store scrub file: rc = 0 [3184328.712827] LustreError: 35192:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069394249956 granted:559613176, total:366445680580 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:366445680580 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3184328.743288] LustreError: 35192:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 46 previous similar messages [3184369.304763] Lustre: oak-MDT0000: OI scrub checkpoint at pos 2010366821 [3184369.311676] Lustre: oak-MDT0000: store scrub file: rc = 0 [3184429.301975] Lustre: oak-MDT0000: OI scrub checkpoint at pos 2029619702 [3184429.308687] Lustre: oak-MDT0000: store scrub file: rc = 0 [3184489.299131] Lustre: oak-MDT0000: OI scrub checkpoint at pos 2046409582 [3184489.305844] Lustre: oak-MDT0000: store scrub file: rc = 0 [3184549.296492] Lustre: oak-MDT0000: OI scrub checkpoint at pos 2070073349 [3184549.303302] Lustre: oak-MDT0000: store scrub file: rc = 0 [3184609.293479] Lustre: oak-MDT0000: OI scrub checkpoint at pos 2083173391 [3184609.300209] Lustre: oak-MDT0000: store scrub file: rc = 0 [3184669.290631] Lustre: oak-MDT0000: OI scrub checkpoint at pos 2095721739 [3184669.297376] Lustre: oak-MDT0000: store scrub file: rc = 0 [3184729.287839] Lustre: oak-MDT0000: OI scrub checkpoint at pos 2107351896 [3184729.294640] Lustre: oak-MDT0000: store scrub file: rc = 0 [3184789.285047] Lustre: oak-MDT0000: OI scrub checkpoint at pos 2118797099 [3184789.291768] Lustre: oak-MDT0000: store scrub file: rc = 0 [3184849.282265] Lustre: oak-MDT0000: OI scrub checkpoint at pos 2134467589 [3184849.289058] Lustre: oak-MDT0000: store scrub file: rc = 0 [3184909.279746] Lustre: oak-MDT0000: OI scrub checkpoint at pos 2142986760 [3184909.286544] Lustre: oak-MDT0000: store scrub file: rc = 0 [3184930.516668] LustreError: 35229:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3184930.546696] LustreError: 35229:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 89 previous similar messages [3184969.276846] Lustre: oak-MDT0000: OI scrub checkpoint at pos 2153907845 [3184969.283634] Lustre: oak-MDT0000: store scrub file: rc = 0 [3185029.274017] Lustre: oak-MDT0000: OI scrub checkpoint at pos 2165392901 [3185029.280765] Lustre: oak-MDT0000: store scrub file: rc = 0 [3185089.270948] Lustre: oak-MDT0000: OI scrub checkpoint at pos 2173474065 [3185089.277669] Lustre: oak-MDT0000: store scrub file: rc = 0 [3185149.268126] Lustre: oak-MDT0000: OI scrub checkpoint at pos 2180519108 [3185149.274851] Lustre: oak-MDT0000: store scrub file: rc = 0 [3185209.265251] Lustre: oak-MDT0000: OI scrub checkpoint at pos 2188700060 [3185209.271982] Lustre: oak-MDT0000: store scrub file: rc = 0 [3185269.262371] Lustre: oak-MDT0000: OI scrub checkpoint at pos 2199572505 [3185269.269106] Lustre: oak-MDT0000: store scrub file: rc = 0 [3185329.259613] Lustre: oak-MDT0000: OI scrub checkpoint at pos 2220995851 [3185329.266329] Lustre: oak-MDT0000: store scrub file: rc = 0 [3185389.256598] Lustre: oak-MDT0000: OI scrub checkpoint at pos 2244036716 [3185389.263329] Lustre: oak-MDT0000: store scrub file: rc = 0 [3185449.253786] Lustre: oak-MDT0000: OI scrub checkpoint at pos 2263357754 [3185449.260507] Lustre: oak-MDT0000: store scrub file: rc = 0 [3185509.250880] Lustre: oak-MDT0000: OI scrub checkpoint at pos 2278618017 [3185509.257613] Lustre: oak-MDT0000: store scrub file: rc = 0 [3185552.157190] LustreError: 36660:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069394249956 granted:559613176, total:366447779600 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:366447779600 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3185552.187650] LustreError: 36660:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 47 previous similar messages [3185569.248148] Lustre: oak-MDT0000: OI scrub checkpoint at pos 2303183365 [3185569.254885] Lustre: oak-MDT0000: store scrub file: rc = 0 [3185629.245201] Lustre: oak-MDT0000: OI scrub checkpoint at pos 2326229646 [3185629.251931] Lustre: oak-MDT0000: store scrub file: rc = 0 [3185689.242367] Lustre: oak-MDT0000: OI scrub checkpoint at pos 2346105147 [3185689.249164] Lustre: oak-MDT0000: store scrub file: rc = 0 [3185749.239521] Lustre: oak-MDT0000: OI scrub checkpoint at pos 2375493405 [3185749.246242] Lustre: oak-MDT0000: store scrub file: rc = 0 [3185809.236673] Lustre: oak-MDT0000: OI scrub checkpoint at pos 2398656291 [3185809.243529] Lustre: oak-MDT0000: store scrub file: rc = 0 [3185869.233830] Lustre: oak-MDT0000: OI scrub checkpoint at pos 2426348624 [3185869.240552] Lustre: oak-MDT0000: store scrub file: rc = 0 [3185929.230989] Lustre: oak-MDT0000: OI scrub checkpoint at pos 2468259305 [3185929.237710] Lustre: oak-MDT0000: store scrub file: rc = 0 [3185989.228139] Lustre: oak-MDT0000: OI scrub checkpoint at pos 2508030879 [3185989.234870] Lustre: oak-MDT0000: store scrub file: rc = 0 [3186035.440943] Lustre: 31315:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1713448997/real 1713448997] req@ffff900814a64c80 x1793449938562048/t0(0) o6->oak-OST0038-osc-MDT0000@10.0.2.105@o2ib5:28/4 lens 544/432 e 0 to 1 dl 1713449795 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [3186035.469929] Lustre: oak-OST0038-osc-MDT0000: Connection to oak-OST0038 (at 10.0.2.105@o2ib5) was lost; in progress operations using this service will wait for recovery to complete [3186035.496796] Lustre: oak-OST0038-osc-MDT0000: Connection restored to 10.0.2.105@o2ib5 (at 10.0.2.105@o2ib5) [3186041.830783] Lustre: oak-MDT0003: haven't heard from client 867ac939-3a13-415e-9e5c-9848bf551c0b (at 10.50.1.52@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901ccc83e800, cur 1713450265 expire 1713450115 last 1713450038 [3186041.852657] Lustre: Skipped 1 previous similar message [3186046.830483] Lustre: oak-MDT0000: haven't heard from client 867ac939-3a13-415e-9e5c-9848bf551c0b (at 10.50.1.52@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900e24838000, cur 1713450270 expire 1713450120 last 1713450043 [3186049.225263] Lustre: oak-MDT0000: OI scrub checkpoint at pos 2529193083 [3186049.232220] Lustre: oak-MDT0000: store scrub file: rc = 0 [3186109.222406] Lustre: oak-MDT0000: OI scrub checkpoint at pos 2546376267 [3186109.229133] Lustre: oak-MDT0000: store scrub file: rc = 0 [3186152.158959] LustreError: 35185:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3186152.189326] LustreError: 35185:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 41 previous similar messages [3186169.219539] Lustre: oak-MDT0000: OI scrub checkpoint at pos 2573560440 [3186169.226272] Lustre: oak-MDT0000: store scrub file: rc = 0 [3186229.216667] Lustre: oak-MDT0000: OI scrub checkpoint at pos 2597142710 [3186229.223382] Lustre: oak-MDT0000: store scrub file: rc = 0 [3186289.213885] Lustre: oak-MDT0000: OI scrub checkpoint at pos 2634368816 [3186289.220700] Lustre: oak-MDT0000: store scrub file: rc = 0 [3186349.211128] Lustre: oak-MDT0000: OI scrub checkpoint at pos 2649517061 [3186349.217857] Lustre: oak-MDT0000: store scrub file: rc = 0 [3186409.208137] Lustre: oak-MDT0000: OI scrub checkpoint at pos 2668377505 [3186409.214859] Lustre: oak-MDT0000: store scrub file: rc = 0 [3186469.205345] Lustre: oak-MDT0000: OI scrub checkpoint at pos 2693660924 [3186469.212063] Lustre: oak-MDT0000: store scrub file: rc = 0 [3186529.202567] Lustre: oak-MDT0000: OI scrub checkpoint at pos 2705831236 [3186529.209447] Lustre: oak-MDT0000: store scrub file: rc = 0 [3186589.199844] Lustre: oak-MDT0000: OI scrub checkpoint at pos 2721372037 [3186589.206598] Lustre: oak-MDT0000: store scrub file: rc = 0 [3186649.196960] Lustre: oak-MDT0000: OI scrub checkpoint at pos 2737658099 [3186649.203688] Lustre: oak-MDT0000: store scrub file: rc = 0 [3186709.194150] Lustre: oak-MDT0000: OI scrub checkpoint at pos 2749170053 [3186709.200954] Lustre: oak-MDT0000: store scrub file: rc = 0 [3186761.588603] LustreError: 36668:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107814850800 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3186761.618275] LustreError: 36668:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 48 previous similar messages [3186769.191223] Lustre: oak-MDT0000: OI scrub checkpoint at pos 2762851706 [3186769.197945] Lustre: oak-MDT0000: store scrub file: rc = 0 [3186829.188428] Lustre: oak-MDT0000: OI scrub checkpoint at pos 2776100639 [3186829.195212] Lustre: oak-MDT0000: store scrub file: rc = 0 [3186889.185564] Lustre: oak-MDT0000: OI scrub checkpoint at pos 2788230533 [3186889.192462] Lustre: oak-MDT0000: store scrub file: rc = 0 [3186949.182769] Lustre: oak-MDT0000: OI scrub checkpoint at pos 2799057997 [3186949.189484] Lustre: oak-MDT0000: store scrub file: rc = 0 [3187009.179873] Lustre: oak-MDT0000: OI scrub checkpoint at pos 2814563811 [3187009.186728] Lustre: oak-MDT0000: store scrub file: rc = 0 [3187069.177072] Lustre: oak-MDT0000: OI scrub checkpoint at pos 2827693386 [3187069.183791] Lustre: oak-MDT0000: store scrub file: rc = 0 [3187129.174137] Lustre: oak-MDT0000: OI scrub checkpoint at pos 2839771096 [3187129.180861] Lustre: oak-MDT0000: store scrub file: rc = 0 [3187189.171403] Lustre: oak-MDT0000: OI scrub checkpoint at pos 2856128261 [3187189.178196] Lustre: oak-MDT0000: store scrub file: rc = 0 [3187249.168458] Lustre: oak-MDT0000: OI scrub checkpoint at pos 2876551379 [3187249.175185] Lustre: oak-MDT0000: store scrub file: rc = 0 [3187309.165597] Lustre: oak-MDT0000: OI scrub checkpoint at pos 2897912702 [3187309.172322] Lustre: oak-MDT0000: store scrub file: rc = 0 [3187342.402985] Lustre: 31315:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1713450258/real 1713450258] req@ffff900814a64c80 x1793449938562048/t0(0) o6->oak-OST0038-osc-MDT0000@10.0.2.105@o2ib5:28/4 lens 544/432 e 0 to 1 dl 1713451056 ref 1 fl Rpc:XQr/2/ffffffff rc 0/-1 job:'' [3187342.431980] Lustre: oak-OST0038-osc-MDT0000: Connection to oak-OST0038 (at 10.0.2.105@o2ib5) was lost; in progress operations using this service will wait for recovery to complete [3187342.458559] Lustre: oak-OST0038-osc-MDT0000: Connection restored to 10.0.2.105@o2ib5 (at 10.0.2.105@o2ib5) [3187363.920053] LustreError: 36662:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107814850800 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3187363.949990] LustreError: 36662:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 45 previous similar messages [3187369.162716] Lustre: oak-MDT0000: OI scrub checkpoint at pos 2916360943 [3187369.169512] Lustre: oak-MDT0000: store scrub file: rc = 0 [3187429.159876] Lustre: oak-MDT0000: OI scrub checkpoint at pos 2935695793 [3187429.166584] Lustre: oak-MDT0000: store scrub file: rc = 0 [3187489.157072] Lustre: oak-MDT0000: OI scrub checkpoint at pos 2959835015 [3187489.163799] Lustre: oak-MDT0000: store scrub file: rc = 0 [3187549.154225] Lustre: oak-MDT0000: OI scrub checkpoint at pos 2984114511 [3187549.160955] Lustre: oak-MDT0000: store scrub file: rc = 0 [3187609.151375] Lustre: oak-MDT0000: OI scrub checkpoint at pos 3021056899 [3187609.158094] Lustre: oak-MDT0000: store scrub file: rc = 0 [3187669.148661] Lustre: oak-MDT0000: OI scrub checkpoint at pos 3047863429 [3187669.155437] Lustre: oak-MDT0000: store scrub file: rc = 0 [3187729.145773] Lustre: oak-MDT0000: OI scrub checkpoint at pos 3077046661 [3187729.152490] Lustre: oak-MDT0000: store scrub file: rc = 0 [3187789.142902] Lustre: oak-MDT0000: OI scrub checkpoint at pos 3103871237 [3187789.149633] Lustre: oak-MDT0000: store scrub file: rc = 0 [3187840.482708] Lustre: oak-MDT0000: OI scrub post with result = 1 [3187840.488745] Lustre: oak-MDT0000: store scrub file: rc = 0 [3187840.494328] Lustre: oak-MDT0000: OI scrub: stop, pos = 3124704561: rc = 1 [3187983.763620] LustreError: 35153:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107814850800 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3187983.793298] LustreError: 35153:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 49 previous similar messages [3188591.001221] LustreError: 36670:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069394249956 granted:559613176, total:366451451136 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:366451451136 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3188591.031670] LustreError: 36670:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 49 previous similar messages [3188847.699784] Lustre: oak-MDT0000: haven't heard from client 559872c7-28bf-44eb-8a65-61f78f21b5f5 (at 10.51.16.6@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902999533400, cur 1713453071 expire 1713452921 last 1713452844 [3188923.695394] Lustre: oak-MDT0000: haven't heard from client 3eddcff1-a53d-424d-9dbd-ba4b3d20ce72 (at 10.51.0.67@o2ib3) in 177 seconds. I think it's dead, and I am evicting it. exp ffff9017b7720000, cur 1713453147 expire 1713452997 last 1713452970 [3188923.717266] Lustre: Skipped 5 previous similar messages [3188999.691519] Lustre: oak-MDT0000: haven't heard from client ed3da3c0-18e0-4550-ab28-de4cd5c9570e (at 10.51.9.44@o2ib3) in 206 seconds. I think it's dead, and I am evicting it. exp ffff9018fb63e000, cur 1713453223 expire 1713453073 last 1713453017 [3188999.713391] Lustre: Skipped 3 previous similar messages [3189075.688753] Lustre: oak-MDT0000: haven't heard from client d02714ea-c4b4-4641-8d1f-156921bffd06 (at 10.50.7.28@o2ib2) in 181 seconds. I think it's dead, and I am evicting it. exp ffff90039c8e9400, cur 1713453299 expire 1713453149 last 1713453118 [3189075.710638] Lustre: Skipped 3 previous similar messages [3189151.684211] Lustre: oak-MDT0000: haven't heard from client b1dfcaef-ef3f-4835-9743-e976ee8817bf (at 10.50.6.29@o2ib2) in 219 seconds. I think it's dead, and I am evicting it. exp ffff901fa4c43c00, cur 1713453375 expire 1713453225 last 1713453156 [3189151.706084] Lustre: Skipped 1 previous similar message [3189206.759998] LustreError: 35144:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3189206.789668] LustreError: 35144:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 49 previous similar messages [3189336.680911] Lustre: oak-MDT0000: haven't heard from client 4c568e88-d533-447f-933f-8efcc0eb1319 (at 10.50.7.25@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903e64df0400, cur 1713453560 expire 1713453410 last 1713453333 [3189336.702979] Lustre: Skipped 1 previous similar message [3189513.673665] Lustre: oak-MDT0000: haven't heard from client 8404c704-7d8a-4e6c-90a0-67f8fd95430c (at 10.50.5.72@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90073d86c000, cur 1713453737 expire 1713453587 last 1713453510 [3189513.695685] Lustre: Skipped 7 previous similar messages [3189757.656650] Lustre: oak-MDT0000: haven't heard from client b29dd779-991a-476f-8d52-c60fc27b0033 (at 10.50.7.63@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902a5343ec00, cur 1713453981 expire 1713453831 last 1713453754 [3189757.678523] Lustre: Skipped 5 previous similar messages [3189808.182641] LustreError: 35154:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107814850800 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3189808.212321] LustreError: 35154:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 45 previous similar messages [3190162.911455] Lustre: 31347:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1713453597/real 1713453597] req@ffff903308f7d580 x1793450036669120/t0(0) o6->oak-OST0041-osc-MDT0000@10.0.2.106@o2ib5:28/4 lens 544/432 e 20 to 1 dl 1713454272 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [3190162.940533] Lustre: oak-OST0041-osc-MDT0000: Connection to oak-OST0041 (at 10.0.2.106@o2ib5) was lost; in progress operations using this service will wait for recovery to complete [3190162.998290] Lustre: oak-OST0041-osc-MDT0000: Connection restored to 10.0.2.106@o2ib5 (at 10.0.2.106@o2ib5) [3190410.654954] LustreError: 35197:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3190410.684804] LustreError: 35197:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 45 previous similar messages [3190557.619171] Lustre: oak-MDT0000: haven't heard from client 26231555-7087-4d47-ba1a-1b07b0fbf44d (at 10.51.12.17@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9020788e8000, cur 1713454781 expire 1713454631 last 1713454554 [3190557.641576] Lustre: Skipped 1 previous similar message [3191026.836606] LustreError: 36687:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3191026.866621] LustreError: 36687:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 46 previous similar messages [3191250.587003] Lustre: oak-MDT0000: haven't heard from client 13966d02-5a59-4820-bbbe-7d3b9326c4c0 (at 10.51.9.18@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9005e00a4c00, cur 1713455474 expire 1713455324 last 1713455247 [3191250.608879] Lustre: Skipped 1 previous similar message [3191632.647642] LustreError: 36660:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107814850800 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3191632.677319] LustreError: 36660:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 56 previous similar messages [3191906.555830] Lustre: oak-MDT0000: haven't heard from client 9cefaa91-496c-4779-b6e8-f10bab3d6528 (at 10.51.9.44@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901bea844400, cur 1713456130 expire 1713455980 last 1713455903 [3191906.577705] Lustre: Skipped 3 previous similar messages [3192234.198274] LustreError: 36655:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3192234.228119] LustreError: 36655:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 299 previous similar messages [3192414.533024] Lustre: oak-MDT0000: haven't heard from client ff70b363-e943-40fe-aad2-e8711c48bee6 (at 10.50.1.57@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902bf9e21000, cur 1713456638 expire 1713456488 last 1713456411 [3192414.554906] Lustre: Skipped 5 previous similar messages [3192693.528565] Lustre: oak-MDT0003: haven't heard from client 3c556781-d17a-4f35-9973-798b43f33a7e (at 10.50.1.64@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901a42d4dc00, cur 1713456917 expire 1713456767 last 1713456690 [3192693.550439] Lustre: Skipped 5 previous similar messages [3192703.521052] Lustre: oak-MDT0000: haven't heard from client 3c556781-d17a-4f35-9973-798b43f33a7e (at 10.50.1.64@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9011e3b4c800, cur 1713456927 expire 1713456777 last 1713456700 [3192855.350252] LustreError: 36654:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069394216016 granted:559613176, total:366461941168 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:366461941168 time:0 qunit: 4194304 edquot:0 may_rel:0 revoke:0 default:no [3192855.380704] LustreError: 36654:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 208 previous similar messages [3193453.486383] Lustre: oak-MDT0000: haven't heard from client 34966653-de3a-40d1-9e92-b38bdd633cee (at 10.51.1.49@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902d364d8000, cur 1713457677 expire 1713457527 last 1713457450 [3193458.933435] LustreError: 35195:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3193458.963289] LustreError: 35195:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 58 previous similar messages [3194068.380293] LustreError: 35225:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3194068.410310] LustreError: 35225:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 50 previous similar messages [3194328.443486] Lustre: oak-MDT0000: haven't heard from client a98a3052-9565-4c51-8a92-5b0bf016cb04 (at 10.51.0.66@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902b7d8cbc00, cur 1713458552 expire 1713458402 last 1713458325 [3194328.465358] Lustre: Skipped 1 previous similar message [3194668.533561] LustreError: 36675:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3194668.563703] LustreError: 36675:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 52 previous similar messages [3194902.416291] Lustre: oak-MDT0000: haven't heard from client 2b1d536f-2644-42e6-aa54-1f9ed91a2fb1 (at 10.51.17.5@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903523c1cc00, cur 1713459126 expire 1713458976 last 1713458899 [3194902.438168] Lustre: Skipped 3 previous similar messages [3195184.859449] LustreError: 33277:0:(qmt_lock.c:425:qmt_lvbo_update()) $$$ failed to release quota space on glimpse 0!=122965850286704 : rc = -22 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:362955749420 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3195271.865234] LustreError: 36667:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3195271.895094] LustreError: 36667:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 182 previous similar messages [3195449.393267] Lustre: oak-MDT0003: haven't heard from client 0974effe-60da-44c0-a929-45bfb8eee5e6 (at 10.50.5.72@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901916032c00, cur 1713459673 expire 1713459523 last 1713459446 [3195449.415508] Lustre: Skipped 1 previous similar message [3195525.387845] Lustre: oak-MDT0000: haven't heard from client d838f133-b62d-4586-afe5-f7883a82b2d4 (at 10.51.4.7@o2ib3) in 211 seconds. I think it's dead, and I am evicting it. exp ffff9029abf96800, cur 1713459749 expire 1713459599 last 1713459538 [3195525.409638] Lustre: Skipped 5 previous similar messages [3195872.877331] LustreError: 35180:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3195872.907185] LustreError: 35180:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 69 previous similar messages [3196487.295408] LustreError: 35219:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3196487.325424] LustreError: 35219:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 83 previous similar messages [3196843.326124] Lustre: oak-MDT0000: haven't heard from client 88418f09-fc85-4d27-a734-368dbe2e5c91 (at 10.51.4.33@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902092d27c00, cur 1713461067 expire 1713460917 last 1713460840 [3196843.348002] Lustre: Skipped 17 previous similar messages [3197091.770722] LustreError: 35157:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107846283640 granted:310379184, total:355159284259 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355159284259 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3197091.801264] LustreError: 35157:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 85 previous similar messages [3197708.365313] LustreError: 31510:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3197708.395161] LustreError: 31510:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 83 previous similar messages [3197977.274385] Lustre: oak-MDT0000: haven't heard from client dc1cd2fb-2fc5-48ef-b7bc-e8bd97897802 (at 10.51.4.60@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9029c5d02400, cur 1713462201 expire 1713462051 last 1713461974 [3197977.296263] Lustre: Skipped 1 previous similar message [3198313.441390] LustreError: 36688:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3198313.471410] LustreError: 36688:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 83 previous similar messages [3198916.366789] LustreError: 35203:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3198916.396468] LustreError: 35203:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 76 previous similar messages [3199050.244459] Lustre: oak-MDT0003: haven't heard from client c2313d78-cd3b-4dc8-a874-d81f87358265 (at 10.51.0.68@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9029e200b400, cur 1713463274 expire 1713463124 last 1713463047 [3199050.266334] Lustre: Skipped 1 previous similar message [3199062.224353] Lustre: oak-MDT0000: haven't heard from client 88429078-7362-4d40-b891-1be2d0b089a5 (at 10.50.0.63@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900e3780bc00, cur 1713463286 expire 1713463136 last 1713463059 [3199062.246242] Lustre: Skipped 1 previous similar message [3199517.657000] LustreError: 35182:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107846290656 granted:486160868, total:355158235683 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355158235683 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3199517.687546] LustreError: 35182:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 76 previous similar messages [3200120.140523] LustreError: 35146:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107846283640 granted:310379184, total:355158235683 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355158235683 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3200120.171223] LustreError: 35146:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 76 previous similar messages [3200687.147231] Lustre: oak-MDT0000: haven't heard from client 41bc528d-0ea2-4ebd-8777-ba1ca53042f7 (at 10.50.6.28@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90251100e400, cur 1713464911 expire 1713464761 last 1713464684 [3200687.169118] Lustre: Skipped 1 previous similar message [3200733.165273] LustreError: 36671:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107846290652 granted:486160868, total:355156138531 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355156138531 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3200733.195956] LustreError: 36671:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 111 previous similar messages [3201337.388989] LustreError: 36671:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107846283640 granted:310379184, total:355156138531 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355156138531 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3201337.419533] LustreError: 36671:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 88 previous similar messages [3201937.589129] LustreError: 35178:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107851449892 granted:310379184, total:352070257664 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352070257664 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3201937.619669] LustreError: 35178:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 1495 previous similar messages [3202547.586061] LustreError: 36694:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107956080788 granted:483067692, total:351066839784 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:351066839784 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3202547.616603] LustreError: 36694:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 343 previous similar messages [3202777.047923] Lustre: oak-MDT0003: haven't heard from client 07cfd590-ad61-408c-9c8b-807163e770bf (at 10.51.9.44@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901cd07cd000, cur 1713467001 expire 1713466851 last 1713466774 [3202777.069806] Lustre: Skipped 1 previous similar message [3202853.042970] Lustre: oak-MDT0000: haven't heard from client 126541da-d001-47ed-9f73-d67d799617f0 (at 10.50.8.10@o2ib2) in 208 seconds. I think it's dead, and I am evicting it. exp ffff9032bf335400, cur 1713467077 expire 1713466927 last 1713466869 [3202853.064851] Lustre: Skipped 5 previous similar messages [3203161.346911] LustreError: 35229:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3203161.376935] LustreError: 35229:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 78 previous similar messages [3203267.023607] Lustre: oak-MDT0000: haven't heard from client cb1d708d-1f3c-426d-aa3b-193915af5638 (at 10.50.5.72@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901248370c00, cur 1713467491 expire 1713467341 last 1713467264 [3203267.045485] Lustre: Skipped 5 previous similar messages [3203762.059534] LustreError: 35180:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3203762.089385] LustreError: 35180:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 84 previous similar messages [3204363.057979] LustreError: 35176:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3204363.087821] LustreError: 35176:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 91 previous similar messages [3204473.964873] Lustre: oak-MDT0000: haven't heard from client 41dacbf5-fd4e-4623-9155-790926690d50 (at 10.51.4.10@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90147c469000, cur 1713468698 expire 1713468548 last 1713468471 [3204473.986951] Lustre: Skipped 3 previous similar messages [3204963.137969] LustreError: 35197:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3204963.167824] LustreError: 35197:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 78 previous similar messages [3205025.940959] Lustre: oak-MDT0000: haven't heard from client 2d1f3e15-b7ca-4c03-82c7-36b758fd4549 (at 10.51.9.18@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900a1fe27400, cur 1713469250 expire 1713469100 last 1713469023 [3205025.962830] Lustre: Skipped 1 previous similar message [3205563.835975] LustreError: 36666:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3205563.865826] LustreError: 36666:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 75 previous similar messages [3205588.910336] Lustre: oak-MDT0003: haven't heard from client 377e1035-2bf0-4e07-b861-331edb008495 (at 10.51.12.13@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901938cdb000, cur 1713469813 expire 1713469663 last 1713469586 [3205588.932298] Lustre: Skipped 3 previous similar messages [3205789.902995] Lustre: oak-MDT0003: haven't heard from client 9c18467b-e0b8-4fca-9920-2d7d2418f868 (at 10.51.5.19@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9024acd21400, cur 1713470014 expire 1713469864 last 1713469787 [3205789.924872] Lustre: Skipped 1 previous similar message [3205799.901681] Lustre: oak-MDT0000: haven't heard from client 9c18467b-e0b8-4fca-9920-2d7d2418f868 (at 10.51.5.19@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903c27624800, cur 1713470024 expire 1713469874 last 1713469797 [3206165.855710] LustreError: 35207:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3206165.885557] LustreError: 35207:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 72 previous similar messages [3206772.542360] LustreError: 36670:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107814850800 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3206772.572037] LustreError: 36670:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 71 previous similar messages [3207232.833303] Lustre: oak-MDT0003: haven't heard from client 802151d2-5c77-458a-85ef-3b693135e8ca (at 10.51.12.13@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901bf3fd7c00, cur 1713471457 expire 1713471307 last 1713471230 [3207245.829147] Lustre: oak-MDT0000: haven't heard from client 802151d2-5c77-458a-85ef-3b693135e8ca (at 10.51.12.13@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901a5784ec00, cur 1713471470 expire 1713471320 last 1713471243 [3207372.551927] LustreError: 35145:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107932883508 granted:486160868, total:351093506128 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:351093506128 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3207372.582466] LustreError: 35145:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 86 previous similar messages [3207754.812313] Lustre: oak-MDT0003: haven't heard from client f2fa5576-46fd-402a-8c77-a85cf5723417 (at 10.50.12.5@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90143ea33c00, cur 1713471979 expire 1713471829 last 1713471752 [3207759.803035] Lustre: oak-MDT0000: haven't heard from client f2fa5576-46fd-402a-8c77-a85cf5723417 (at 10.50.12.5@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90067a783000, cur 1713471984 expire 1713471834 last 1713471757 [3207830.814736] Lustre: oak-MDT0003: haven't heard from client b443065b-2bfc-460e-b4f2-f793e8936a85 (at 10.51.4.41@o2ib3) in 173 seconds. I think it's dead, and I am evicting it. exp ffff9002a7c32000, cur 1713472055 expire 1713471905 last 1713471882 [3207972.723497] LustreError: 36668:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3207972.753346] LustreError: 36668:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 121 previous similar messages [3208589.174237] LustreError: 35202:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107932883552 granted:486160868, total:351093506128 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:351093506128 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3208589.204784] LustreError: 35202:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 78 previous similar messages [3209196.717147] LustreError: 36657:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107932883552 granted:486160868, total:351093506128 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:351093506128 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3209196.747698] LustreError: 36657:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 77 previous similar messages [3209227.731059] Lustre: oak-MDT0003: haven't heard from client 58fafffd-6905-403a-b3e4-7b7e3b0afdf4 (at 10.51.5.30@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901f795d2c00, cur 1713473452 expire 1713473302 last 1713473225 [3209227.752934] Lustre: Skipped 1 previous similar message [3209809.512946] LustreError: 35195:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107851461128 granted:310379184, total:351093506128 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:351093506128 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3209809.543680] LustreError: 35195:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 74 previous similar messages [3210415.553419] LustreError: 35152:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536452124 granted:559613176, total:351093506128 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:351093506128 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3210415.583964] LustreError: 35152:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 84 previous similar messages [3211017.830763] LustreError: 36694:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107942024360 granted:424654352, total:351093506128 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:351093506128 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3211017.861399] LustreError: 36694:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 82 previous similar messages [3211481.624305] Lustre: oak-MDT0000: haven't heard from client 224cd378-5c1a-4db1-aff8-a53c6c8d5cbf (at 10.51.6.13@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90068933ac00, cur 1713475706 expire 1713475556 last 1713475479 [3211481.646363] Lustre: Skipped 1 previous similar message [3211626.421757] LustreError: 36669:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3211626.451436] LustreError: 36669:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 5169 previous similar messages [3212230.781984] LustreError: 35149:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3212230.811837] LustreError: 35149:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 86 previous similar messages [3212694.561863] Lustre: oak-MDT0003: haven't heard from client 316b1ec5-d06d-4bff-bb97-6004157667f5 (at 10.51.11.24@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9019d7d2b800, cur 1713476919 expire 1713476769 last 1713476692 [3212694.583822] Lustre: Skipped 5 previous similar messages [3212832.036757] LustreError: 35171:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107956080844 granted:483067692, total:351069501552 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:351069501552 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3212832.067672] LustreError: 35171:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 81 previous similar messages [3213432.302272] LustreError: 35223:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3213432.302273] LustreError: 35163:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107956088032 granted:483067692, total:351076842548 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:351076842548 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3213432.302276] LustreError: 35163:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 641 previous similar messages [3214042.539138] LustreError: 36691:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107956107264 granted:483067692, total:351088976764 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:351088976764 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3214042.569671] LustreError: 36691:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 167 previous similar messages [3214646.879825] LustreError: 36659:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069535421696 granted:559613176, total:351135133620 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:351135133620 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3214646.910368] LustreError: 36659:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 129 previous similar messages [3215258.445675] LustreError: 35148:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852220724 granted:310379184, total:351165545844 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:351165545844 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3215258.476212] LustreError: 35148:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 126 previous similar messages [3215858.643154] LustreError: 35205:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069535421696 granted:559613176, total:351212739388 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:351212739388 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3215858.673697] LustreError: 35205:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 121 previous similar messages [3216045.397608] Lustre: oak-MDT0000: haven't heard from client e28670ba-e36d-4c6e-9afd-819c21fd3466 (at 10.51.1.28@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901f52bbc800, cur 1713480270 expire 1713480120 last 1713480043 [3216045.419660] Lustre: Skipped 1 previous similar message [3216458.631409] LustreError: 36693:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107942026452 granted:424654352, total:351306079276 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:351306079276 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3216458.661944] LustreError: 36693:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 148 previous similar messages [3217059.659786] LustreError: 35152:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107814850800 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3217059.689463] LustreError: 35152:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 154 previous similar messages [3217371.331556] Lustre: oak-MDT0000: haven't heard from client 95fc93f1-c7d8-4d2c-af9f-42fada887845 (at 10.51.1.33@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902993c70c00, cur 1713481596 expire 1713481446 last 1713481369 [3217371.353427] Lustre: Skipped 1 previous similar message [3217663.368815] LustreError: 35221:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107942025608 granted:424654352, total:351353277044 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:351353277044 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3217663.399349] LustreError: 35221:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 161 previous similar messages [3217768.312096] Lustre: oak-MDT0000: haven't heard from client 55503253-5d1c-4d86-bc77-52a83cf43fae (at 10.51.14.15@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9008d7868400, cur 1713481993 expire 1713481843 last 1713481766 [3217768.334061] Lustre: Skipped 5 previous similar messages [3217844.308481] Lustre: oak-MDT0000: haven't heard from client ad0cf2b7-43fd-4b3e-a9c6-67f0e36d5646 (at 10.51.10.32@o2ib3) in 193 seconds. I think it's dead, and I am evicting it. exp ffff90360c6b4c00, cur 1713482069 expire 1713481919 last 1713481876 [3217844.330440] Lustre: Skipped 1 previous similar message [3217920.304584] Lustre: oak-MDT0000: haven't heard from client 5ad1c1af-28d3-42d7-aec6-13de59517269 (at 10.51.10.14@o2ib3) in 207 seconds. I think it's dead, and I am evicting it. exp ffff9004961ee800, cur 1713482145 expire 1713481995 last 1713481938 [3217920.326542] Lustre: Skipped 1 previous similar message [3217996.301493] Lustre: oak-MDT0000: haven't heard from client f3d7d681-2e50-4ac4-9066-2cfa6ec5f7d3 (at 10.51.8.3@o2ib3) in 153 seconds. I think it's dead, and I am evicting it. exp ffff90064f1e8400, cur 1713482221 expire 1713482071 last 1713482068 [3217996.323283] Lustre: Skipped 1 previous similar message [3218072.300895] Lustre: oak-MDT0000: haven't heard from client f8613695-c3e5-4e67-9952-644ac2ce37e4 (at 10.50.8.25@o2ib2) in 164 seconds. I think it's dead, and I am evicting it. exp ffff901278c0e000, cur 1713482297 expire 1713482147 last 1713482133 [3218265.149693] LustreError: 35177:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3218265.179540] LustreError: 35177:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 132 previous similar messages [3218518.275306] Lustre: oak-MDT0003: haven't heard from client 321493ed-0b30-418d-92a3-e82e92a16b98 (at 10.50.1.57@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903047744800, cur 1713482743 expire 1713482593 last 1713482516 [3218518.297181] Lustre: Skipped 4 previous similar messages [3218865.856306] LustreError: 36662:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3218865.886479] LustreError: 36662:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 138 previous similar messages [3219475.246618] LustreError: 35174:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3219475.246619] LustreError: 36686:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107956106308 granted:483067692, total:351537864196 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:351537864196 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3219475.246622] LustreError: 36686:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 154 previous similar messages [3219668.224124] Lustre: oak-MDT0000: haven't heard from client 5b4ccbdc-85fe-4b0c-9d14-87d5a4b31917 (at 10.51.7.53@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902d0dcd2400, cur 1713483893 expire 1713483743 last 1713483666 [3219668.245998] Lustre: Skipped 1 previous similar message [3220091.922249] LustreError: 36680:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107956106008 granted:483067692, total:351568282904 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:351568282904 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3220091.952786] LustreError: 36680:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 139 previous similar messages [3220263.195074] Lustre: oak-MDT0003: haven't heard from client e990663c-e806-4ce7-8066-672745f7b800 (at 10.51.4.36@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903a3f34a800, cur 1713484488 expire 1713484338 last 1713484261 [3220263.216953] Lustre: Skipped 3 previous similar messages [3220695.256604] LustreError: 35229:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107942024572 granted:424654352, total:351599747480 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:351599747480 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3220695.287144] LustreError: 35229:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 133 previous similar messages [3221306.030286] LustreError: 35203:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3221306.059965] LustreError: 35203:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 162 previous similar messages [3221370.141273] Lustre: oak-MDT0003: haven't heard from client f1ac0782-23f3-4d5b-b4f5-abb7dbbdc7cd (at 10.51.10.49@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903076448800, cur 1713485595 expire 1713485445 last 1713485368 [3221370.163236] Lustre: Skipped 3 previous similar messages [3221906.074449] LustreError: 35154:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3221906.104125] LustreError: 35154:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 151 previous similar messages [3222506.160749] LustreError: 35205:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3222506.190426] LustreError: 35205:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 176 previous similar messages [3222616.078773] Lustre: oak-MDT0000: haven't heard from client c5a38e3b-5e56-4e49-91a5-fbbc9b23a8fd (at 10.51.8.44@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903ce0ce4400, cur 1713486841 expire 1713486691 last 1713486614 [3222616.100813] Lustre: Skipped 1 previous similar message [3222634.085581] Lustre: oak-MDT0003: haven't heard from client c5a38e3b-5e56-4e49-91a5-fbbc9b23a8fd (at 10.51.8.44@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903b084b0c00, cur 1713486859 expire 1713486709 last 1713486632 [3222692.078206] Lustre: oak-MDT0000: haven't heard from client 8a02442f-5697-4111-8a8a-8e96f6109598 (at 10.51.12.16@o2ib3) in 200 seconds. I think it's dead, and I am evicting it. exp ffff902f7a88b800, cur 1713486917 expire 1713486767 last 1713486717 [3222710.074637] Lustre: oak-MDT0003: haven't heard from client 8a02442f-5697-4111-8a8a-8e96f6109598 (at 10.51.12.16@o2ib3) in 218 seconds. I think it's dead, and I am evicting it. exp ffff9040cd6c8c00, cur 1713486935 expire 1713486785 last 1713486717 [3223112.829400] LustreError: 36687:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107956104596 granted:483067692, total:351817889016 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:351817889016 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3223112.859932] LustreError: 36687:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 180 previous similar messages [3223266.057543] Lustre: oak-MDT0003: haven't heard from client 857f42d4-2d75-4779-9337-a9b3a725adfb (at 10.51.8.41@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90186bf31400, cur 1713487491 expire 1713487341 last 1713487264 [3223712.905284] LustreError: 35175:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107956010552 granted:483067692, total:351864037956 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:351864037956 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3223712.935822] LustreError: 35175:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 1770 previous similar messages [3224313.210682] LustreError: 36690:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107940627916 granted:424654352, total:351895499956 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:351895499956 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3224313.241448] LustreError: 36690:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 936 previous similar messages [3224914.411233] LustreError: 35161:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107956011480 granted:483067692, total:351973100056 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:351973100056 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3224914.441776] LustreError: 35161:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 147 previous similar messages [3225200.954066] Lustre: oak-MDT0000: haven't heard from client 774853ac-cfc9-4512-a404-44fcb4f091bd (at 10.51.8.43@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901e29161000, cur 1713489426 expire 1713489276 last 1713489199 [3225200.975940] Lustre: Skipped 1 previous similar message [3225205.957591] Lustre: oak-MDT0003: haven't heard from client 774853ac-cfc9-4512-a404-44fcb4f091bd (at 10.51.8.43@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900e3a502000, cur 1713489431 expire 1713489281 last 1713489204 [3225515.271900] LustreError: 36660:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069535405480 granted:559613176, total:351952810552 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:351952810552 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3225515.302441] LustreError: 36660:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 931 previous similar messages [3226115.246559] LustreError: 35155:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069535323576 granted:559613176, total:352179357112 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352179357112 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3226115.277265] LustreError: 35155:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 1598 previous similar messages [3226352.901373] Lustre: oak-MDT0000: haven't heard from client b1a3f64c-2e9d-45ce-8bd5-9311c24faabe (at 10.51.8.34@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903bb77e7400, cur 1713490578 expire 1713490428 last 1713490351 [3226428.895790] Lustre: oak-MDT0000: haven't heard from client 29f82974-68e6-4f2b-b087-4a89a6166b0e (at 10.51.1.68@o2ib3) in 211 seconds. I think it's dead, and I am evicting it. exp ffff9010ade73c00, cur 1713490654 expire 1713490504 last 1713490443 [3226428.917667] Lustre: Skipped 1 previous similar message [3226716.867403] LustreError: 35221:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107933865156 granted:424654352, total:352405890520 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352405890520 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3226716.897941] LustreError: 35221:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 6022 previous similar messages [3226948.713361] Lustre: 31324:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1713490496/real 1713490496] req@ffff9003010d3f00 x1793451139503680/t0(0) o6->oak-OST005f-osc-MDT0000@10.0.2.106@o2ib5:28/4 lens 544/432 e 20 to 1 dl 1713491171 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [3226948.742435] Lustre: oak-OST005f-osc-MDT0000: Connection to oak-OST005f (at 10.0.2.106@o2ib5) was lost; in progress operations using this service will wait for recovery to complete [3226948.769153] Lustre: oak-OST005f-osc-MDT0000: Connection restored to 10.0.2.106@o2ib5 (at 10.0.2.106@o2ib5) [3227318.121546] LustreError: 35234:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107956000640 granted:483067692, total:352334802356 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352334802356 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3227318.152084] LustreError: 35234:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 482 previous similar messages [3227921.159379] LustreError: 35158:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107838072724 granted:310379184, total:352397724232 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352397724232 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3227921.190090] LustreError: 35158:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 151 previous similar messages [3228526.518421] LustreError: 35199:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107928230184 granted:486160868, total:352506804388 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352506804388 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3228526.518422] LustreError: 36670:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107814850800 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3228526.518425] LustreError: 36670:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 130 previous similar messages [3228555.803281] Lustre: oak-MDT0000: haven't heard from client 0053c51f-1e76-405d-86db-c801d6e4bb00 (at 10.51.11.13@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902c35a1c000, cur 1713492781 expire 1713492631 last 1713492554 [3228555.825242] Lustre: Skipped 1 previous similar message [3228562.810531] Lustre: oak-MDT0003: haven't heard from client 0053c51f-1e76-405d-86db-c801d6e4bb00 (at 10.51.11.13@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9040fab94c00, cur 1713492788 expire 1713492638 last 1713492561 [3228818.784819] Lustre: oak-MDT0000: haven't heard from client 19ac2c80-b1b1-4674-8bb8-b20c5b20c751 (at 10.51.13.2@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90194fe54000, cur 1713493044 expire 1713492894 last 1713492817 [3229128.368016] LustreError: 36682:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107933353612 granted:424654352, total:352632649512 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352632649512 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3229128.398563] LustreError: 36682:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 1374 previous similar messages [3229728.380145] LustreError: 36692:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107932417176 granted:424654352, total:352710261404 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352710261404 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3229728.410684] LustreError: 36692:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 713 previous similar messages [3230337.525595] LustreError: 35220:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107955508592 granted:483067692, total:352674662720 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352674662720 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3230337.525597] LustreError: 35237:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3230337.525601] LustreError: 35237:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 131 previous similar messages [3230940.746709] LustreError: 35178:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069535405472 granted:559613176, total:352522124672 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352522124672 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3230940.777248] LustreError: 35178:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 167 previous similar messages [3231544.437293] LustreError: 36659:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107814850800 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3231544.466970] LustreError: 36659:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 139 previous similar messages [3232172.387532] LustreError: 35146:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3232172.417382] LustreError: 35146:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 133 previous similar messages [3232394.619341] Lustre: oak-MDT0000: haven't heard from client 437c6296-2eda-43ca-b8ef-62fa11626601 (at 10.50.5.72@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90169afed000, cur 1713496620 expire 1713496470 last 1713496393 [3232394.641218] Lustre: Skipped 1 previous similar message [3232772.384721] LustreError: 35226:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107955640872 granted:483067692, total:352747243648 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352747243648 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3232772.415267] LustreError: 35226:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 3490 previous similar messages [3233380.261678] LustreError: 35218:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107931718236 granted:424654352, total:353014030388 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353014030388 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3233380.292215] LustreError: 35218:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 3408 previous similar messages [3233676.863453] Lustre: 31358:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1713497006/real 1713497006] req@ffff903d9e99de80 x1793451332070912/t0(0) o6->oak-OST004d-osc-MDT0000@10.0.2.106@o2ib5:28/4 lens 544/432 e 7 to 1 dl 1713497681 ref 1 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'' [3233676.892441] Lustre: oak-OST004d-osc-MDT0000: Connection to oak-OST004d (at 10.0.2.106@o2ib5) was lost; in progress operations using this service will wait for recovery to complete [3233676.919175] Lustre: oak-OST004d-osc-MDT0000: Connection restored to 10.0.2.106@o2ib5 (at 10.0.2.106@o2ib5) [3233980.309048] LustreError: 35234:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107937562736 granted:424654352, total:353040239944 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353040239944 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3233980.339582] LustreError: 35234:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 4037 previous similar messages [3234381.525339] Lustre: oak-MDT0000: haven't heard from client 031c185f-7eeb-43f2-84e7-447ddc36a427 (at 10.50.8.47@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902092d21800, cur 1713498607 expire 1713498457 last 1713498380 [3234381.547217] Lustre: Skipped 3 previous similar messages [3234583.525500] LustreError: 35237:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107937051900 granted:424654352, total:353137518992 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353137518992 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3234583.556039] LustreError: 35237:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 2742 previous similar messages [3235187.166630] LustreError: 35169:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107937996252 granted:424654352, total:353090909816 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353090909816 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3235187.197168] LustreError: 35169:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 278 previous similar messages [3235800.302539] LustreError: 36695:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107939821564 granted:424654352, total:353046597960 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353046597960 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3235800.333076] LustreError: 36695:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 144 previous similar messages [3236400.349839] LustreError: 36691:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107940702756 granted:424654352, total:352964811068 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352964811068 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3236400.380386] LustreError: 36691:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 828 previous similar messages [3237005.843066] LustreError: 35208:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107814850800 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3237005.872742] LustreError: 35208:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 1028 previous similar messages [3237605.888411] LustreError: 35144:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107929109224 granted:486160868, total:353157957228 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353157957228 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3237605.918955] LustreError: 35144:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 113 previous similar messages [3238205.963832] LustreError: 35208:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069518429568 granted:559613176, total:353037825884 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353037825884 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3238205.994374] LustreError: 35208:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 4622 previous similar messages [3238596.330358] Lustre: oak-MDT0000: haven't heard from client 7edeb6dc-ffbf-4dbf-95a8-1f6abcda1154 (at 10.50.10.3@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900d3c358400, cur 1713502822 expire 1713502672 last 1713502595 [3238596.352236] Lustre: Skipped 1 previous similar message [3238672.326113] Lustre: oak-MDT0003: haven't heard from client b0a52542-3f71-4679-96ec-8083d4718df4 (at 10.50.3.58@o2ib2) in 185 seconds. I think it's dead, and I am evicting it. exp ffff901d5adbf000, cur 1713502898 expire 1713502748 last 1713502713 [3238672.347986] Lustre: Skipped 1 previous similar message [3238805.995166] LustreError: 36650:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069485632304 granted:559613176, total:353007867268 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353007867268 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3238806.025711] LustreError: 36650:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 27726 previous similar messages [3239405.972440] LustreError: 36693:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107921000396 granted:424654352, total:353125070404 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353125070404 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3239406.002983] LustreError: 36693:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 13651 previous similar messages [3240006.926946] LustreError: 35175:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3240006.956965] LustreError: 35175:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 11631 previous similar messages [3240565.239509] Lustre: oak-MDT0003: haven't heard from client d63ead59-fe45-4c20-b4e9-248b0f47f380 (at 10.50.12.15@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901a42d28800, cur 1713504791 expire 1713504641 last 1713504564 [3240565.261472] Lustre: Skipped 1 previous similar message [3240582.236804] Lustre: oak-MDT0000: haven't heard from client d63ead59-fe45-4c20-b4e9-248b0f47f380 (at 10.50.12.15@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9002fce60800, cur 1713504808 expire 1713504658 last 1713504581 [3240607.257699] LustreError: 35151:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107849032272 granted:310379184, total:353193717916 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353193717916 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3240607.288238] LustreError: 35151:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 1735 previous similar messages [3241210.178564] LustreError: 35157:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069533390804 granted:559613176, total:353002161152 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353002161152 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3241210.209112] LustreError: 35157:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 267 previous similar messages [3241492.195626] Lustre: oak-MDT0000: haven't heard from client 2b74f382-f827-4e6b-88c4-0edb0e47aa2a (at 10.51.3.61@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90247fb59800, cur 1713505718 expire 1713505568 last 1713505491 [3241504.193620] Lustre: oak-MDT0003: haven't heard from client 2b74f382-f827-4e6b-88c4-0edb0e47aa2a (at 10.51.3.61@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902e24546800, cur 1713505730 expire 1713505580 last 1713505503 [3241504.215493] Lustre: Skipped 5 previous similar messages [3241811.526883] LustreError: 35152:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107850084884 granted:310379184, total:352903041976 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352903041976 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3241811.557436] LustreError: 35152:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 168 previous similar messages [3242067.168206] Lustre: oak-MDT0003: haven't heard from client f6a37995-e002-4e8b-8b4a-f26ac0aa191c (at 10.50.16.8@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902a5ca6e000, cur 1713506293 expire 1713506143 last 1713506066 [3242067.190088] Lustre: Skipped 5 previous similar messages [3242386.076039] LustreError: 44917:0:(llog_cat.c:418:llog_cat_id2handle()) oak-MDD0003: error opening log id [0x29553:0x1:0x0]:0: rc = -2 [3242386.088222] LustreError: 44917:0:(mdd_device.c:441:llog_changelog_cancel()) oak-MDD0003: cancel idx 28533 of catalog [0x2c:0xa:0x0]: rc = -2 [3242416.403915] LustreError: 35180:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107851018124 granted:310379184, total:352682603972 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352682603972 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3242416.434454] LustreError: 35180:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 168 previous similar messages [3242593.149500] Lustre: oak-MDT0000: haven't heard from client dd38608f-3a91-4fa3-ba3a-0a1e2ad74440 (at 10.51.1.50@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90299c114800, cur 1713506819 expire 1713506669 last 1713506592 [3242593.171373] Lustre: Skipped 1 previous similar message [3242604.155482] Lustre: oak-MDT0003: haven't heard from client dd38608f-3a91-4fa3-ba3a-0a1e2ad74440 (at 10.51.1.50@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9020d51ae400, cur 1713506830 expire 1713506680 last 1713506603 [3242998.126583] Lustre: oak-MDT0003: haven't heard from client 74a8bc20-a6d6-4ff7-8e00-545bba1b6032 (at 10.50.1.13@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90240c348c00, cur 1713507224 expire 1713507074 last 1713506997 [3243022.620527] LustreError: 35178:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3243022.650385] LustreError: 35178:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 167 previous similar messages [3243629.070280] LustreError: 31507:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107941096592 granted:424654352, total:352756276780 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352756276780 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3243629.100824] LustreError: 31507:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 152 previous similar messages [3244233.758843] LustreError: 35192:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107850071724 granted:310379184, total:352833883048 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352833883048 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3244233.789386] LustreError: 35192:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 973 previous similar messages [3244467.058929] Lustre: oak-MDT0000: haven't heard from client 3bbea537-b94a-4ae3-b4a4-e3d5274b2774 (at 10.51.4.55@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9012f7a87800, cur 1713508693 expire 1713508543 last 1713508466 [3244467.080802] Lustre: Skipped 3 previous similar messages [3244833.771030] LustreError: 31510:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107849129368 granted:310379184, total:352896807072 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352896807072 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3244833.801566] LustreError: 31510:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 8609 previous similar messages [3245436.129992] LustreError: 35151:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3245436.159675] LustreError: 35151:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 2795 previous similar messages [3246037.854152] LustreError: 35196:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107851018136 granted:310379184, total:352843787064 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352843787064 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3246037.885009] LustreError: 35196:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 127 previous similar messages [3246643.722671] LustreError: 36689:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107940150200 granted:424654352, total:352806253336 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352806253336 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3246643.753329] LustreError: 36689:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 610 previous similar messages [3247146.930446] Lustre: oak-MDT0003: haven't heard from client c02a00de-c3dd-4e75-8402-977add78ef35 (at 10.51.12.4@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9039545ccc00, cur 1713511373 expire 1713511223 last 1713511146 [3247146.952325] Lustre: Skipped 1 previous similar message [3247248.718294] LustreError: 35184:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3247248.748317] LustreError: 35184:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 129 previous similar messages [3247852.230611] LustreError: 31510:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107930909180 granted:486160868, total:352769177596 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352769177596 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3247852.261150] LustreError: 31510:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 140 previous similar messages [3248465.213304] LustreError: 35155:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107851003740 granted:310379184, total:352814294208 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352814294208 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3248465.243850] LustreError: 35155:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 245 previous similar messages [3249065.225488] LustreError: 35192:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107851003740 granted:310379184, total:352830024096 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352830024096 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3249065.256036] LustreError: 35192:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 115 previous similar messages [3249665.237609] LustreError: 35179:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107851003740 granted:310379184, total:352828975520 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352828975520 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3249665.268156] LustreError: 35179:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 125 previous similar messages [3250265.249644] LustreError: 35198:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107851003740 granted:310379184, total:352828975520 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352828975520 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3250265.280187] LustreError: 35198:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 98 previous similar messages [3250275.789180] Lustre: oak-MDT0003: haven't heard from client 0ddba76d-03ae-49f9-be25-b67fd3361e07 (at 10.50.1.11@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9029c9aac000, cur 1713514502 expire 1713514352 last 1713514275 [3250275.811053] Lustre: Skipped 1 previous similar message [3250865.261926] LustreError: 35204:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107851003740 granted:310379184, total:352828975520 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352828975520 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3250865.292467] LustreError: 35204:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 86 previous similar messages [3251469.919016] LustreError: 35176:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107930882712 granted:486160868, total:352828975520 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352828975520 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3251469.919018] LustreError: 35195:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107814850800 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3251469.919020] LustreError: 35195:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 144 previous similar messages [3252069.997295] LustreError: 35181:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107851003052 granted:310379184, total:352828975520 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352828975520 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3252070.027841] LustreError: 35181:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 97 previous similar messages [3252240.698014] Lustre: oak-MDT0000: haven't heard from client 02650f6a-7e55-4269-adc8-72da9dd8ab42 (at 10.51.1.38@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9020e7480c00, cur 1713516467 expire 1713516317 last 1713516240 [3252240.719888] Lustre: Skipped 1 previous similar message [3252250.697786] Lustre: oak-MDT0003: haven't heard from client 02650f6a-7e55-4269-adc8-72da9dd8ab42 (at 10.51.1.38@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90133f87d800, cur 1713516477 expire 1713516327 last 1713516250 [3252670.044665] LustreError: 35192:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107851003052 granted:310379184, total:352828975520 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352828975520 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3252670.075203] LustreError: 35192:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 147 previous similar messages [3253270.092014] LustreError: 35152:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107851003052 granted:310379184, total:352828975520 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352828975520 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3253270.122558] LustreError: 35152:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 104 previous similar messages [3253870.139205] LustreError: 36672:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107851003052 granted:310379184, total:352828975520 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352828975520 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3253870.170048] LustreError: 36672:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 88 previous similar messages [3254473.483547] LustreError: 35195:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069534712408 granted:559613176, total:352828975520 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352828975520 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3254473.514085] LustreError: 35195:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 78 previous similar messages [3254966.577366] Lustre: oak-MDT0003: haven't heard from client afec2b30-4a6e-4e5c-be7f-d3363bcbc56e (at 10.50.1.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902f2e038c00, cur 1713519193 expire 1713519043 last 1713518966 [3254985.574413] Lustre: oak-MDT0000: haven't heard from client afec2b30-4a6e-4e5c-be7f-d3363bcbc56e (at 10.50.1.37@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902f2e03a400, cur 1713519212 expire 1713519062 last 1713518985 [3255042.569856] Lustre: oak-MDT0003: haven't heard from client a58fa77a-2bed-4997-95cf-a92e1bc4d6c7 (at 10.50.10.42@o2ib2) in 153 seconds. I think it's dead, and I am evicting it. exp ffff903a84c47000, cur 1713519269 expire 1713519119 last 1713519116 [3255061.566325] Lustre: oak-MDT0000: haven't heard from client a58fa77a-2bed-4997-95cf-a92e1bc4d6c7 (at 10.50.10.42@o2ib2) in 172 seconds. I think it's dead, and I am evicting it. exp ffff903cd31dd000, cur 1713519288 expire 1713519138 last 1713519116 [3255074.688703] LustreError: 35156:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107850884356 granted:310379184, total:352828975520 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352828975520 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3255074.719247] LustreError: 35156:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 102 previous similar messages [3255552.546842] Lustre: oak-MDT0003: haven't heard from client e6fe9f6d-9b56-4508-afe1-7d040574f140 (at 10.50.14.9@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9002a7d81c00, cur 1713519779 expire 1713519629 last 1713519552 [3255674.959478] LustreError: 36672:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107848969316 granted:310379184, total:352828975520 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352828975520 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3255674.990017] LustreError: 36672:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 552 previous similar messages [3256066.523308] Lustre: oak-MDT0000: haven't heard from client d5ad461b-4c49-4bf9-aafb-1783872a1731 (at 10.51.3.71@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9037fd5b4000, cur 1713520293 expire 1713520143 last 1713520066 [3256066.545357] Lustre: Skipped 1 previous similar message [3256284.279724] LustreError: 35226:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3256284.309744] LustreError: 35226:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 677 previous similar messages [3256617.494028] Lustre: oak-MDT0000: haven't heard from client ab7b0054-23ee-4d38-b128-b2989527314d (at 10.50.1.25@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9037e023a800, cur 1713520844 expire 1713520694 last 1713520617 [3256617.515912] Lustre: Skipped 1 previous similar message [3256890.613019] LustreError: 35151:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3256890.642694] LustreError: 35151:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 78 previous similar messages [3257492.681626] LustreError: 36694:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107940807512 granted:424654352, total:352828975520 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352828975520 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3257492.712327] LustreError: 36694:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 78 previous similar messages [3258093.048991] LustreError: 36667:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069534623044 granted:559613176, total:352828975520 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352828975520 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3258093.079534] LustreError: 36667:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 73 previous similar messages [3258707.881349] LustreError: 35206:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107930882704 granted:486160868, total:352828975520 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352828975520 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3258707.881353] LustreError: 36666:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107814850800 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3258707.881360] LustreError: 36666:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 81 previous similar messages [3259309.907180] LustreError: 35176:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3259309.937026] LustreError: 35176:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 77 previous similar messages [3259925.384204] LustreError: 35205:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3259925.413875] LustreError: 35205:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 80 previous similar messages [3260525.396141] LustreError: 36659:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3260525.425815] LustreError: 36659:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 88 previous similar messages [3261136.169197] LustreError: 36666:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3261136.198875] LustreError: 36666:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 86 previous similar messages [3261483.265810] Lustre: oak-MDT0000: haven't heard from client 18bb2daf-bcf1-4e40-ab30-c91255d8ecab (at 10.51.3.15@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902443db3000, cur 1713525710 expire 1713525560 last 1713525483 [3261483.287691] Lustre: Skipped 1 previous similar message [3261773.857629] LustreError: 36691:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107955229284 granted:483067692, total:352828975520 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352828975520 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3261773.857631] LustreError: 36694:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3261773.857634] LustreError: 36694:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 85 previous similar messages [3262395.450995] LustreError: 35236:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107940807512 granted:424654352, total:352828975520 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352828975520 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3262395.481536] LustreError: 35236:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 82 previous similar messages [3263000.431949] LustreError: 35158:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3263000.461794] LustreError: 35158:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 77 previous similar messages [3263600.443259] LustreError: 35155:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3263600.473112] LustreError: 35155:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 81 previous similar messages [3264210.190926] LustreError: 35170:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107940807512 granted:424654352, total:352828975520 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352828975520 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3264210.221472] LustreError: 35170:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 82 previous similar messages [3264827.021572] LustreError: 35166:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3264827.051586] LustreError: 35166:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 72 previous similar messages [3265433.169209] LustreError: 36653:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069534623044 granted:559613176, total:352828975520 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352828975520 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3265433.199754] LustreError: 36653:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 83 previous similar messages [3265627.074830] Lustre: oak-MDT0000: haven't heard from client f184b639-a51a-4c78-aeff-4e2128bd9733 (at 10.51.3.29@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902e46f31400, cur 1713529854 expire 1713529704 last 1713529627 [3265627.096707] Lustre: Skipped 1 previous similar message [3266035.797127] LustreError: 36689:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3266035.827147] LustreError: 36689:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 79 previous similar messages [3266635.867710] LustreError: 36656:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3266635.897379] LustreError: 36656:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 72 previous similar messages [3267244.041357] LustreError: 36700:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107955229284 granted:483067692, total:352828975520 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352828975520 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3267244.071897] LustreError: 36700:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 82 previous similar messages [3267854.283496] LustreError: 35147:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3267854.313174] LustreError: 35147:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 73 previous similar messages [3268188.955779] Lustre: oak-MDT0000: haven't heard from client 2a52814e-6e5f-4b80-8702-d572412af838 (at 10.51.12.24@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9029e200dc00, cur 1713532416 expire 1713532266 last 1713532189 [3268188.977738] Lustre: Skipped 1 previous similar message [3268203.956832] Lustre: oak-MDT0003: haven't heard from client 2a52814e-6e5f-4b80-8702-d572412af838 (at 10.51.12.24@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9029e2008400, cur 1713532431 expire 1713532281 last 1713532204 [3268483.879198] LustreError: 35224:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3268483.909220] LustreError: 35224:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 80 previous similar messages [3269045.915318] Lustre: oak-MDT0000: haven't heard from client 08db67f8-ecd9-4fe4-831d-c9734cdfc2a8 (at 10.51.1.18@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900f1d466400, cur 1713533273 expire 1713533123 last 1713533046 [3269105.422889] LustreError: 35182:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3269105.452739] LustreError: 35182:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 75 previous similar messages [3269710.448030] LustreError: 36650:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107846591476 granted:310379184, total:352828975520 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352828975520 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3269710.478576] LustreError: 36650:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 87 previous similar messages [3270312.560360] LustreError: 36672:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107930882704 granted:486160868, total:352828975520 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352828975520 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3270312.590904] LustreError: 36672:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 80 previous similar messages [3270923.642350] LustreError: 35192:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3270923.672025] LustreError: 35192:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 76 previous similar messages [3271525.702694] LustreError: 36672:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107846591476 granted:310379184, total:352828975520 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352828975520 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3271525.733243] LustreError: 36672:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 82 previous similar messages [3272006.784192] Lustre: oak-MDT0003: haven't heard from client 3fbb16ad-5839-48c4-9983-a358deb31258 (at 10.50.6.19@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90067e1a9000, cur 1713536234 expire 1713536084 last 1713536007 [3272006.806067] Lustre: Skipped 3 previous similar messages [3272136.896602] LustreError: 35181:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107814850800 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3272136.926280] LustreError: 35181:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 81 previous similar messages [3272738.702041] LustreError: 36653:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069534623044 granted:559613176, total:352828975520 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352828975520 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3272738.732578] LustreError: 36653:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 75 previous similar messages [3273299.720676] Lustre: oak-MDT0000: haven't heard from client 4ef8443e-a735-4f0f-bdd9-857a350968e0 (at 10.51.11.13@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902994dc5c00, cur 1713537527 expire 1713537377 last 1713537300 [3273299.742770] Lustre: Skipped 1 previous similar message [3273346.405911] LustreError: 36693:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3273346.405914] LustreError: 35234:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107955229284 granted:483067692, total:352828975520 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352828975520 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3273346.405918] LustreError: 35234:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 78 previous similar messages [3273949.646497] LustreError: 35195:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3273949.676178] LustreError: 35195:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 75 previous similar messages [3274553.938321] LustreError: 36696:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107940807512 granted:424654352, total:352828975520 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352828975520 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3274553.968866] LustreError: 36696:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 82 previous similar messages [3274833.653668] Lustre: oak-MDT0000: haven't heard from client 8214dd68-ec77-4062-a53f-0197008bd76f (at 10.51.4.36@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902876d5e000, cur 1713539061 expire 1713538911 last 1713538834 [3274833.675543] Lustre: Skipped 1 previous similar message [3274909.648912] Lustre: oak-MDT0003: haven't heard from client 52fa3289-01e2-4645-ba2a-cde8ea3e59d7 (at 10.50.8.47@o2ib2) in 224 seconds. I think it's dead, and I am evicting it. exp ffff90189b8de800, cur 1713539137 expire 1713538987 last 1713538913 [3274909.670795] Lustre: Skipped 3 previous similar messages [3275154.436329] LustreError: 36655:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107846591476 granted:310379184, total:352828975520 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352828975520 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3275154.466887] LustreError: 36655:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 83 previous similar messages [3275276.639088] Lustre: oak-MDT0000: haven't heard from client f8e6ba86-1cd5-4431-9170-cc82d7f7b080 (at 10.51.10.71@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90143ea80000, cur 1713539504 expire 1713539354 last 1713539277 [3275276.661047] Lustre: Skipped 15 previous similar messages [3275755.605220] LustreError: 36692:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107955229284 granted:483067692, total:352828975520 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352828975520 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3275755.635753] LustreError: 36692:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 78 previous similar messages [3276022.605871] Lustre: oak-MDT0000: haven't heard from client 6b40efb7-6e4e-4d41-b37a-da6de90de08a (at 10.50.12.10@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9005bf817c00, cur 1713540250 expire 1713540100 last 1713540023 [3276022.627840] Lustre: Skipped 1 previous similar message [3276038.603759] Lustre: oak-MDT0003: haven't heard from client 6b40efb7-6e4e-4d41-b37a-da6de90de08a (at 10.50.12.10@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902a5ca6dc00, cur 1713540266 expire 1713540116 last 1713540039 [3276038.625722] Lustre: Skipped 1 previous similar message [3276355.617741] LustreError: 35224:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107955229284 granted:483067692, total:352828975520 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352828975520 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3276355.648277] LustreError: 35224:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 85 previous similar messages [3276713.570432] Lustre: oak-MDT0003: haven't heard from client bf195787-d407-49cd-bb48-fd858b41274f (at 10.50.4.49@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901f0c55b400, cur 1713540941 expire 1713540791 last 1713540714 [3276713.592315] Lustre: Skipped 1 previous similar message [3276718.563993] Lustre: oak-MDT0000: haven't heard from client bf195787-d407-49cd-bb48-fd858b41274f (at 10.50.4.49@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901e29166000, cur 1713540946 expire 1713540796 last 1713540719 [3276951.556969] Lustre: oak-MDT0000: haven't heard from client 46ee2244-0660-453a-9cf3-f63a24fdd7d2 (at 10.51.9.15@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90382339fc00, cur 1713541179 expire 1713541029 last 1713540952 [3276955.561899] Lustre: oak-MDT0003: haven't heard from client b9ef2de3-2075-4ea1-848d-335b33774c53 (at 10.51.9.23@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9028aab6cc00, cur 1713541183 expire 1713541033 last 1713540956 [3276955.583778] Lustre: Skipped 19 previous similar messages [3276955.664276] LustreError: 35169:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3276955.664278] LustreError: 36685:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107955229284 granted:483067692, total:352828975520 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352828975520 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3276955.664281] LustreError: 36685:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 85 previous similar messages [3277530.527725] Lustre: oak-MDT0000: haven't heard from client 38d6d0de-38df-4643-9aad-5c63a763814b (at 10.50.8.15@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901ccc9c3400, cur 1713541758 expire 1713541608 last 1713541531 [3277530.549600] Lustre: Skipped 19 previous similar messages [3277555.740790] LustreError: 35234:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3277555.770805] LustreError: 35234:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 86 previous similar messages [3277721.519258] Lustre: oak-MDT0003: haven't heard from client cc4becc2-da7f-4b12-8818-ec3b2099e477 (at 10.51.1.37@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903864e9ac00, cur 1713541949 expire 1713541799 last 1713541722 [3277721.541129] Lustre: Skipped 1 previous similar message [3278161.161959] LustreError: 35223:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107940807512 granted:424654352, total:352828975520 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352828975520 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3278161.192493] LustreError: 35223:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 81 previous similar messages [3278184.506388] Lustre: oak-MDT0003: haven't heard from client 9e35656b-aa20-483f-abab-7bd7ba64f65d (at 10.51.18.1@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901426083c00, cur 1713542412 expire 1713542262 last 1713542185 [3278184.528261] Lustre: Skipped 1 previous similar message [3278738.475534] Lustre: oak-MDT0000: haven't heard from client a498e4b3-b421-4998-84ed-f7a7b1a38057 (at 10.51.17.10@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9015656a6c00, cur 1713542966 expire 1713542816 last 1713542739 [3278738.497497] Lustre: Skipped 1 previous similar message [3278761.212435] LustreError: 36679:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107940807512 granted:424654352, total:352828975520 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352828975520 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3278761.242970] LustreError: 36679:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 84 previous similar messages [3279361.261044] LustreError: 35172:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107940807512 granted:424654352, total:352828975520 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352828975520 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3279361.291717] LustreError: 35172:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 80 previous similar messages [3279961.308352] LustreError: 36698:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107940807512 granted:424654352, total:352828975520 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352828975520 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3279961.338895] LustreError: 36698:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 84 previous similar messages [3280570.344894] LustreError: 35153:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3280570.374748] LustreError: 35153:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 91 previous similar messages [3281178.976997] LustreError: 35206:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3281179.006848] LustreError: 35206:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 103 previous similar messages [3281799.401649] LustreError: 35196:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3281799.431491] LustreError: 35196:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 115 previous similar messages [3281941.323310] Lustre: oak-MDT0003: haven't heard from client 0853a8f7-dafa-4228-83a4-3c58c1d5d1d9 (at 10.50.10.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90384fa2c800, cur 1713546169 expire 1713546019 last 1713545942 [3281941.345195] Lustre: Skipped 1 previous similar message [3281952.329887] Lustre: oak-MDT0000: haven't heard from client 0853a8f7-dafa-4228-83a4-3c58c1d5d1d9 (at 10.50.10.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9020788f5400, cur 1713546180 expire 1713546030 last 1713545953 [3282326.306725] Lustre: oak-MDT0000: haven't heard from client 4da52ed1-082c-4979-b69a-1c5eac2e54d5 (at 10.50.10.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9009cc812000, cur 1713546554 expire 1713546404 last 1713546327 [3282399.413309] LustreError: 35207:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3282399.443168] LustreError: 35207:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 91 previous similar messages [3282999.424708] LustreError: 35200:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3282999.454557] LustreError: 35200:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 105 previous similar messages [3283601.862065] LustreError: 35202:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107846591476 granted:310379184, total:352828975520 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352828975520 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3283601.892605] LustreError: 35202:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 103 previous similar messages [3283939.233401] Lustre: oak-MDT0003: haven't heard from client 9c3e4cde-9e3c-4ca4-8f9f-fd8670101eb5 (at 10.51.5.62@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9018cf369000, cur 1713548167 expire 1713548017 last 1713547940 [3283939.255283] Lustre: Skipped 15 previous similar messages [3284204.631320] LustreError: 35202:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3284204.661172] LustreError: 35202:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 115 previous similar messages [3284806.656455] LustreError: 35204:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107930882552 granted:486160868, total:352827926944 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352827926944 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3284806.686997] LustreError: 35204:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 100 previous similar messages [3285407.653004] LustreError: 36677:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3285407.683015] LustreError: 36677:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 98 previous similar messages [3286007.701110] LustreError: 36699:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3286007.731122] LustreError: 36699:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 82 previous similar messages [3286159.129645] Lustre: oak-MDT0003: haven't heard from client fbba125c-8de3-45bd-a5ef-20cfa6a9b3f5 (at 10.51.5.57@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903b4ea05000, cur 1713550387 expire 1713550237 last 1713550160 [3286159.151516] Lustre: Skipped 5 previous similar messages [3286235.126035] Lustre: oak-MDT0003: haven't heard from client 58bb3a45-c3a6-4c01-ad0b-a405331641b2 (at 10.50.17.25@o2ib2) in 217 seconds. I think it's dead, and I am evicting it. exp ffff903c4b211c00, cur 1713550463 expire 1713550313 last 1713550246 [3286235.148003] Lustre: Skipped 1 previous similar message [3286425.117207] Lustre: oak-MDT0003: haven't heard from client cd5e3950-73df-44e4-9e46-ad3dfd75e770 (at 10.51.2.49@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900f3887b000, cur 1713550653 expire 1713550503 last 1713550426 [3286425.139081] Lustre: Skipped 3 previous similar messages [3286594.110254] Lustre: oak-MDT0003: haven't heard from client f8ae384d-b1a7-4530-9fd6-c5bb5effd0c2 (at 10.51.2.68@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9030424a3400, cur 1713550822 expire 1713550672 last 1713550595 [3286594.132129] Lustre: Skipped 13 previous similar messages [3286607.713405] LustreError: 35229:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107952101684 granted:483067692, total:352506669788 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352506669788 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3286607.743941] LustreError: 35229:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 797 previous similar messages [3286939.094323] Lustre: oak-MDT0003: haven't heard from client 7ef35428-db2e-458d-8f36-e7bf23b10393 (at 10.51.4.60@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903d1b4b7400, cur 1713551167 expire 1713551017 last 1713550940 [3286939.116201] Lustre: Skipped 1 previous similar message [3287015.089156] Lustre: oak-MDT0000: haven't heard from client dbe2bb20-df12-4d25-b10e-281bbad8f47c (at 10.51.3.5@o2ib3) in 176 seconds. I think it's dead, and I am evicting it. exp ffff90073d8ca800, cur 1713551243 expire 1713551093 last 1713551067 [3287015.110947] Lustre: Skipped 1 previous similar message [3287207.725669] LustreError: 36676:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3287207.725671] LustreError: 36685:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107948250292 granted:483067692, total:352506669788 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352506669788 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3287207.725674] LustreError: 36685:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 979 previous similar messages [3287809.193455] LustreError: 35234:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107944485412 granted:483067692, total:352506669788 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352506669788 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3287809.223996] LustreError: 35234:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 1014 previous similar messages [3288156.037329] Lustre: oak-MDT0000: haven't heard from client 7fd81010-086c-4f8b-b7d3-5647ef1d0aa3 (at 10.50.6.15@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901248370c00, cur 1713552384 expire 1713552234 last 1713552157 [3288156.059209] Lustre: Skipped 1 previous similar message [3288409.678239] LustreError: 35234:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107940677288 granted:483067692, total:352506669788 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352506669788 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3288409.708778] LustreError: 35234:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 2049 previous similar messages [3288711.011949] Lustre: oak-MDT0000: haven't heard from client c64885d2-8719-4b2f-8c22-87828253ad4d (at 10.51.3.36@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90073d8f9c00, cur 1713552939 expire 1713552789 last 1713552712 [3288711.033828] Lustre: Skipped 3 previous similar messages [3288718.014218] Lustre: oak-MDT0003: haven't heard from client c64885d2-8719-4b2f-8c22-87828253ad4d (at 10.51.3.36@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901bf3fd1800, cur 1713552946 expire 1713552796 last 1713552719 [3288900.007282] Lustre: oak-MDT0003: haven't heard from client 526a248a-5c04-41f3-a768-5089151081e1 (at 10.50.6.30@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903cf8bc2800, cur 1713553128 expire 1713552978 last 1713552901 [3289010.507674] LustreError: 36688:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107936891060 granted:483067692, total:352583235148 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352583235148 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3289010.538211] LustreError: 36688:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 952 previous similar messages [3289333.982058] Lustre: oak-MDT0003: haven't heard from client c83ad6d8-108f-4e89-824b-1dc32b17f490 (at 10.50.10.35@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900e37941400, cur 1713553562 expire 1713553412 last 1713553335 [3289334.004314] Lustre: Skipped 1 previous similar message [3289611.127743] LustreError: 35236:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107933139748 granted:483067692, total:352583235148 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352583235148 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3289611.158284] LustreError: 35236:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 966 previous similar messages [3290084.948351] Lustre: oak-MDT0003: haven't heard from client f7e98f1e-78be-4104-9557-6d7e4fa302bb (at 10.51.1.66@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901190e7b000, cur 1713554313 expire 1713554163 last 1713554086 [3290084.970226] Lustre: Skipped 3 previous similar messages [3290214.861001] LustreError: 36661:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069534590012 granted:559613176, total:352577553776 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352577553776 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3290214.891547] LustreError: 36661:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 1096 previous similar messages [3290815.377297] LustreError: 36695:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107927753892 granted:483067692, total:352581566352 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352581566352 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3290815.407830] LustreError: 36695:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 1232 previous similar messages [3291415.524036] LustreError: 31510:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069531061240 granted:559613176, total:352601997764 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352601997764 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3291415.554580] LustreError: 31510:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 1612 previous similar messages [3292017.726574] LustreError: 35185:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107920210084 granted:483067692, total:352613105164 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352613105164 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3292017.757112] LustreError: 35185:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 1333 previous similar messages [3292232.849685] Lustre: oak-MDT0000: haven't heard from client ab43b0d0-4c50-40db-b45a-69b5037d16f8 (at 10.51.14.9@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900463944c00, cur 1713556461 expire 1713556311 last 1713556234 [3292232.871562] Lustre: Skipped 1 previous similar message [3292237.850402] Lustre: oak-MDT0003: haven't heard from client ab43b0d0-4c50-40db-b45a-69b5037d16f8 (at 10.51.14.9@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900e68218000, cur 1713556466 expire 1713556316 last 1713556239 [3292416.844230] Lustre: oak-MDT0000: haven't heard from client 13cbccca-050e-4fc9-bfb0-25e2785fc51b (at 10.51.7.30@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90073d9ae400, cur 1713556645 expire 1713556495 last 1713556418 [3292422.848505] Lustre: oak-MDT0003: haven't heard from client 81e15f82-8622-4ccc-afdf-b7f22717a0ba (at 10.51.7.5@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903cb3687000, cur 1713556651 expire 1713556501 last 1713556424 [3292422.870291] Lustre: Skipped 2 previous similar messages [3292618.536173] LustreError: 36676:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107916406120 granted:483067692, total:352608416312 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352608416312 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3292618.566714] LustreError: 36676:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 1014 previous similar messages [3292622.830584] Lustre: oak-MDT0003: haven't heard from client b3b12f4e-904a-4eab-a0c6-a5fdbeea18d4 (at 10.51.13.21@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9032bf330c00, cur 1713556851 expire 1713556701 last 1713556624 [3292622.852666] Lustre: Skipped 2 previous similar messages [3292810.828240] Lustre: oak-MDT0000: haven't heard from client 9f9c23b5-e291-4570-90ff-3e39d4766da6 (at 10.51.12.12@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9026faf65000, cur 1713557039 expire 1713556889 last 1713556812 [3292810.850203] Lustre: Skipped 1 previous similar message [3293218.790069] LustreError: 36677:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107938569452 granted:424654352, total:352600506492 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352600506492 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3293218.820605] LustreError: 36677:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 1241 previous similar messages [3293819.079039] LustreError: 35223:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107908821996 granted:483067692, total:352625628892 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352625628892 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3293819.109581] LustreError: 35223:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 1659 previous similar messages [3294420.413707] LustreError: 36654:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107930658556 granted:486160868, total:352653263496 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352653263496 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3294420.444254] LustreError: 36654:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 1647 previous similar messages [3294641.737295] Lustre: oak-MDT0000: haven't heard from client 882b7810-a93f-499f-9b35-23bb0eaf94f7 (at 10.51.1.42@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900f3efa5000, cur 1713558870 expire 1713558720 last 1713558643 [3294641.759174] Lustre: Skipped 1 previous similar message [3295021.174550] LustreError: 35208:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3295021.204401] LustreError: 35208:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 1615 previous similar messages [3295206.710991] Lustre: oak-MDT0003: haven't heard from client 97869943-0cb4-4a8c-9c56-0c55ac353850 (at 10.50.4.14@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9040fdf49800, cur 1713559435 expire 1713559285 last 1713559208 [3295206.732867] Lustre: Skipped 1 previous similar message [3295623.560629] LustreError: 36687:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107897567888 granted:483067692, total:352222586624 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352222586624 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3295623.591180] LustreError: 36687:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 1281 previous similar messages [3296224.166543] LustreError: 35223:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107893840284 granted:483067692, total:352233277376 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352233277376 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3296224.197078] LustreError: 35223:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 1178 previous similar messages [3296827.522324] LustreError: 36685:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107888930604 granted:483067692, total:352252333116 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352252333116 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3296827.552863] LustreError: 36685:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 1579 previous similar messages [3297427.559303] LustreError: 36697:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107884795008 granted:483067692, total:352275217556 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352275217556 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3297427.589846] LustreError: 36697:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 1498 previous similar messages [3298029.910751] LustreError: 36683:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107881003360 granted:483067692, total:352277704940 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352277704940 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3298029.941300] LustreError: 36683:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 1046 previous similar messages [3298629.900170] LustreError: 36666:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107953921032 granted:486160868, total:352303090728 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352303090728 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3298629.930711] LustreError: 36666:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 2054 previous similar messages [3299229.878470] LustreError: 36653:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107835440268 granted:310379184, total:352562173860 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352562173860 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3299229.909014] LustreError: 36653:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 7630 previous similar messages [3299830.143719] LustreError: 36697:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107933586484 granted:424654352, total:352851226064 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352851226064 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3299830.174259] LustreError: 36697:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 12669 previous similar messages [3300430.462610] LustreError: 35196:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069518385648 granted:559613176, total:352806228380 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352806228380 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3300430.493154] LustreError: 35196:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 2619 previous similar messages [3301030.449410] LustreError: 36668:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107816093076 granted:310379184, total:353514533436 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353514533436 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3301030.480122] LustreError: 36668:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 18412 previous similar messages [3301631.522599] LustreError: 35200:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069523877808 granted:559613176, total:353692562564 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353692562564 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3301631.553142] LustreError: 35200:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 7186 previous similar messages [3302232.566758] LustreError: 36669:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107815227456 granted:310379184, total:353763411832 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353763411832 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3302232.597299] LustreError: 36669:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 1360 previous similar messages [3302834.012743] LustreError: 36694:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107849764700 granted:483067692, total:353719249304 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353719249304 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3302834.043280] LustreError: 36694:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 178 previous similar messages [3303442.222143] LustreError: 35163:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107920341632 granted:424654352, total:353782698172 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353782698172 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3303442.252676] LustreError: 35163:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 2356 previous similar messages [3304042.378526] LustreError: 35146:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107815140680 granted:310379184, total:353884954468 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353884954468 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3304042.409068] LustreError: 35146:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 682 previous similar messages [3304644.872179] LustreError: 36698:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3304644.902193] LustreError: 36698:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 947 previous similar messages [3304913.257003] Lustre: oak-MDT0000: haven't heard from client 8f7ed9d4-afba-49c8-a5b1-39ece71d1ab5 (at 10.51.10.30@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902c6c3ae000, cur 1713569142 expire 1713568992 last 1713568915 [3304913.278966] Lustre: Skipped 1 previous similar message [3304918.256064] Lustre: oak-MDT0003: haven't heard from client ca943c28-17c4-4a34-b72f-e22b84dfbaef (at 10.51.7.54@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901d5ad8ec00, cur 1713569147 expire 1713568997 last 1713568920 [3304918.277940] Lustre: Skipped 1 previous similar message [3305252.477132] LustreError: 36661:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107815015640 granted:310379184, total:353799300636 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353799300636 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3305252.507689] LustreError: 36661:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 93 previous similar messages [3305763.216402] Lustre: oak-MDT0000: haven't heard from client 8bb9f0bf-d163-417a-955c-6c7b2feec92b (at 10.51.10.35@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900e88669400, cur 1713569992 expire 1713569842 last 1713569765 [3305763.238359] Lustre: Skipped 1 previous similar message [3305855.862868] LustreError: 36658:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107933296080 granted:486160868, total:353679950900 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353679950900 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3305855.893412] LustreError: 36658:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 80 previous similar messages [3306472.944579] LustreError: 35172:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107922492480 granted:424654352, total:353661223808 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353661223808 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3306472.975121] LustreError: 35172:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 76 previous similar messages [3306623.098278] Lustre: oak-MDT0000: index object [0x200000003:0x1665:0x0] (8/32) registered [3306623.109678] Lustre: oak-MDT0003: index object [0x200000003:0x3e7:0x0] (8/32) registered [3306894.180256] Lustre: oak-MDT0003: haven't heard from client 1e6db4a6-bb3c-4700-9983-af9c2355a292 (at 10.50.3.72@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90128802ec00, cur 1713571123 expire 1713570973 last 1713570896 [3306894.202136] Lustre: Skipped 1 previous similar message [3307086.592954] LustreError: 31507:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3307086.622973] LustreError: 31507:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 77 previous similar messages [3307311.145816] Lustre: oak-MDT0003: haven't heard from client 1419f9e9-7478-467e-a324-88287daa68f9 (at 10.50.10.35@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900b15c53000, cur 1713571540 expire 1713571390 last 1713571313 [3307311.167783] Lustre: Skipped 19 previous similar messages [3307318.146878] Lustre: oak-MDT0000: haven't heard from client 1419f9e9-7478-467e-a324-88287daa68f9 (at 10.50.10.35@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9014504d7800, cur 1713571547 expire 1713571397 last 1713571320 [3307639.135346] Lustre: oak-MDT0000: haven't heard from client 0deb59c4-26c8-421f-8b3b-e7edcfdbdd03 (at 10.51.8.70@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902991eca000, cur 1713571868 expire 1713571718 last 1713571641 [3307644.134105] Lustre: oak-MDT0003: haven't heard from client 0deb59c4-26c8-421f-8b3b-e7edcfdbdd03 (at 10.51.8.70@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903fe5164c00, cur 1713571873 expire 1713571723 last 1713571646 [3307689.481114] LustreError: 35176:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107950386852 granted:486160868, total:353645610168 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353645610168 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3307689.511654] LustreError: 35176:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 77 previous similar messages [3308290.583665] LustreError: 35221:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3308290.613690] LustreError: 35221:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 68 previous similar messages [3308859.072319] Lustre: oak-MDT0003: haven't heard from client 6fc26b77-934b-496c-a709-205fed50159a (at 10.51.14.9@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9015abb02000, cur 1713573088 expire 1713572938 last 1713572861 [3308910.139069] LustreError: 35166:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107922492480 granted:424654352, total:353616162096 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353616162096 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3308910.169604] LustreError: 35166:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 74 previous similar messages [3309046.066244] Lustre: oak-MDT0000: haven't heard from client 74edc8ed-9272-4f65-8c0a-d4012eade911 (at 10.51.15.4@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901ec4af2c00, cur 1713573275 expire 1713573125 last 1713573048 [3309046.088120] Lustre: Skipped 3 previous similar messages [3309511.265769] LustreError: 35198:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3309511.295708] LustreError: 35198:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 74 previous similar messages [3309575.038719] Lustre: oak-MDT0003: haven't heard from client 2c8ff145-b438-465b-b677-c58531c190f2 (at 10.51.15.4@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90127e8a2c00, cur 1713573804 expire 1713573654 last 1713573577 [3309575.060598] Lustre: Skipped 1 previous similar message [3309586.049165] Lustre: oak-MDT0000: haven't heard from client 2c8ff145-b438-465b-b677-c58531c190f2 (at 10.51.15.4@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9019cc209000, cur 1713573815 expire 1713573665 last 1713573588 [3310113.028726] LustreError: 35149:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3310113.058576] LustreError: 35149:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 81 previous similar messages [3310718.570662] LustreError: 36669:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3310718.600342] LustreError: 36669:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 217 previous similar messages [3311322.958011] Lustre: oak-MDT0000: haven't heard from client dd4fbc26-a0b2-4f78-8624-ed3c141a3858 (at 10.51.5.44@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901ec4aab800, cur 1713575552 expire 1713575402 last 1713575325 [3311327.123559] LustreError: 35183:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107831824680 granted:310379184, total:353600926400 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353600926400 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3311327.154098] LustreError: 35183:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 72 previous similar messages [3311929.236491] LustreError: 35158:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069533353792 granted:559613176, total:353600926400 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353600926400 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3311929.267033] LustreError: 35158:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 75 previous similar messages [3312253.914312] Lustre: oak-MDT0000: haven't heard from client c944af8c-e1db-44f4-ac13-112f6118c30e (at 10.51.15.19@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903ab0445000, cur 1713576483 expire 1713576333 last 1713576256 [3312253.936450] Lustre: Skipped 1 previous similar message [3312531.575240] LustreError: 35182:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107960533820 granted:486160868, total:353591249112 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353591249112 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3312531.605781] LustreError: 35182:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 80 previous similar messages [3312587.899292] Lustre: oak-MDT0003: haven't heard from client feef575d-c3f6-4ee9-8fee-173ce10c3f17 (at 10.51.7.70@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90143ea7f400, cur 1713576817 expire 1713576667 last 1713576590 [3312587.921312] Lustre: Skipped 1 previous similar message [3313085.876828] Lustre: oak-MDT0000: haven't heard from client c75cb14b-054f-4f92-b876-46049efd3418 (at 10.51.9.67@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90143ea79000, cur 1713577315 expire 1713577165 last 1713577088 [3313085.898705] Lustre: Skipped 1 previous similar message [3313139.849841] LustreError: 35151:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3313139.879520] LustreError: 35151:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 72 previous similar messages [3313756.163211] LustreError: 36672:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3313756.193020] LustreError: 36672:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 77 previous similar messages [3314364.432913] LustreError: 35179:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3314364.462766] LustreError: 35179:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 79 previous similar messages [3314498.809995] Lustre: oak-MDT0000: haven't heard from client ed137f0d-3402-4265-86fa-5fbfe7aa310d (at 10.51.8.49@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902078960400, cur 1713578728 expire 1713578578 last 1713578501 [3314498.831871] Lustre: Skipped 3 previous similar messages [3314981.408524] LustreError: 35207:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3314981.438201] LustreError: 35207:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 72 previous similar messages [3315596.752659] LustreError: 35166:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107925735380 granted:424654352, total:353554582536 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353554582536 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3315596.783197] LustreError: 35166:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 73 previous similar messages [3316213.037646] LustreError: 35198:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3316213.067322] LustreError: 35198:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 76 previous similar messages [3316282.730395] Lustre: oak-MDT0003: haven't heard from client 010a938e-c99d-4de7-bcc1-5109f0a90ee7 (at 10.51.11.5@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9036a1f29000, cur 1713580512 expire 1713580362 last 1713580285 [3316282.752273] Lustre: Skipped 49 previous similar messages [3316640.719947] Lustre: oak-MDT0000: haven't heard from client 61088783-b936-4472-a413-fe906da0271b (at 10.51.15.14@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901b97609400, cur 1713580870 expire 1713580720 last 1713580643 [3316640.741904] Lustre: Skipped 5 previous similar messages [3316646.711823] Lustre: oak-MDT0003: haven't heard from client 61088783-b936-4472-a413-fe906da0271b (at 10.51.15.14@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900f3887c400, cur 1713580876 expire 1713580726 last 1713580649 [3316854.849947] LustreError: 36678:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107944480028 granted:483067692, total:353540425620 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353540425620 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3316854.880882] LustreError: 36678:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 80 previous similar messages [3317469.044791] LustreError: 35206:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069533353792 granted:559613176, total:353540425620 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353540425620 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3317469.075331] LustreError: 35206:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 79 previous similar messages [3317532.671445] Lustre: oak-MDT0000: haven't heard from client d7a57cd1-5170-430b-b72e-e2468c0c3312 (at 10.51.8.29@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90064f075400, cur 1713581762 expire 1713581612 last 1713581535 [3317687.663500] Lustre: oak-MDT0003: haven't heard from client 8611d552-f27a-4687-9c4d-27533f306ee7 (at 10.51.1.51@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9038574aa000, cur 1713581917 expire 1713581767 last 1713581690 [3317687.685379] Lustre: Skipped 1 previous similar message [3317868.655940] Lustre: oak-MDT0000: haven't heard from client 011b813c-7350-4f0f-adbd-57cd0ffd3701 (at 10.51.9.28@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901d5ad37400, cur 1713582098 expire 1713581948 last 1713581871 [3317868.677970] Lustre: Skipped 1 previous similar message [3317944.652110] Lustre: oak-MDT0000: haven't heard from client d7e653c1-c25b-45a0-a836-6cbe9a0d4b9f (at 10.51.17.12@o2ib3) in 159 seconds. I think it's dead, and I am evicting it. exp ffff90207880fc00, cur 1713582174 expire 1713582024 last 1713582015 [3317944.674068] Lustre: Skipped 1 previous similar message [3318069.093983] LustreError: 36667:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069533353792 granted:559613176, total:353540425620 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353540425620 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3318069.124534] LustreError: 36667:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 176 previous similar messages [3318433.630307] Lustre: oak-MDT0000: haven't heard from client 798c4ec1-fde4-4949-b9c2-427bd8a1d494 (at 10.51.11.10@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900894e8cc00, cur 1713582663 expire 1713582513 last 1713582436 [3318433.652272] Lustre: Skipped 1 previous similar message [3318669.601681] LustreError: 35227:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107934635868 granted:424654352, total:353509148476 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353509148476 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3318669.632227] LustreError: 35227:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 390 previous similar messages [3319071.600325] Lustre: oak-MDT0003: haven't heard from client 59bcdc56-57e1-490b-abcd-8e4826a378e3 (at 10.51.8.13@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90208f4ce400, cur 1713583301 expire 1713583151 last 1713583074 [3319071.622203] Lustre: Skipped 5 previous similar messages [3319280.971757] LustreError: 35196:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107814850800 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3319280.971759] LustreError: 35143:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107964446548 granted:486160868, total:353509148476 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353509148476 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3319280.971762] LustreError: 35143:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 262 previous similar messages [3319454.589901] Lustre: oak-MDT0003: haven't heard from client 8ade3ed5-90a5-4aa8-b6c5-ba969d26e9f4 (at 10.50.4.56@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902078904000, cur 1713583684 expire 1713583534 last 1713583457 [3319454.611784] Lustre: Skipped 1 previous similar message [3319460.582971] Lustre: oak-MDT0000: haven't heard from client 8ade3ed5-90a5-4aa8-b6c5-ba969d26e9f4 (at 10.50.4.56@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901ccc9c5000, cur 1713583690 expire 1713583540 last 1713583463 [3319674.579547] Lustre: oak-MDT0003: haven't heard from client 728fa773-d76f-4442-abc3-25b31e2621d4 (at 10.51.1.18@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902855ea9400, cur 1713583904 expire 1713583754 last 1713583677 [3319881.562707] LustreError: 35147:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107963482948 granted:486160868, total:353509148476 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353509148476 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3319881.593255] LustreError: 35147:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 323 previous similar messages [3320068.555718] Lustre: oak-MDT0000: haven't heard from client ae87508c-5df5-4480-8c70-fc7fddf32a06 (at 10.51.6.60@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901f794f2c00, cur 1713584298 expire 1713584148 last 1713584071 [3320068.577590] Lustre: Skipped 1 previous similar message [3320491.745529] LustreError: 35144:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3320491.775379] LustreError: 35144:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 202 previous similar messages [3321006.518111] Lustre: oak-MDT0003: haven't heard from client c5787b23-77e7-4414-972c-b24a1ed52d08 (at 10.51.12.19@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900e887d6c00, cur 1713585236 expire 1713585086 last 1713585009 [3321006.540072] Lustre: Skipped 1 previous similar message [3321092.343194] LustreError: 35227:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107933313504 granted:424654352, total:353487599108 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353487599108 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3321092.373727] LustreError: 35227:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 241 previous similar messages [3321534.486503] Lustre: oak-MDT0000: haven't heard from client c52c661c-490d-4a9e-8905-b26b53b4dae0 (at 10.51.9.3@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903b39eff400, cur 1713585764 expire 1713585614 last 1713585537 [3321534.508295] Lustre: Skipped 1 previous similar message [3321733.734667] LustreError: 35197:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3321733.764346] LustreError: 35197:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 113 previous similar messages [3322274.453777] Lustre: oak-MDT0003: haven't heard from client 0866a9d0-7b8c-4899-8076-22eebb71eed6 (at 10.51.9.47@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903b084b6000, cur 1713586504 expire 1713586354 last 1713586277 [3322274.475657] Lustre: Skipped 1 previous similar message [3322335.444581] LustreError: 36655:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107962212632 granted:486160868, total:353467824684 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353467824684 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3322335.475119] LustreError: 36655:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 284 previous similar messages [3322944.758483] LustreError: 35162:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107944480028 granted:483067692, total:353465179376 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353465179376 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3322944.758486] LustreError: 36677:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3322944.758490] LustreError: 36677:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 111 previous similar messages [3323171.417323] Lustre: oak-MDT0000: haven't heard from client 205bc6ec-81d0-419d-b0ce-8222f21f5758 (at 10.51.12.7@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902078801400, cur 1713587401 expire 1713587251 last 1713587174 [3323171.439209] Lustre: Skipped 1 previous similar message [3323544.835740] LustreError: 35218:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107947382200 granted:483067692, total:353460839204 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353460839204 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3323544.866282] LustreError: 35218:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 77 previous similar messages [3324144.883215] LustreError: 36679:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3324144.913235] LustreError: 36679:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 83 previous similar messages [3324253.546679] Lustre: oak-MDT0003: Client 0185e88c-011d-4925-902d-38a7b371ae19 (at 10.51.6.60@o2ib3) reconnecting [3324253.556941] Lustre: Skipped 153 previous similar messages [3324298.360952] Lustre: oak-MDT0000: haven't heard from client fe464947-148a-4f68-97c0-398d8e00c470 (at 10.51.13.6@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902078805c00, cur 1713588528 expire 1713588378 last 1713588301 [3324298.382834] Lustre: Skipped 1 previous similar message [3324554.362576] Lustre: oak-MDT0000: haven't heard from client 9c194555-2649-4e0b-8bcd-0187bd7bc669 (at 10.51.5.66@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901d5ad74c00, cur 1713588784 expire 1713588634 last 1713588557 [3324554.384610] Lustre: Skipped 1 previous similar message [3324561.353482] Lustre: oak-MDT0003: haven't heard from client 9c194555-2649-4e0b-8bcd-0187bd7bc669 (at 10.51.5.66@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900496073400, cur 1713588791 expire 1713588641 last 1713588564 [3324561.375355] Lustre: Skipped 1 previous similar message [3324630.347326] Lustre: oak-MDT0000: haven't heard from client 2f73f8b5-bd38-48cd-a7b1-e2ce40688e68 (at 10.50.10.11@o2ib2) in 217 seconds. I think it's dead, and I am evicting it. exp ffff901e0f8f7400, cur 1713588860 expire 1713588710 last 1713588643 [3324630.369290] Lustre: Skipped 1 previous similar message [3324637.359660] Lustre: oak-MDT0003: haven't heard from client 2f73f8b5-bd38-48cd-a7b1-e2ce40688e68 (at 10.50.10.11@o2ib2) in 224 seconds. I think it's dead, and I am evicting it. exp ffff901e0f95b000, cur 1713588867 expire 1713588717 last 1713588643 [3324745.174570] LustreError: 35165:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107947382200 granted:483067692, total:353440748448 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353440748448 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3324745.205109] LustreError: 35165:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 81 previous similar messages [3325012.337931] Lustre: oak-MDT0003: haven't heard from client e894eec0-30cf-4054-adb7-9cc3acc0b7ed (at 10.51.4.34@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902c68c13800, cur 1713589242 expire 1713589092 last 1713589015 [3325348.781733] LustreError: 36692:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107933196540 granted:424654352, total:353434098216 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353434098216 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3325348.812272] LustreError: 36692:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 121 previous similar messages [3325448.317452] Lustre: oak-MDT0003: haven't heard from client de1ade7c-b0dc-4710-9785-e21cf665f2cb (at 10.50.7.52@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9011628fdc00, cur 1713589678 expire 1713589528 last 1713589451 [3325448.339323] Lustre: Skipped 1 previous similar message [3325620.299474] Lustre: oak-MDT0000: haven't heard from client 559e4b95-b17d-449f-abcf-fc97555d929b (at 10.51.7.50@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90139bb87400, cur 1713589850 expire 1713589700 last 1713589623 [3325620.321349] Lustre: Skipped 1 previous similar message [3325950.354376] LustreError: 35155:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852951716 granted:310379184, total:353433049640 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353433049640 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3325950.384917] LustreError: 35155:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 85 previous similar messages [3326221.272456] Lustre: oak-MDT0003: haven't heard from client 2aac7bfe-9cf5-4466-9e00-337f35bf0242 (at 10.51.9.30@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900cc4319400, cur 1713590451 expire 1713590301 last 1713590224 [3326221.294329] Lustre: Skipped 3 previous similar messages [3326554.776499] LustreError: 35200:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536797144 granted:559613176, total:353433049640 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353433049640 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3326554.807039] LustreError: 35200:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 80 previous similar messages [3327156.267353] LustreError: 36700:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107933196540 granted:424654352, total:353433049640 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353433049640 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3327156.297891] LustreError: 36700:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 83 previous similar messages [3327759.327765] LustreError: 35152:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536797144 granted:559613176, total:353433049640 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353433049640 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3327759.358305] LustreError: 35152:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 79 previous similar messages [3328067.186558] Lustre: oak-MDT0000: haven't heard from client 68c8085c-a6da-4024-ad64-5e7e7468aa7d (at 10.51.11.15@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903b3cd73400, cur 1713592297 expire 1713592147 last 1713592070 [3328067.208532] Lustre: Skipped 21 previous similar messages [3328263.176973] Lustre: oak-MDT0003: haven't heard from client eb852a29-c6e3-47c8-a0bc-237109878327 (at 10.51.9.71@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903a61399400, cur 1713592493 expire 1713592343 last 1713592266 [3328263.198852] Lustre: Skipped 1 previous similar message [3328382.208624] LustreError: 35220:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107933196540 granted:424654352, total:353433049640 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353433049640 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3328382.239165] LustreError: 35220:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 78 previous similar messages [3328690.157815] Lustre: oak-MDT0000: haven't heard from client c48ab8e8-6d94-4f34-a6fc-603854442269 (at 10.51.7.10@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90143eb87c00, cur 1713592920 expire 1713592770 last 1713592693 [3328690.179696] Lustre: Skipped 1 previous similar message [3328982.647819] LustreError: 35174:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107949061112 granted:483067692, total:353433049640 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353433049640 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3328982.678361] LustreError: 35174:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 78 previous similar messages [3329074.139917] Lustre: oak-MDT0003: haven't heard from client 634fe0c8-c85e-4d20-86d0-562129bcd344 (at 10.51.2.11@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9010608e8800, cur 1713593304 expire 1713593154 last 1713593077 [3329074.161815] Lustre: Skipped 1 previous similar message [3329401.127615] Lustre: oak-MDT0003: haven't heard from client dbe81a20-e5cd-46c4-b8ae-3534af76655e (at 10.51.2.9@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900f13114000, cur 1713593631 expire 1713593481 last 1713593404 [3329401.149400] Lustre: Skipped 3 previous similar messages [3329582.629242] LustreError: 36656:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852951716 granted:310379184, total:353433049640 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353433049640 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3329582.659786] LustreError: 36656:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 73 previous similar messages [3330044.095523] Lustre: oak-MDT0000: haven't heard from client bb34a1f7-6ad5-42b3-9214-05bbedd26365 (at 10.51.6.34@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903b88c23400, cur 1713594274 expire 1713594124 last 1713594047 [3330044.117405] Lustre: Skipped 1 previous similar message [3330187.373124] LustreError: 36686:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107949061112 granted:483067692, total:353433049640 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353433049640 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3330187.403662] LustreError: 36686:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 78 previous similar messages [3330789.152826] LustreError: 35204:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107814850800 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3330789.182498] LustreError: 35204:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 75 previous similar messages [3331404.047553] LustreError: 36654:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852950724 granted:310379184, total:353433049640 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353433049640 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3331404.078142] LustreError: 36654:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 542 previous similar messages [3331736.020483] Lustre: oak-MDT0000: haven't heard from client 957e33dc-67bd-4dd8-a650-c3102a4059d4 (at 10.51.3.19@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff904028d63800, cur 1713595966 expire 1713595816 last 1713595739 [3331736.042358] Lustre: Skipped 3 previous similar messages [3332007.269622] LustreError: 36658:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3332007.299304] LustreError: 36658:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 109 previous similar messages [3332497.982240] Lustre: oak-MDT0000: haven't heard from client 4c723a0b-6c57-4eb6-be62-98fc3cdea8fd (at 10.51.5.33@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903c54cd1000, cur 1713596728 expire 1713596578 last 1713596501 [3332498.004113] Lustre: Skipped 3 previous similar messages [3332616.549616] LustreError: 35182:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536795932 granted:559613176, total:353433049640 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353433049640 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3332616.580162] LustreError: 35182:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 79 previous similar messages [3333216.597904] LustreError: 35200:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536795932 granted:559613176, total:353433049640 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353433049640 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3333216.628445] LustreError: 35200:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 80 previous similar messages [3333821.489587] LustreError: 36681:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107933061284 granted:424654352, total:353433049640 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353433049640 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3333821.520121] LustreError: 36681:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 78 previous similar messages [3334240.907712] Lustre: oak-MDT0003: haven't heard from client 34cb3593-070e-4d8d-8ec4-e5d320364f1e (at 10.51.9.36@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90119db98800, cur 1713598471 expire 1713598321 last 1713598244 [3334240.929594] Lustre: Skipped 1 previous similar message [3334422.537175] LustreError: 35143:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536795932 granted:559613176, total:353433049640 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353433049640 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3334422.567710] LustreError: 35143:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 81 previous similar messages [3334438.893442] Lustre: oak-MDT0000: haven't heard from client 3661e7a4-2724-4688-acd8-a53da343d4bb (at 10.51.1.19@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9020fa3bd800, cur 1713598669 expire 1713598519 last 1713598442 [3334438.915656] Lustre: Skipped 1 previous similar message [3334745.878349] Lustre: oak-MDT0000: haven't heard from client 4fe368c5-f79a-488c-bc59-30b4869bf3ec (at 10.51.3.70@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901bde5bc000, cur 1713598976 expire 1713598826 last 1713598749 [3334745.900224] Lustre: Skipped 1 previous similar message [3334900.871152] Lustre: oak-MDT0003: haven't heard from client 53711778-e8be-42df-afb4-30f44f6beb34 (at 10.51.1.32@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900822937400, cur 1713599131 expire 1713598981 last 1713598904 [3334900.893031] Lustre: Skipped 1 previous similar message [3335023.754546] LustreError: 36669:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3335023.784226] LustreError: 36669:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 82 previous similar messages [3335623.765695] LustreError: 35149:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3335623.795374] LustreError: 35149:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 86 previous similar messages [3336235.228551] LustreError: 36660:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3336235.258401] LustreError: 36660:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 86 previous similar messages [3336372.804219] Lustre: oak-MDT0003: haven't heard from client 361c9abe-fa7b-4cac-b9e5-21219e17c79b (at 10.51.5.41@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9029abf90000, cur 1713600603 expire 1713600453 last 1713600376 [3336372.826091] Lustre: Skipped 1 previous similar message [3336805.782198] Lustre: oak-MDT0000: haven't heard from client 59c3c5c8-5349-4e33-b076-35c22d1c7985 (at 10.51.1.23@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902486069000, cur 1713601036 expire 1713600886 last 1713600809 [3336805.804078] Lustre: Skipped 3 previous similar messages [3336842.975490] LustreError: 36650:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536795932 granted:559613176, total:353433049640 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353433049640 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3336843.006028] LustreError: 36650:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 83 previous similar messages [3337442.988659] LustreError: 31510:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536795932 granted:559613176, total:353433049640 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353433049640 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3337443.019204] LustreError: 31510:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 88 previous similar messages [3338044.925714] LustreError: 35201:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3338044.955395] LustreError: 35201:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 94 previous similar messages [3338306.716228] Lustre: oak-MDT0003: haven't heard from client f7cf232d-3b7e-42d3-88c4-bd4081d6f24a (at 10.51.1.43@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901086e95400, cur 1713602537 expire 1713602387 last 1713602310 [3338306.738105] Lustre: Skipped 1 previous similar message [3338644.937027] LustreError: 35155:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3338644.966709] LustreError: 35155:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 86 previous similar messages [3338882.686967] Lustre: oak-MDT0000: haven't heard from client 7b2a8678-e83f-4f29-b852-46140c580432 (at 10.51.8.23@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9008ce1b1800, cur 1713603113 expire 1713602963 last 1713602886 [3338882.708843] Lustre: Skipped 1 previous similar message [3339244.949333] LustreError: 35183:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3339244.979009] LustreError: 35183:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 85 previous similar messages [3339635.651471] Lustre: oak-MDT0003: haven't heard from client d026cbd7-d148-4b88-bc0f-0eae04b290ce (at 10.51.12.2@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900e68210000, cur 1713603866 expire 1713603716 last 1713603639 [3339635.673343] Lustre: Skipped 1 previous similar message [3339845.330741] LustreError: 35203:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3339845.360419] LustreError: 35203:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 82 previous similar messages [3340448.659537] LustreError: 35174:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107933061280 granted:424654352, total:353433049640 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353433049640 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3340448.690259] LustreError: 35174:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 84 previous similar messages [3341019.589234] Lustre: oak-MDT0003: haven't heard from client 6e29175b-8091-4c7a-8a05-a4fcf6ad5155 (at 10.51.3.41@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9040fd23f800, cur 1713605250 expire 1713605100 last 1713605023 [3341019.611105] Lustre: Skipped 1 previous similar message [3341048.672054] LustreError: 35175:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107933061280 granted:424654352, total:353433049640 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353433049640 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3341048.702590] LustreError: 35175:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 85 previous similar messages [3341595.563946] Lustre: oak-MDT0000: haven't heard from client 52ec5f00-9ccc-4ef6-97b5-69aa11a4cdb8 (at 10.51.17.17@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903e64b6ec00, cur 1713605826 expire 1713605676 last 1713605599 [3341595.585908] Lustre: Skipped 1 previous similar message [3341648.684511] LustreError: 36685:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107933061280 granted:424654352, total:353433049640 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353433049640 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3341648.715050] LustreError: 36685:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 84 previous similar messages [3342253.258173] LustreError: 36667:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536795900 granted:559613176, total:353433049640 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353433049640 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3342253.288717] LustreError: 36667:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 80 previous similar messages [3342699.512458] Lustre: oak-MDT0000: haven't heard from client 3ccbc23d-576b-4b6c-8e7b-739c1328a7ea (at 10.51.5.59@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903bb76f3400, cur 1713606930 expire 1713606780 last 1713606703 [3342699.534333] Lustre: Skipped 1 previous similar message [3342853.272684] LustreError: 36653:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536795900 granted:559613176, total:353433049640 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353433049640 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3342853.303260] LustreError: 36653:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 84 previous similar messages [3343455.156101] LustreError: 36656:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3343455.185954] LustreError: 36656:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 85 previous similar messages [3344055.201495] LustreError: 35203:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3344055.231341] LustreError: 35203:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 84 previous similar messages [3344655.549922] LustreError: 35205:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852950284 granted:310379184, total:353433049640 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353433049640 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3344655.580463] LustreError: 35205:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 85 previous similar messages [3345099.407394] Lustre: oak-MDT0000: haven't heard from client 27a78250-e830-4699-bd5b-11d5e001dd7b (at 10.51.15.7@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9032bf337c00, cur 1713609330 expire 1713609180 last 1713609103 [3345099.429266] Lustre: Skipped 3 previous similar messages [3345106.400668] Lustre: oak-MDT0003: haven't heard from client 27a78250-e830-4699-bd5b-11d5e001dd7b (at 10.51.15.7@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903856b1d800, cur 1713609337 expire 1713609187 last 1713609110 [3345262.240712] LustreError: 35161:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107933061280 granted:424654352, total:353433049640 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353433049640 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3345262.271253] LustreError: 35161:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 85 previous similar messages [3345862.292141] LustreError: 36691:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107933061280 granted:424654352, total:353433049640 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353433049640 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3345862.322673] LustreError: 36691:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 85 previous similar messages [3346225.351559] Lustre: oak-MDT0000: haven't heard from client 6cfe3fcb-d467-4449-8bfd-b6514c60b37c (at 10.51.9.65@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9032987c6c00, cur 1713610456 expire 1713610306 last 1713610229 [3346227.351585] Lustre: oak-MDT0003: haven't heard from client 6cfe3fcb-d467-4449-8bfd-b6514c60b37c (at 10.51.9.65@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902bee068800, cur 1713610458 expire 1713610308 last 1713610231 [3346462.879076] LustreError: 35196:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3346462.908931] LustreError: 35196:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 82 previous similar messages [3347062.924546] LustreError: 35177:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3347062.954392] LustreError: 35177:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 86 previous similar messages [3347662.937109] LustreError: 35203:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3347662.966957] LustreError: 35203:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 83 previous similar messages [3348262.982632] LustreError: 35204:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3348263.012491] LustreError: 35204:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 85 previous similar messages [3348864.424032] LustreError: 35178:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536795900 granted:559613176, total:353433049640 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353433049640 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3348864.454573] LustreError: 35178:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 81 previous similar messages [3348890.232156] Lustre: oak-MDT0000: haven't heard from client f2adb14a-369f-41b7-a157-ca36c5ea8835 (at 10.50.3.71@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903c259b7000, cur 1713613121 expire 1713612971 last 1713612894 [3349103.218492] Lustre: oak-MDT0000: haven't heard from client 40bd503f-7a74-4e29-a673-b4ceb03fffef (at 10.50.13.8@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900496181000, cur 1713613334 expire 1713613184 last 1713613107 [3349103.240364] Lustre: Skipped 1 previous similar message [3349464.436691] LustreError: 36657:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536795900 granted:559613176, total:353433049640 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353433049640 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3349464.467227] LustreError: 36657:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 86 previous similar messages [3350071.468879] LustreError: 36659:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3350071.498721] LustreError: 36659:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 83 previous similar messages [3350675.060364] LustreError: 35148:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3350675.090216] LustreError: 35148:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 80 previous similar messages [3351275.071907] LustreError: 35149:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3351275.101757] LustreError: 35149:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 81 previous similar messages [3351344.134722] Lustre: oak-MDT0003: haven't heard from client cc9b5868-e2e9-4b85-a9ce-42b14dcbbe60 (at 10.51.5.49@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90241d6c8400, cur 1713615575 expire 1713615425 last 1713615348 [3351344.156611] Lustre: Skipped 1 previous similar message [3351355.115817] Lustre: oak-MDT0000: haven't heard from client cc9b5868-e2e9-4b85-a9ce-42b14dcbbe60 (at 10.51.5.49@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90299957d800, cur 1713615586 expire 1713615436 last 1713615359 [3351770.096955] Lustre: oak-MDT0000: haven't heard from client db36df4b-e59e-44a7-840d-b21476404ca2 (at 10.51.5.39@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9040fdf15000, cur 1713616001 expire 1713615851 last 1713615774 [3351875.743937] LustreError: 36681:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107949059416 granted:483067692, total:353433049640 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353433049640 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3351875.774474] LustreError: 36681:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 86 previous similar messages [3352475.791341] LustreError: 35222:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107949059416 granted:483067692, total:353433049640 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353433049640 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3352475.821875] LustreError: 35222:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 81 previous similar messages [3353075.838958] LustreError: 35218:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3353075.838959] LustreError: 36698:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107949059416 granted:483067692, total:353433049640 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353433049640 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3353075.838961] LustreError: 36698:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 86 previous similar messages [3353675.917484] LustreError: 35167:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107949059416 granted:483067692, total:353433049640 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353433049640 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3353675.948023] LustreError: 35167:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 85 previous similar messages [3354275.963902] LustreError: 35172:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107949059416 granted:483067692, total:353433049640 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353433049640 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3354275.963903] LustreError: 36698:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3354275.963906] LustreError: 36698:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 87 previous similar messages [3354876.627777] LustreError: 35178:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3354876.657452] LustreError: 35178:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 88 previous similar messages [3355485.416920] LustreError: 36671:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107965009044 granted:486160868, total:353433049640 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353433049640 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3355485.447460] LustreError: 36671:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 87 previous similar messages [3356088.164324] LustreError: 36664:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536795900 granted:559613176, total:353433049640 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353433049640 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3356088.194868] LustreError: 36664:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 86 previous similar messages [3356688.211605] LustreError: 35179:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536795900 granted:559613176, total:353433049640 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353433049640 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3356688.242148] LustreError: 35179:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 84 previous similar messages [3357288.259263] LustreError: 35202:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536795900 granted:559613176, total:353433049640 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353433049640 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3357288.289806] LustreError: 35202:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 85 previous similar messages [3357888.308020] LustreError: 31509:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536795900 granted:559613176, total:353433049640 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353433049640 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3357888.338566] LustreError: 31509:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 86 previous similar messages [3358488.356304] LustreError: 36672:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536795900 granted:559613176, total:353433049640 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353433049640 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3358488.386840] LustreError: 36672:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 85 previous similar messages [3359089.219742] LustreError: 35208:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852950284 granted:310379184, total:353433049640 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353433049640 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3359089.250281] LustreError: 35208:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 89 previous similar messages [3359585.738997] Lustre: oak-MDT0000: haven't heard from client 5dbfff71-195f-4628-a4a3-8caa9fcd2167 (at 10.51.11.15@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903d26bda000, cur 1713623817 expire 1713623667 last 1713623590 [3359585.760965] Lustre: Skipped 1 previous similar message [3359661.735502] Lustre: oak-MDT0003: haven't heard from client ba066270-bfd2-493a-b32c-cdfa35cd00f5 (at 10.50.3.63@o2ib2) in 203 seconds. I think it's dead, and I am evicting it. exp ffff9022a2dea000, cur 1713623893 expire 1713623743 last 1713623690 [3359661.757379] Lustre: Skipped 1 previous similar message [3359689.267382] LustreError: 35151:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852950284 granted:310379184, total:353433049640 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353433049640 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3359689.297929] LustreError: 35151:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 88 previous similar messages [3359910.724736] Lustre: oak-MDT0003: haven't heard from client 07befd86-f11b-42ac-961c-5ef5b7b585d6 (at 10.51.11.5@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90398d73d000, cur 1713624142 expire 1713623992 last 1713623915 [3359910.746609] Lustre: Skipped 27 previous similar messages [3360075.715662] Lustre: oak-MDT0003: haven't heard from client 08db689c-7bb2-4eb7-8b91-444ae4a777fa (at 10.51.11.6@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9019e50c7800, cur 1713624307 expire 1713624157 last 1713624080 [3360075.737539] Lustre: Skipped 1 previous similar message [3360244.710523] Lustre: oak-MDT0003: haven't heard from client 1dc27073-822a-45ed-939f-c32b7778ee6f (at 10.51.8.58@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902da2b43800, cur 1713624476 expire 1713624326 last 1713624249 [3360244.732400] Lustre: Skipped 3 previous similar messages [3360290.790813] LustreError: 35182:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3360290.820490] LustreError: 35182:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 79 previous similar messages [3360414.700162] Lustre: oak-MDT0000: haven't heard from client b4954147-4df3-46bb-ac47-6b1dbb06de8f (at 10.51.12.18@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903bf676c800, cur 1713624646 expire 1713624496 last 1713624419 [3360414.722121] Lustre: Skipped 1 previous similar message [3360890.837379] LustreError: 35182:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3360890.867054] LustreError: 35182:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 85 previous similar messages [3361490.899903] LustreError: 35154:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3361490.929583] LustreError: 35154:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 83 previous similar messages [3361835.636986] Lustre: oak-MDT0000: haven't heard from client 627a4408-b91e-41ae-8b20-9b97ed3297ee (at 10.51.9.63@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903a6a9d1400, cur 1713626067 expire 1713625917 last 1713625840 [3361835.658861] Lustre: Skipped 1 previous similar message [3362090.945577] LustreError: 35200:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3362090.975255] LustreError: 35200:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 88 previous similar messages [3362711.888727] LustreError: 36682:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107933061280 granted:424654352, total:353433049640 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353433049640 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3362711.919266] LustreError: 36682:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 81 previous similar messages [3363311.936415] LustreError: 35220:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107933061280 granted:424654352, total:353433049640 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353433049640 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3363311.966948] LustreError: 35220:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 82 previous similar messages [3363840.544631] Lustre: oak-MDT0003: haven't heard from client e2fa155a-c269-4fbb-b455-a348f7ff2b2c (at 10.50.15.9@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90335aaee000, cur 1713628072 expire 1713627922 last 1713627845 [3363840.566503] Lustre: Skipped 3 previous similar messages [3363924.300900] LustreError: 35206:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107814850800 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3363924.330574] LustreError: 35206:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 86 previous similar messages [3364531.068182] LustreError: 35143:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852950284 granted:310379184, total:353433049640 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353433049640 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3364531.098726] LustreError: 35143:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 78 previous similar messages [3365009.491089] Lustre: oak-MDT0003: haven't heard from client ccf3348a-0d04-44f0-91d5-e7be6a0df62b (at 10.51.4.43@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901ec4af7c00, cur 1713629241 expire 1713629091 last 1713629014 [3365009.512966] Lustre: Skipped 1 previous similar message [3365132.682674] LustreError: 36664:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3365132.712526] LustreError: 36664:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 86 previous similar messages [3365740.563907] LustreError: 36655:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852950284 granted:310379184, total:353433049640 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353433049640 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3365740.594453] LustreError: 36655:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 89 previous similar messages [3365855.451650] Lustre: oak-MDT0000: haven't heard from client 276d3ab9-0477-40c9-aa30-792532b1e5d6 (at 10.51.13.20@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900138121000, cur 1713630087 expire 1713629937 last 1713629860 [3365855.473614] Lustre: Skipped 1 previous similar message [3366073.441650] Lustre: oak-MDT0000: haven't heard from client b4d665f9-b754-42cb-a310-f981efc483aa (at 10.51.7.49@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9025f0893000, cur 1713630305 expire 1713630155 last 1713630078 [3366073.463532] Lustre: Skipped 1 previous similar message [3366085.443143] Lustre: oak-MDT0003: haven't heard from client b4d665f9-b754-42cb-a310-f981efc483aa (at 10.51.7.49@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9037e02a9000, cur 1713630317 expire 1713630167 last 1713630090 [3366343.043162] LustreError: 36656:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107965009044 granted:486160868, total:353433049640 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353433049640 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3366343.043164] LustreError: 36672:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107814850800 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3366343.043167] LustreError: 36672:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 83 previous similar messages [3366392.432071] Lustre: oak-MDT0003: haven't heard from client 1ebf2d62-6fe8-4985-8119-ad7e09c3c0f4 (at 10.51.16.1@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901a42dbe400, cur 1713630624 expire 1713630474 last 1713630397 [3366409.426790] Lustre: oak-MDT0000: haven't heard from client 1ebf2d62-6fe8-4985-8119-ad7e09c3c0f4 (at 10.51.16.1@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900b3e80d800, cur 1713630641 expire 1713630491 last 1713630414 [3366953.449241] LustreError: 35197:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852950284 granted:310379184, total:353433049640 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353433049640 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3366953.479786] LustreError: 35197:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 85 previous similar messages [3367553.496836] LustreError: 36654:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852950284 granted:310379184, total:353433049640 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353433049640 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3367553.527383] LustreError: 36654:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 82 previous similar messages [3368162.018982] LustreError: 35148:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536795900 granted:559613176, total:353433049640 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353433049640 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3368162.049521] LustreError: 35148:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 83 previous similar messages [3368235.342553] Lustre: oak-MDT0000: haven't heard from client 5b417abf-9133-49fe-9c10-069d07bbebb0 (at 10.51.10.45@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9017c2677800, cur 1713632467 expire 1713632317 last 1713632240 [3368764.734005] LustreError: 35168:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107933061280 granted:424654352, total:353433049640 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353433049640 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3368764.764538] LustreError: 35168:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 83 previous similar messages [3369198.299724] Lustre: oak-MDT0000: haven't heard from client 5b2c2f0e-b1af-420b-8183-2d38f01776c0 (at 10.51.9.35@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9040cd6cd800, cur 1713633430 expire 1713633280 last 1713633203 [3369198.321607] Lustre: Skipped 1 previous similar message [3369379.354132] LustreError: 35154:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107814850800 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3369379.383812] LustreError: 35154:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 79 previous similar messages [3369916.266033] Lustre: oak-MDT0000: haven't heard from client 7075c8e2-3da7-4598-a3bc-d5bc3bbaded4 (at 10.51.8.66@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903d3ebbb800, cur 1713634148 expire 1713633998 last 1713633921 [3369916.287907] Lustre: Skipped 1 previous similar message [3369993.719577] LustreError: 35184:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107933061280 granted:424654352, total:353433049640 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353433049640 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3369993.750115] LustreError: 35184:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 84 previous similar messages [3370597.907646] LustreError: 36655:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536795900 granted:559613176, total:353433049640 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353433049640 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3370597.938189] LustreError: 36655:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 86 previous similar messages [3371048.213915] Lustre: oak-MDT0003: haven't heard from client 07cee168-abd1-4b31-9558-27a9250f496a (at 10.51.13.13@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903fe4ef4400, cur 1713635280 expire 1713635130 last 1713635053 [3371048.235879] Lustre: Skipped 1 previous similar message [3371059.217860] Lustre: oak-MDT0000: haven't heard from client 07cee168-abd1-4b31-9558-27a9250f496a (at 10.51.13.13@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903535789800, cur 1713635291 expire 1713635141 last 1713635064 [3371205.820769] LustreError: 35206:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3371205.850613] LustreError: 35206:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 86 previous similar messages [3371810.911815] LustreError: 35169:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107933061280 granted:424654352, total:353433049640 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353433049640 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3371810.942358] LustreError: 35169:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 81 previous similar messages [3372410.959493] LustreError: 35174:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107933061280 granted:424654352, total:353433049640 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353433049640 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3372410.990031] LustreError: 35174:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 83 previous similar messages [3373011.007967] LustreError: 36695:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107934666272 granted:424654352, total:353432001064 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353432001064 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3373011.038597] LustreError: 36695:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 101 previous similar messages [3373044.122995] Lustre: oak-MDT0000: haven't heard from client 85db2dd7-d58f-4cdb-bea2-3885dc58f43b (at 10.51.2.66@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9026dd4d6800, cur 1713637276 expire 1713637126 last 1713637049 [3373250.121193] Lustre: oak-MDT0000: haven't heard from client 89af72b7-74c1-4bf3-9d04-09a4cc96fef5 (at 10.51.2.38@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903d2a2c8c00, cur 1713637482 expire 1713637332 last 1713637255 [3373250.143069] Lustre: Skipped 3 previous similar messages [3373265.112916] Lustre: oak-MDT0003: haven't heard from client 89af72b7-74c1-4bf3-9d04-09a4cc96fef5 (at 10.51.2.38@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903940d27400, cur 1713637497 expire 1713637347 last 1713637270 [3373501.113647] Lustre: oak-MDT0000: haven't heard from client 834f3ebe-f34a-4023-aaa3-ca4b05558655 (at 10.51.2.41@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900f13110800, cur 1713637733 expire 1713637583 last 1713637506 [3373611.055143] LustreError: 36688:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107933074884 granted:424654352, total:353432001064 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353432001064 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3373611.085681] LustreError: 36688:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 471 previous similar messages [3374211.322669] LustreError: 36666:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3374211.352355] LustreError: 36666:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 89 previous similar messages [3374400.064171] Lustre: oak-MDT0000: haven't heard from client 33f8ed0f-09d6-4f29-a839-83150632294e (at 10.51.1.58@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90166c49cc00, cur 1713638632 expire 1713638482 last 1713638405 [3374400.086050] Lustre: Skipped 1 previous similar message [3374693.047827] Lustre: oak-MDT0000: haven't heard from client ba6c5ad1-7596-472c-babe-c15b1be10ea2 (at 10.51.12.3@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900710944c00, cur 1713638925 expire 1713638775 last 1713638698 [3374693.069705] Lustre: Skipped 1 previous similar message [3374703.051742] Lustre: oak-MDT0003: haven't heard from client ba6c5ad1-7596-472c-babe-c15b1be10ea2 (at 10.51.12.3@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903d2a335800, cur 1713638935 expire 1713638785 last 1713638708 [3374811.338999] LustreError: 35197:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3374811.368866] LustreError: 35197:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 125 previous similar messages [3374960.034206] Lustre: oak-MDT0000: haven't heard from client d0f58696-2fea-4489-b6c5-e1c1cc324e9b (at 10.51.2.53@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9005871a9000, cur 1713639192 expire 1713639042 last 1713638965 [3375415.903122] LustreError: 36668:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3375415.932969] LustreError: 36668:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 85 previous similar messages [3375482.012555] Lustre: oak-MDT0003: haven't heard from client bd2804b1-210c-4dd6-bcf0-082e97277dd1 (at 10.50.10.46@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902c25ca3000, cur 1713639714 expire 1713639564 last 1713639487 [3375482.034517] Lustre: Skipped 1 previous similar message [3376021.937770] LustreError: 36697:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107934665032 granted:424654352, total:353432001064 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353432001064 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3376021.968309] LustreError: 36697:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 84 previous similar messages [3376413.967053] Lustre: oak-MDT0000: haven't heard from client 49625bbe-79d6-4292-aa11-ed9a5cd6ce89 (at 10.51.8.46@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9030cf7cac00, cur 1713640646 expire 1713640496 last 1713640419 [3376413.988933] Lustre: Skipped 1 previous similar message [3376621.985169] LustreError: 36691:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107934665032 granted:424654352, total:353432001064 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353432001064 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3376622.015709] LustreError: 36691:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 85 previous similar messages [3377233.922101] LustreError: 35185:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107949056292 granted:483067692, total:353432001064 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353432001064 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3377233.922102] LustreError: 36699:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3377233.922105] LustreError: 36699:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 84 previous similar messages [3377833.998412] LustreError: 36694:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3377833.998414] LustreError: 36678:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107949056292 granted:483067692, total:353432001064 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353432001064 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3377833.998419] LustreError: 36678:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 82 previous similar messages [3378363.881671] Lustre: oak-MDT0003: haven't heard from client 92cca6d8-2c50-4dd4-9c80-3094223f4855 (at 10.51.4.33@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903cec785000, cur 1713642596 expire 1713642446 last 1713642369 [3378363.903549] Lustre: Skipped 1 previous similar message [3378434.091750] LustreError: 36675:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107949056292 granted:483067692, total:353432001064 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353432001064 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3378434.122287] LustreError: 36675:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 88 previous similar messages [3379034.138092] LustreError: 35184:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3379034.138093] LustreError: 31507:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107949056292 granted:483067692, total:353432001064 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353432001064 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3379034.138097] LustreError: 31507:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 84 previous similar messages [3379658.050541] LustreError: 35145:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3379658.080396] LustreError: 35145:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 82 previous similar messages [3380258.061671] LustreError: 35146:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3380258.091514] LustreError: 35146:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 86 previous similar messages [3380858.073860] LustreError: 35208:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3380858.103710] LustreError: 35208:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 87 previous similar messages [3381347.739055] Lustre: oak-MDT0003: haven't heard from client 13fa2ef1-2321-4667-9fbf-c08eba2bec65 (at 10.50.8.18@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9033918c6000, cur 1713645580 expire 1713645430 last 1713645353 [3381347.760929] Lustre: Skipped 1 previous similar message [3381472.189226] LustreError: 35181:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536794032 granted:559613176, total:353432001064 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353432001064 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3381472.219764] LustreError: 35181:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 80 previous similar messages [3382072.238262] LustreError: 35151:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536794032 granted:559613176, total:353432001064 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353432001064 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3382072.268800] LustreError: 35151:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 84 previous similar messages [3382686.343448] LustreError: 35201:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107965008752 granted:486160868, total:353432001064 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353432001064 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3382686.373986] LustreError: 35201:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 88 previous similar messages [3382956.667366] Lustre: oak-MDT0000: haven't heard from client b2c5295b-f259-4e88-9609-236c58159275 (at 10.50.6.32@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902f6db06c00, cur 1713647189 expire 1713647039 last 1713646962 [3382956.689244] Lustre: Skipped 1 previous similar message [3382966.670644] Lustre: oak-MDT0003: haven't heard from client b2c5295b-f259-4e88-9609-236c58159275 (at 10.50.6.32@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903b9c383400, cur 1713647199 expire 1713647049 last 1713646972 [3383286.390613] LustreError: 31510:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107814850800 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3383286.420287] LustreError: 31510:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 86 previous similar messages [3383434.641527] Lustre: oak-MDT0000: haven't heard from client 943798a2-93ce-426c-abfe-e8ebcc4bfc98 (at 10.51.2.27@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901190e12400, cur 1713647667 expire 1713647517 last 1713647440 [3383886.486118] LustreError: 35179:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3383886.515966] LustreError: 35179:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 84 previous similar messages [3384486.496902] LustreError: 35176:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3384486.526948] LustreError: 35176:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 86 previous similar messages [3385086.541361] LustreError: 35157:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3385086.571215] LustreError: 35157:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 83 previous similar messages [3385692.997419] LustreError: 35204:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3385693.027594] LustreError: 35204:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 87 previous similar messages [3386293.041853] LustreError: 35204:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3386293.071738] LustreError: 35204:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 86 previous similar messages [3386893.052255] LustreError: 36656:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3386893.082102] LustreError: 36656:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 82 previous similar messages [3387493.096634] LustreError: 35207:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3387493.126478] LustreError: 35207:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 84 previous similar messages [3388093.142537] LustreError: 36662:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3388093.172382] LustreError: 36662:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 85 previous similar messages [3388693.261652] LustreError: 35176:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3388693.291504] LustreError: 35176:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 81 previous similar messages [3389293.475905] LustreError: 35208:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536794032 granted:559613176, total:353432001064 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353432001064 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3389293.506449] LustreError: 35208:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 88 previous similar messages [3389893.522168] LustreError: 35196:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536794032 granted:559613176, total:353432001064 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353432001064 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3389893.552707] LustreError: 35196:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 88 previous similar messages [3390520.526152] LustreError: 35156:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3390520.555997] LustreError: 35156:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 80 previous similar messages [3391127.412884] LustreError: 35181:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107965008752 granted:486160868, total:353432001064 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353432001064 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3391127.443441] LustreError: 35181:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 85 previous similar messages [3391167.276062] Lustre: oak-MDT0003: haven't heard from client dd5cd140-4cc5-4dda-8575-d7f49ce7fc63 (at 10.51.16.23@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9020600fb400, cur 1713655400 expire 1713655250 last 1713655173 [3391167.298021] Lustre: Skipped 5 previous similar messages [3391728.429806] LustreError: 35151:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107814850800 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3391728.429807] LustreError: 35146:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107965008752 granted:486160868, total:353432001064 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353432001064 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3391728.429810] LustreError: 35146:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 80 previous similar messages [3391912.238956] Lustre: oak-MDT0000: haven't heard from client 4c812e54-6c93-4a27-8d56-6264107948d6 (at 10.50.9.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901a42da3800, cur 1713656145 expire 1713655995 last 1713655918 [3391912.260745] Lustre: Skipped 1 previous similar message [3392328.471289] LustreError: 36661:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107965008752 granted:486160868, total:353432001064 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353432001064 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3392328.501835] LustreError: 36661:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 85 previous similar messages [3392928.483131] LustreError: 35203:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107965008752 granted:486160868, total:353432001064 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353432001064 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3392928.513671] LustreError: 35203:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 87 previous similar messages [3393552.468470] LustreError: 36661:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852945552 granted:310379184, total:353432001064 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353432001064 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3393552.499011] LustreError: 36661:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 84 previous similar messages [3393856.145378] Lustre: oak-MDT0000: haven't heard from client 59815872-d1f6-445b-92c0-e3c982513b13 (at 10.50.9.8@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90072e8af000, cur 1713658089 expire 1713657939 last 1713657862 [3393856.167168] Lustre: Skipped 15 previous similar messages [3394066.134173] Lustre: oak-MDT0000: haven't heard from client 84283cd6-ec2a-4e62-a8a0-56e7c90c8a6b (at 10.51.2.7@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900ea8f56400, cur 1713658299 expire 1713658149 last 1713658072 [3394066.155962] Lustre: Skipped 1 previous similar message [3394152.480884] LustreError: 36667:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852945552 granted:310379184, total:353432001064 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353432001064 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3394152.511436] LustreError: 36667:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 78 previous similar messages [3394755.466851] LustreError: 35220:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107949056292 granted:483067692, total:353432001064 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353432001064 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3394755.497397] LustreError: 35220:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 88 previous similar messages [3395355.477875] LustreError: 35225:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107949056292 granted:483067692, total:353432001064 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353432001064 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3395355.477876] LustreError: 36676:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3395355.477879] LustreError: 36676:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 88 previous similar messages [3395585.061417] Lustre: oak-MDT0000: haven't heard from client 88d9728d-70cf-447f-abde-28d9de1bd5b1 (at 10.50.12.4@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90390d1edc00, cur 1713659818 expire 1713659668 last 1713659591 [3395585.083292] Lustre: Skipped 3 previous similar messages [3395968.573142] LustreError: 36690:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107934665032 granted:424654352, total:353432001064 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353432001064 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3395968.603680] LustreError: 36690:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 88 previous similar messages [3396568.619018] LustreError: 35164:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107934665032 granted:424654352, total:353432001064 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353432001064 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3396568.649568] LustreError: 35164:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 82 previous similar messages [3396677.007924] Lustre: oak-MDT0000: haven't heard from client 8b72caa6-2a73-4dd6-93de-61d95b105bd4 (at 10.51.4.72@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902c68c16800, cur 1713660910 expire 1713660760 last 1713660683 [3396677.029795] Lustre: Skipped 1 previous similar message [3397168.665660] LustreError: 35221:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107934665032 granted:424654352, total:353432001064 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353432001064 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3397168.696198] LustreError: 35221:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 87 previous similar messages [3397780.669261] LustreError: 35178:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852945552 granted:310379184, total:353432001064 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353432001064 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3397780.699805] LustreError: 35178:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 84 previous similar messages [3397968.947536] Lustre: oak-MDT0000: haven't heard from client 6266540b-3d8c-4db6-9358-a58149f2e9b5 (at 10.51.1.64@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901bea885800, cur 1713662202 expire 1713662052 last 1713661975 [3397968.969422] Lustre: Skipped 1 previous similar message [3398295.932065] Lustre: oak-MDT0000: haven't heard from client d4f34d14-381a-4036-b552-82ac9a7f527f (at 10.51.2.70@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9022c4652800, cur 1713662529 expire 1713662379 last 1713662302 [3398295.953935] Lustre: Skipped 5 previous similar messages [3398312.931238] Lustre: oak-MDT0003: haven't heard from client d4f34d14-381a-4036-b552-82ac9a7f527f (at 10.51.2.70@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903aa07ba800, cur 1713662546 expire 1713662396 last 1713662319 [3398371.928428] Lustre: oak-MDT0000: haven't heard from client 83f25a04-9a53-4e6a-be78-8bf2bcec3f76 (at 10.51.8.46@o2ib3) in 199 seconds. I think it's dead, and I am evicting it. exp ffff903b08621000, cur 1713662605 expire 1713662455 last 1713662406 [3398380.681061] LustreError: 35157:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852945552 granted:310379184, total:353432001064 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353432001064 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3398380.711599] LustreError: 35157:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 85 previous similar messages [3398388.927966] Lustre: oak-MDT0003: haven't heard from client 83f25a04-9a53-4e6a-be78-8bf2bcec3f76 (at 10.51.8.46@o2ib3) in 216 seconds. I think it's dead, and I am evicting it. exp ffff903adff11c00, cur 1713662622 expire 1713662472 last 1713662406 [3398447.925042] Lustre: oak-MDT0000: haven't heard from client eea5b64a-3ef6-4029-9aec-580161956bbe (at 10.51.5.63@o2ib3) in 195 seconds. I think it's dead, and I am evicting it. exp ffff902991ecf800, cur 1713662681 expire 1713662531 last 1713662486 [3398464.924546] Lustre: oak-MDT0003: haven't heard from client eea5b64a-3ef6-4029-9aec-580161956bbe (at 10.51.5.63@o2ib3) in 207 seconds. I think it's dead, and I am evicting it. exp ffff903fe4ef2400, cur 1713662698 expire 1713662548 last 1713662491 [3398983.870762] LustreError: 35182:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107965008752 granted:486160868, total:353432001064 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353432001064 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3398983.901300] LustreError: 35182:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 86 previous similar messages [3399585.596988] LustreError: 35185:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3399585.627006] LustreError: 35185:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 87 previous similar messages [3399647.872663] Lustre: oak-MDT0000: haven't heard from client a24c94d9-9cc3-4934-9335-d19ba170ab6b (at 10.51.4.5@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9010e8539c00, cur 1713663881 expire 1713663731 last 1713663654 [3400156.843834] Lustre: oak-MDT0003: haven't heard from client 272b0149-b671-48e8-b31d-d63bcd979659 (at 10.51.2.7@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901f79582c00, cur 1713664390 expire 1713664240 last 1713664163 [3400156.865619] Lustre: Skipped 7 previous similar messages [3400187.509782] LustreError: 36694:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107934665032 granted:424654352, total:353432001064 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353432001064 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3400187.540319] LustreError: 36694:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 83 previous similar messages [3400787.557919] LustreError: 35226:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107934665032 granted:424654352, total:353432001064 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353432001064 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3400787.588466] LustreError: 35226:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 86 previous similar messages [3401387.606949] LustreError: 35166:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107934665032 granted:424654352, total:353432001064 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353432001064 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3401387.637493] LustreError: 35166:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 85 previous similar messages [3401867.764003] Lustre: oak-MDT0003: haven't heard from client 42c21add-95bc-41f2-ae91-6047437816e5 (at 10.51.8.48@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903c80ea4800, cur 1713666101 expire 1713665951 last 1713665874 [3401867.785919] Lustre: Skipped 3 previous similar messages [3401999.997836] LustreError: 35179:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852945552 granted:310379184, total:353432001064 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353432001064 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3402000.028546] LustreError: 35179:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 85 previous similar messages [3402123.755166] Lustre: oak-MDT0000: haven't heard from client 81823084-f29f-4c35-adec-84c8a1e14557 (at 10.51.10.37@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90299880fc00, cur 1713666357 expire 1713666207 last 1713666130 [3402123.777127] Lustre: Skipped 1 previous similar message [3402601.717970] LustreError: 35147:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852945552 granted:310379184, total:353432001064 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353432001064 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3402601.748514] LustreError: 35147:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 86 previous similar messages [3403201.730624] LustreError: 36662:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852945552 granted:310379184, total:353432001064 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353432001064 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3403201.761168] LustreError: 36662:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 84 previous similar messages [3403801.743149] LustreError: 35200:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852945552 granted:310379184, total:353432001064 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353432001064 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3403801.773712] LustreError: 35200:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 85 previous similar messages [3404401.755507] LustreError: 36666:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852945552 granted:310379184, total:353432001064 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353432001064 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3404401.786055] LustreError: 36666:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 87 previous similar messages [3404784.635885] Lustre: oak-MDT0000: haven't heard from client 647285f9-3316-4296-ace2-fa8dc2420f0c (at 10.50.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9019d7ca5c00, cur 1713669018 expire 1713668868 last 1713668791 [3404784.657670] Lustre: Skipped 1 previous similar message [3404789.633881] Lustre: oak-MDT0003: haven't heard from client 647285f9-3316-4296-ace2-fa8dc2420f0c (at 10.50.6.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9019d7d5c000, cur 1713669023 expire 1713668873 last 1713668796 [3405001.768067] LustreError: 35179:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852945552 granted:310379184, total:353432001064 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353432001064 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3405001.798603] LustreError: 35179:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 89 previous similar messages [3405603.692148] LustreError: 35236:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107934665032 granted:424654352, total:353432001064 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353432001064 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3405603.722687] LustreError: 35236:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 86 previous similar messages [3406203.739770] LustreError: 35237:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107934665032 granted:424654352, total:353432001064 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353432001064 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3406203.770310] LustreError: 35237:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 87 previous similar messages [3406803.788201] LustreError: 35222:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107934665032 granted:424654352, total:353432001064 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353432001064 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3406803.818894] LustreError: 35222:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 85 previous similar messages [3407403.835778] LustreError: 36686:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107934665032 granted:424654352, total:353432001064 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353432001064 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3407403.866503] LustreError: 36686:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 86 previous similar messages [3407804.493895] Lustre: oak-MDT0003: haven't heard from client 4c139e1a-2c79-4ce5-a598-f17e4caacba3 (at 10.51.5.71@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903b0621dc00, cur 1713672038 expire 1713671888 last 1713671811 [3408003.884780] LustreError: 36693:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107934665032 granted:424654352, total:353432001064 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353432001064 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3408003.915320] LustreError: 36693:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 83 previous similar messages [3408603.933713] LustreError: 35165:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107934665032 granted:424654352, total:353432001064 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353432001064 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3408603.964251] LustreError: 35165:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 81 previous similar messages [3409203.981150] LustreError: 36686:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107934665032 granted:424654352, total:353432001064 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353432001064 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3409204.011692] LustreError: 36686:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 79 previous similar messages [3409817.870239] LustreError: 35143:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536794032 granted:559613176, total:353432001064 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353432001064 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3409817.900988] LustreError: 35143:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 76 previous similar messages [3410417.850906] LustreError: 36655:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3410417.880878] LustreError: 36655:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 83 previous similar messages [3410765.357807] Lustre: oak-MDT0003: haven't heard from client 1ecb1e3a-0f9f-4b21-85ea-b44b254f8174 (at 10.50.6.23@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9020fce30400, cur 1713674999 expire 1713674849 last 1713674772 [3410765.379685] Lustre: Skipped 1 previous similar message [3411017.855242] LustreError: 36669:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536794032 granted:559613176, total:353432001064 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353432001064 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3411017.885786] LustreError: 36669:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 82 previous similar messages [3411617.835689] LustreError: 31510:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3411617.865530] LustreError: 31510:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 77 previous similar messages [3412221.744432] LustreError: 36677:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107949056292 granted:483067692, total:353432001064 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353432001064 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3412221.774972] LustreError: 36677:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 85 previous similar messages [3412821.756688] LustreError: 35164:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107949056292 granted:483067692, total:353432001064 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353432001064 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3412821.787230] LustreError: 35164:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 85 previous similar messages [3413421.872469] LustreError: 31509:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852945552 granted:310379184, total:353432001064 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353432001064 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3413421.903011] LustreError: 31509:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 86 previous similar messages [3414021.884878] LustreError: 35202:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852945552 granted:310379184, total:353432001064 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353432001064 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3414021.915691] LustreError: 35202:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 86 previous similar messages [3414621.897183] LustreError: 35144:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852945552 granted:310379184, total:353432001064 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353432001064 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3414621.927892] LustreError: 35144:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 86 previous similar messages [3414822.170828] Lustre: oak-MDT0003: haven't heard from client 71825029-9d75-4285-bcf1-e825d4e46f83 (at 10.51.6.13@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900f5bb56400, cur 1713679056 expire 1713678906 last 1713678829 [3414822.192700] Lustre: Skipped 1 previous similar message [3414827.171442] Lustre: oak-MDT0000: haven't heard from client 71825029-9d75-4285-bcf1-e825d4e46f83 (at 10.51.6.13@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900dd12ca400, cur 1713679061 expire 1713678911 last 1713678834 [3415226.179065] LustreError: 35205:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536794032 granted:559613176, total:353432001064 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353432001064 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3415226.209800] LustreError: 35205:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 80 previous similar messages [3415838.962801] LustreError: 35180:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3415838.992473] LustreError: 35180:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 83 previous similar messages [3416447.376136] LustreError: 36681:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107949056292 granted:483067692, total:353432001064 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353432001064 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3416447.406676] LustreError: 36681:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 87 previous similar messages [3417049.794418] LustreError: 36678:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107934665032 granted:424654352, total:353432001064 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353432001064 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3417049.824949] LustreError: 36678:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 81 previous similar messages [3417656.485090] LustreError: 36660:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107965008752 granted:486160868, total:353432001064 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353432001064 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3417656.515627] LustreError: 36660:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 77 previous similar messages [3418267.292553] LustreError: 36661:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3418267.322603] LustreError: 36661:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 78 previous similar messages [3418869.524269] LustreError: 35229:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107949056292 granted:483067692, total:353432001064 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353432001064 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3418869.524270] LustreError: 36698:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3418869.524272] LustreError: 36698:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 79 previous similar messages [3419367.958330] Lustre: oak-MDT0003: haven't heard from client b984b4b4-6d1d-45e6-8404-95d75ff9fbe8 (at 10.51.9.13@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902c110e8000, cur 1713683602 expire 1713683452 last 1713683375 [3419473.466938] LustreError: 36685:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107949056292 granted:483067692, total:353432001064 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353432001064 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3419473.497472] LustreError: 36685:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 72 previous similar messages [3420090.796678] LustreError: 36692:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107934665032 granted:424654352, total:353432001064 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353432001064 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3420090.827216] LustreError: 36692:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 82 previous similar messages [3420694.186364] LustreError: 35167:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107934665032 granted:424654352, total:353432001064 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353432001064 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3420694.216898] LustreError: 35167:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 76 previous similar messages [3421298.455710] LustreError: 35178:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3421298.485683] LustreError: 35178:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 78 previous similar messages [3421903.999581] LustreError: 35169:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107934665032 granted:424654352, total:353432001064 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353432001064 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3421904.030114] LustreError: 35169:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 79 previous similar messages [3422504.691896] LustreError: 36692:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107949056292 granted:483067692, total:353432001064 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353432001064 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3422504.722594] LustreError: 36692:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 77 previous similar messages [3423106.681260] LustreError: 35227:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107934665032 granted:424654352, total:353432001064 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353432001064 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3423106.711796] LustreError: 35227:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 79 previous similar messages [3423706.870785] LustreError: 36676:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107949056292 granted:483067692, total:353432001064 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353432001064 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3423706.901322] LustreError: 36676:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 77 previous similar messages [3424311.264872] LustreError: 35195:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3424311.294547] LustreError: 35195:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 105 previous similar messages [3424920.640726] LustreError: 36660:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107814850800 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3424920.670402] LustreError: 36660:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 80 previous similar messages [3425526.619653] LustreError: 36672:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3425526.649500] LustreError: 36672:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 76 previous similar messages [3426130.037228] LustreError: 36665:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107814850800 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3426130.066907] LustreError: 36665:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 73 previous similar messages [3426534.623456] Lustre: oak-MDT0003: haven't heard from client 4ce42d4f-c5c4-4165-974d-81423bbf5477 (at 10.50.1.51@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903b41ef5c00, cur 1713690769 expire 1713690619 last 1713690542 [3426534.645338] Lustre: Skipped 3 previous similar messages [3426751.102070] LustreError: 36654:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536794020 granted:559613176, total:353427806760 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353427806760 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3426751.132613] LustreError: 36654:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 76 previous similar messages [3427351.702129] LustreError: 35197:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3427351.731980] LustreError: 35197:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 77 previous similar messages [3427542.578971] Lustre: oak-MDT0000: haven't heard from client b6664cc8-78fe-429b-80d1-6665c9eada6d (at 10.51.1.34@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900579d3c800, cur 1713691777 expire 1713691627 last 1713691550 [3427542.600856] Lustre: Skipped 1 previous similar message [3427959.600275] LustreError: 36688:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3427959.600276] LustreError: 35163:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107949056240 granted:483067692, total:353427806760 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353427806760 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3427959.600279] LustreError: 35163:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 74 previous similar messages [3428490.532882] Lustre: oak-MDT0003: haven't heard from client 1869aa2c-1f2a-4c8d-b803-7118dbb1933a (at 10.50.1.26@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900e2495a400, cur 1713692725 expire 1713692575 last 1713692498 [3428490.554943] Lustre: Skipped 1 previous similar message [3428566.530851] Lustre: oak-MDT0000: haven't heard from client ee0f0722-253d-4a8e-8348-bbcaa6d16793 (at 10.50.16.9@o2ib2) in 193 seconds. I think it's dead, and I am evicting it. exp ffff901d5adb4800, cur 1713692801 expire 1713692651 last 1713692608 [3428566.552724] Lustre: Skipped 1 previous similar message [3428568.625305] LustreError: 36655:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107814850800 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3428568.625307] LustreError: 36662:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107965008704 granted:486160868, total:353427806760 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353427806760 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3428568.625310] LustreError: 36662:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 75 previous similar messages [3428920.517079] Lustre: oak-MDT0000: haven't heard from client 8d693977-2adc-4d3e-ba21-bf2df1fa388d (at 10.50.6.8@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903a3952f800, cur 1713693155 expire 1713693005 last 1713692928 [3428920.539041] Lustre: Skipped 1 previous similar message [3428924.511832] Lustre: oak-MDT0003: haven't heard from client 623c675f-c9d5-4e87-b2b1-fc8cc8653468 (at 10.50.10.2@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9026a1533400, cur 1713693159 expire 1713693009 last 1713692932 [3428924.533708] Lustre: Skipped 4 previous similar messages [3429184.198727] LustreError: 36655:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852943888 granted:310379184, total:353427806760 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353427806760 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3429184.229268] LustreError: 36655:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 78 previous similar messages [3429477.490892] Lustre: oak-MDT0000: haven't heard from client 31ab9408-e93d-4628-a96c-78681033431a (at 10.51.1.44@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901086cd1800, cur 1713693712 expire 1713693562 last 1713693485 [3429477.512766] Lustre: Skipped 4 previous similar messages [3429804.003918] LustreError: 35178:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3429804.033598] LustreError: 35178:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 75 previous similar messages [3430087.457939] Lustre: oak-MDT0000: haven't heard from client 5fc72d50-2470-4b8a-be8c-ba2f25e4befc (at 10.50.1.55@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903bc287e400, cur 1713694322 expire 1713694172 last 1713694095 [3430087.480136] Lustre: Skipped 31 previous similar messages [3430421.554891] LustreError: 35226:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107934665712 granted:424654352, total:353427806760 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353427806760 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3430421.585434] LustreError: 35226:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 68 previous similar messages [3431022.074417] LustreError: 36656:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3431022.104269] LustreError: 36656:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 65 previous similar messages [3431639.992265] LustreError: 35197:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536794020 granted:559613176, total:353427806760 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353427806760 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3431640.022810] LustreError: 35197:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 71 previous similar messages [3432255.080060] LustreError: 36672:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3432255.109739] LustreError: 36672:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 78 previous similar messages [3432337.353646] Lustre: oak-MDT0000: haven't heard from client 99b88d54-8282-4422-9697-0d77bf1379ad (at 10.50.7.62@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900d9f806400, cur 1713696572 expire 1713696422 last 1713696345 [3432337.375523] Lustre: Skipped 1 previous similar message [3432857.477338] LustreError: 35199:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536794020 granted:559613176, total:353427806760 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353427806760 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3432857.507879] LustreError: 35199:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 86 previous similar messages [3433457.766906] LustreError: 36654:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536794020 granted:559613176, total:353427806760 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353427806760 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3433457.797458] LustreError: 36654:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 87 previous similar messages [3434050.274587] Lustre: oak-MDT0000: haven't heard from client 5f793d32-43a7-4279-a43c-17a8a8ec06af (at 10.51.12.7@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900e5ab8e800, cur 1713698285 expire 1713698135 last 1713698058 [3434050.296466] Lustre: Skipped 1 previous similar message [3434059.783682] LustreError: 31510:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536794020 granted:559613176, total:353427806760 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353427806760 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3434059.814410] LustreError: 31510:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 86 previous similar messages [3434660.722288] LustreError: 36659:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536794020 granted:559613176, total:353427806760 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353427806760 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3434660.752833] LustreError: 36659:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 85 previous similar messages [3435267.225228] LustreError: 35181:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852943888 granted:310379184, total:353427806760 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353427806760 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3435267.255774] LustreError: 35181:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 89 previous similar messages [3435875.530436] LustreError: 35205:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107965008704 granted:486160868, total:353427806760 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353427806760 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3435875.560981] LustreError: 35205:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 90 previous similar messages [3436498.281023] LustreError: 35168:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3436498.311043] LustreError: 35168:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 91 previous similar messages [3437107.180451] LustreError: 36650:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3437107.210300] LustreError: 36650:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 84 previous similar messages [3437724.193822] LustreError: 36667:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536794020 granted:559613176, total:353427806760 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353427806760 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3437724.224363] LustreError: 36667:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 84 previous similar messages [3438324.599366] LustreError: 36666:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536794020 granted:559613176, total:353427806760 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353427806760 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3438324.629908] LustreError: 36666:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 82 previous similar messages [3438931.019730] LustreError: 36667:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852943888 granted:310379184, total:353427806760 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353427806760 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3438931.050267] LustreError: 36667:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 84 previous similar messages [3439531.062725] LustreError: 31510:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852943888 granted:310379184, total:353427806760 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353427806760 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3439531.093394] LustreError: 31510:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 85 previous similar messages [3440137.116294] LustreError: 35220:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107934665712 granted:424654352, total:353427806760 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353427806760 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3440137.146831] LustreError: 35220:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 86 previous similar messages [3440738.964268] LustreError: 36661:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107965008704 granted:486160868, total:353427806760 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353427806760 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3440738.994811] LustreError: 36661:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 88 previous similar messages [3441356.862752] LustreError: 35175:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3441356.892769] LustreError: 35175:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 88 previous similar messages [3441968.173944] LustreError: 35153:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3441968.203791] LustreError: 35153:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 87 previous similar messages [3442570.158835] LustreError: 36685:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3442570.188857] LustreError: 36685:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 89 previous similar messages [3443180.771778] LustreError: 35146:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3443180.801622] LustreError: 35146:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 90 previous similar messages [3443780.858406] LustreError: 35152:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3443780.888240] LustreError: 35152:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 89 previous similar messages [3444383.741792] LustreError: 35144:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536794020 granted:559613176, total:353427806760 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353427806760 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3444383.772338] LustreError: 35144:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 87 previous similar messages [3444983.820293] LustreError: 36655:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536794020 granted:559613176, total:353427806760 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353427806760 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3444983.850834] LustreError: 36655:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 89 previous similar messages [3445514.737158] Lustre: oak-MDT0003: haven't heard from client 8950d4a8-81bc-4788-96f9-3e5021feec7b (at 10.51.6.59@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901f795fa000, cur 1713709750 expire 1713709600 last 1713709523 [3445514.759184] Lustre: Skipped 3 previous similar messages [3445583.967320] LustreError: 36664:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536794020 granted:559613176, total:353427806760 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353427806760 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3445583.997867] LustreError: 36664:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 88 previous similar messages [3446190.726868] LustreError: 35195:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852943888 granted:310379184, total:353427806760 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353427806760 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3446190.757411] LustreError: 35195:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 90 previous similar messages [3446782.689538] Lustre: oak-MDT0003: haven't heard from client b3bf9f1a-a993-4b34-908c-45a329d30821 (at 10.51.7.39@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90143ea23400, cur 1713711018 expire 1713710868 last 1713710791 [3446782.711413] Lustre: Skipped 1 previous similar message [3446787.677939] Lustre: oak-MDT0000: haven't heard from client b3bf9f1a-a993-4b34-908c-45a329d30821 (at 10.51.7.39@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90143ea31c00, cur 1713711023 expire 1713710873 last 1713710796 [3446787.699814] Lustre: Skipped 2 previous similar messages [3446798.719512] LustreError: 35192:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107814850800 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3446798.749185] LustreError: 35192:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 91 previous similar messages [3447398.807615] LustreError: 35203:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107965008704 granted:486160868, total:353427806760 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353427806760 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3447398.838159] LustreError: 35203:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 89 previous similar messages [3448027.781992] LustreError: 35153:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3448027.812014] LustreError: 35153:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 91 previous similar messages [3448360.604204] Lustre: oak-MDT0000: haven't heard from client a733c9c2-31a5-4bfa-a44a-bd38a4d32463 (at 10.51.14.2@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9029c8a9dc00, cur 1713712596 expire 1713712446 last 1713712369 [3448360.626078] Lustre: Skipped 2 previous similar messages [3448627.894738] LustreError: 35147:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3448627.924583] LustreError: 35147:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 88 previous similar messages [3449152.567423] Lustre: oak-MDT0003: haven't heard from client 3636a767-f794-4352-be3f-2e3e619059ee (at 10.50.3.36@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900e378d8800, cur 1713713388 expire 1713713238 last 1713713161 [3449152.589292] Lustre: Skipped 1 previous similar message [3449233.509302] LustreError: 31507:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3449233.539321] LustreError: 31507:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 91 previous similar messages [3449833.486489] LustreError: 35163:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107949056240 granted:483067692, total:353427806760 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353427806760 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3449833.517027] LustreError: 35163:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 88 previous similar messages [3450434.069846] LustreError: 35235:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107949056240 granted:483067692, total:353427806760 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353427806760 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3450434.100384] LustreError: 35235:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 89 previous similar messages [3451040.529944] LustreError: 35155:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3451040.559617] LustreError: 35155:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 89 previous similar messages [3451644.871300] LustreError: 35157:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536794020 granted:559613176, total:353427806760 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353427806760 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3451644.901843] LustreError: 35157:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 87 previous similar messages [3452245.130932] LustreError: 31510:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536794020 granted:559613176, total:353427806760 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353427806760 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3452245.161631] LustreError: 31510:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 88 previous similar messages [3452845.907681] LustreError: 35206:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3452845.937349] LustreError: 35206:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 90 previous similar messages [3453316.372623] Lustre: oak-MDT0003: haven't heard from client d1805cfb-e404-4805-a14c-5b01b825a171 (at 10.51.9.16@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900c379d0c00, cur 1713717552 expire 1713717402 last 1713717325 [3453316.394499] Lustre: Skipped 1 previous similar message [3453446.467057] LustreError: 35144:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3453446.496732] LustreError: 35144:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 87 previous similar messages [3454053.398676] LustreError: 35147:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852943888 granted:310379184, total:353427806760 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353427806760 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3454053.429221] LustreError: 35147:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 90 previous similar messages [3454659.427740] LustreError: 36668:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107965008704 granted:486160868, total:353427806760 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353427806760 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3454659.458281] LustreError: 36668:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 91 previous similar messages [3455262.224075] LustreError: 36682:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107934665712 granted:424654352, total:353427806760 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353427806760 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3455262.254608] LustreError: 36682:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 87 previous similar messages [3455862.363021] LustreError: 36694:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107934665712 granted:424654352, total:353427806760 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353427806760 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3455862.393740] LustreError: 36694:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 88 previous similar messages [3456462.624556] LustreError: 36682:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107934665712 granted:424654352, total:353427806760 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353427806760 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3456462.655098] LustreError: 36682:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 89 previous similar messages [3457088.323557] LustreError: 36659:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3457088.353550] LustreError: 36659:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 90 previous similar messages [3457688.369829] LustreError: 35181:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3457688.399675] LustreError: 35181:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 85 previous similar messages [3458288.758158] LustreError: 35197:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3458288.788003] LustreError: 35197:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 89 previous similar messages [3458513.128137] Lustre: oak-MDT0000: haven't heard from client d2c55f23-2658-4c1e-b1f8-6b63ea2b7798 (at 10.51.3.45@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9027eb14c800, cur 1713722749 expire 1713722599 last 1713722522 [3458513.150009] Lustre: Skipped 1 previous similar message [3458523.127662] Lustre: oak-MDT0003: haven't heard from client fcfe0622-ae26-422a-82d0-067c88a795e3 (at 10.51.4.9@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901c6fba3c00, cur 1713722759 expire 1713722609 last 1713722532 [3458523.149600] Lustre: Skipped 7 previous similar messages [3458898.127999] LustreError: 36688:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107949056240 granted:483067692, total:353427806760 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353427806760 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3458898.158540] LustreError: 36688:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 90 previous similar messages [3459498.356255] LustreError: 35219:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3459498.386272] LustreError: 35219:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 88 previous similar messages [3460112.151202] LustreError: 35180:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3460112.180879] LustreError: 35180:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 88 previous similar messages [3460156.049880] Lustre: oak-MDT0000: haven't heard from client 819cb039-6ad9-4cf7-94d5-d15fa6c08bf4 (at 10.50.1.41@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9037e0239800, cur 1713724392 expire 1713724242 last 1713724165 [3460156.071885] Lustre: Skipped 7 previous similar messages [3460719.144346] LustreError: 35219:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107934665712 granted:424654352, total:353427806760 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353427806760 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3460719.175040] LustreError: 35219:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 84 previous similar messages [3461329.212600] LustreError: 36654:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852943888 granted:310379184, total:353427806760 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353427806760 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3461329.243139] LustreError: 36654:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 84 previous similar messages [3461931.892391] LustreError: 36658:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852943888 granted:310379184, total:353427806760 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353427806760 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3461931.922932] LustreError: 36658:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 89 previous similar messages [3462272.946158] Lustre: oak-MDT0003: haven't heard from client c49b1681-c4d9-4b8f-8723-866b6d183278 (at 10.50.1.49@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902655a79c00, cur 1713726509 expire 1713726359 last 1713726282 [3462272.968028] Lustre: Skipped 1 previous similar message [3462277.946423] Lustre: oak-MDT0000: haven't heard from client c49b1681-c4d9-4b8f-8723-866b6d183278 (at 10.50.1.49@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90240c262000, cur 1713726514 expire 1713726364 last 1713726287 [3462529.934469] Lustre: oak-MDT0000: haven't heard from client ff0a90d5-14e0-4983-9382-cf21de78f4eb (at 10.50.1.28@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903b1fc8b000, cur 1713726766 expire 1713726616 last 1713726539 [3462531.983537] LustreError: 35149:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852943888 granted:310379184, total:353427806760 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353427806760 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3462532.014271] LustreError: 35149:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 86 previous similar messages [3463133.156905] LustreError: 35179:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852943888 granted:310379184, total:353427806760 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353427806760 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3463133.187444] LustreError: 35179:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 87 previous similar messages [3463744.797631] LustreError: 36655:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536794020 granted:559613176, total:353427806760 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353427806760 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3463744.828174] LustreError: 36655:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 89 previous similar messages [3464258.849865] Lustre: oak-MDT0000: haven't heard from client 1108dfc9-d686-40a5-8ddc-4f9964cf063b (at 10.51.15.16@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9002fce91c00, cur 1713728495 expire 1713728345 last 1713728268 [3464258.871832] Lustre: Skipped 3 previous similar messages [3464345.119388] LustreError: 35149:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536794020 granted:559613176, total:353427806760 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353427806760 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3464345.149930] LustreError: 35149:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 88 previous similar messages [3464945.111663] LustreError: 35153:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536794020 granted:559613176, total:353427806760 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353427806760 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3464945.142368] LustreError: 35153:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 89 previous similar messages [3465359.796295] Lustre: oak-MDT0000: haven't heard from client 162f3b89-2c8f-42c0-8534-bd28a6e1661d (at 10.50.1.12@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9002ad756000, cur 1713729596 expire 1713729446 last 1713729369 [3465359.818665] Lustre: Skipped 1 previous similar message [3465554.881855] LustreError: 35205:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107965008704 granted:486160868, total:353427806760 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353427806760 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3465554.912554] LustreError: 35205:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 91 previous similar messages [3466154.894616] LustreError: 36665:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107965008704 granted:486160868, total:353427806760 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353427806760 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3466154.925306] LustreError: 36665:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 89 previous similar messages [3466755.183147] LustreError: 35176:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107965008704 granted:486160868, total:353427806760 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353427806760 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3466755.214177] LustreError: 35176:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 89 previous similar messages [3467025.712468] Lustre: oak-MDT0003: haven't heard from client 50dea4f1-dd28-4141-b1e3-03845159f262 (at 10.50.1.4@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903d1b4b3000, cur 1713731262 expire 1713731112 last 1713731035 [3467025.734257] Lustre: Skipped 1 previous similar message [3467031.715794] Lustre: oak-MDT0000: haven't heard from client 50dea4f1-dd28-4141-b1e3-03845159f262 (at 10.50.1.4@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903a0bf4b400, cur 1713731268 expire 1713731118 last 1713731041 [3467371.784479] LustreError: 35184:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3467371.814630] LustreError: 35184:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 91 previous similar messages [3467973.606638] LustreError: 36692:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3467973.636652] LustreError: 36692:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 89 previous similar messages [3468574.077359] LustreError: 36693:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107949056240 granted:483067692, total:353427806760 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353427806760 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3468574.107894] LustreError: 36693:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 87 previous similar messages [3469175.283989] LustreError: 35218:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107949056240 granted:483067692, total:353427806760 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353427806760 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3469175.314531] LustreError: 35218:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 89 previous similar messages [3469254.602718] Lustre: oak-MDT0003: haven't heard from client cb5c916f-013a-4580-8950-cd194f120dac (at 10.50.4.11@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902078835800, cur 1713733491 expire 1713733341 last 1713733264 [3469261.602270] Lustre: oak-MDT0000: haven't heard from client cb5c916f-013a-4580-8950-cd194f120dac (at 10.50.4.11@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90073d8ad000, cur 1713733498 expire 1713733348 last 1713733271 [3469261.624264] Lustre: Skipped 1 previous similar message [3469330.602780] Lustre: oak-MDT0003: haven't heard from client 5df38e79-2465-4e0c-97be-0936d4598775 (at 10.50.2.18@o2ib2) in 186 seconds. I think it's dead, and I am evicting it. exp ffff90381cb70800, cur 1713733567 expire 1713733417 last 1713733381 [3469330.624657] Lustre: Skipped 1 previous similar message [3469337.599495] Lustre: oak-MDT0000: haven't heard from client 5df38e79-2465-4e0c-97be-0936d4598775 (at 10.50.2.18@o2ib2) in 193 seconds. I think it's dead, and I am evicting it. exp ffff902fe5d51000, cur 1713733574 expire 1713733424 last 1713733381 [3469776.943478] LustreError: 35151:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3469776.973329] LustreError: 35151:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 90 previous similar messages [3470378.694917] LustreError: 35234:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107934665712 granted:424654352, total:353427806760 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353427806760 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3470378.725740] LustreError: 35234:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 82 previous similar messages [3470978.886056] LustreError: 36675:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107934665712 granted:424654352, total:353427806760 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353427806760 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3470978.917134] LustreError: 36675:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 82 previous similar messages [3471132.509702] Lustre: oak-MDT0003: haven't heard from client a8532752-981f-4316-afe2-fab9edc82877 (at 10.50.2.20@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903520f3c400, cur 1713735369 expire 1713735219 last 1713735142 [3471598.948214] LustreError: 35234:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107949056240 granted:483067692, total:353427806760 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353427806760 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3471598.978761] LustreError: 35234:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 84 previous similar messages [3472201.471756] LustreError: 35208:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3472201.501433] LustreError: 35208:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 83 previous similar messages [3472674.444493] Lustre: oak-MDT0000: haven't heard from client a0c872a1-1e4a-4e17-a2cf-84edbb869d7a (at 10.51.10.19@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901f794f4800, cur 1713736911 expire 1713736761 last 1713736684 [3472674.466451] Lustre: Skipped 1 previous similar message [3472814.956987] LustreError: 35206:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107814850800 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3472814.986808] LustreError: 35206:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 80 previous similar messages [3473436.192458] LustreError: 35198:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536794020 granted:559613176, total:353427806760 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353427806760 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3473436.222997] LustreError: 35198:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 74 previous similar messages [3474052.594222] LustreError: 36672:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3474052.624068] LustreError: 36672:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 78 previous similar messages [3474663.877671] LustreError: 35158:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107814850800 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3474663.907342] LustreError: 35158:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 82 previous similar messages [3475265.366774] LustreError: 35179:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3475265.396617] LustreError: 35179:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 78 previous similar messages [3475697.284455] Lustre: oak-MDT0003: haven't heard from client 54f0675a-7301-4de8-9b76-215ee513b544 (at 10.50.10.18@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9029de3ef800, cur 1713739934 expire 1713739784 last 1713739707 [3475697.306419] Lustre: Skipped 1 previous similar message [3475710.285243] Lustre: oak-MDT0000: haven't heard from client 54f0675a-7301-4de8-9b76-215ee513b544 (at 10.50.10.18@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9040fd31f800, cur 1713739947 expire 1713739797 last 1713739720 [3475773.281073] Lustre: oak-MDT0003: haven't heard from client 4ea848d8-36a8-4a40-bd76-e1f5e709ddea (at 10.51.7.33@o2ib3) in 220 seconds. I think it's dead, and I am evicting it. exp ffff90143ebed000, cur 1713740010 expire 1713739860 last 1713739790 [3475786.278912] Lustre: oak-MDT0000: haven't heard from client 8a8ba81a-3732-450e-b6f5-f8e92398996e (at 10.51.7.43@o2ib3) in 216 seconds. I think it's dead, and I am evicting it. exp ffff9032e1c7d800, cur 1713740023 expire 1713739873 last 1713739807 [3475786.300790] Lustre: Skipped 4 previous similar messages [3475868.073394] LustreError: 36700:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107934665712 granted:424654352, total:353427806760 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353427806760 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3475868.104099] LustreError: 36700:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 80 previous similar messages [3476030.271301] Lustre: oak-MDT0000: haven't heard from client 53a6c389-b982-442f-8600-47d802158138 (at 10.51.5.31@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903e94159000, cur 1713740267 expire 1713740117 last 1713740040 [3476030.293174] Lustre: Skipped 4 previous similar messages [3476481.426312] LustreError: 35205:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3476481.456165] LustreError: 35205:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 81 previous similar messages [3477081.436607] LustreError: 35200:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3477081.466459] LustreError: 35200:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 81 previous similar messages [3477691.569658] LustreError: 35156:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852943888 granted:310379184, total:353427806760 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353427806760 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3477691.600459] LustreError: 35156:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 75 previous similar messages [3478294.024211] LustreError: 35226:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925660916 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925660916 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3478294.054415] LustreError: 35226:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 71 previous similar messages [3478897.059665] LustreError: 35199:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536794020 granted:559613176, total:353427806760 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353427806760 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3478897.090208] LustreError: 35199:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 74 previous similar messages [3479526.241397] LustreError: 35195:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3479526.271241] LustreError: 35195:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 83 previous similar messages [3480126.250615] LustreError: 36661:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3480126.280465] LustreError: 36661:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 84 previous similar messages [3480739.448565] LustreError: 36667:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852943888 granted:310379184, total:353427806760 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353427806760 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3480739.479108] LustreError: 36667:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 73 previous similar messages [3481359.260403] LustreError: 36668:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536794020 granted:559613176, total:353427806760 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353427806760 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3481359.290939] LustreError: 36668:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 77 previous similar messages [3481998.069702] LustreError: 35199:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3481998.099552] LustreError: 35199:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 74 previous similar messages [3482323.954843] Lustre: oak-MDT0000: haven't heard from client 369a6ec9-77a7-4f5b-aaf1-c8bf15ad5ac8 (at 10.51.16.17@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900380929c00, cur 1713746561 expire 1713746411 last 1713746334 [3482323.976807] Lustre: Skipped 1 previous similar message [3482604.840025] LustreError: 35163:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59925848088 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59925848088 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3482604.870044] LustreError: 35163:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 80 previous similar messages [3482933.924684] Lustre: oak-MDT0000: haven't heard from client 60e096d9-6f00-4977-b147-d002f1f448c0 (at 10.51.15.13@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903c80ea1800, cur 1713747171 expire 1713747021 last 1713746944 [3482933.946642] Lustre: Skipped 1 previous similar message [3482947.923938] Lustre: oak-MDT0003: haven't heard from client 60e096d9-6f00-4977-b147-d002f1f448c0 (at 10.51.15.13@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903a6139ec00, cur 1713747185 expire 1713747035 last 1713746958 [3483211.973429] LustreError: 36669:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536794020 granted:559613176, total:353426758184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353426758184 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3483212.003973] LustreError: 36669:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 76 previous similar messages [3483811.983753] LustreError: 35199:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536794020 granted:559613176, total:353426758184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353426758184 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3483812.014294] LustreError: 35199:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 78 previous similar messages [3484429.741611] LustreError: 35237:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107934665712 granted:424654352, total:353426758184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353426758184 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3484429.772148] LustreError: 35237:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 77 previous similar messages [3485031.344235] LustreError: 35154:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107965008704 granted:486160868, total:353426758184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353426758184 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3485031.374771] LustreError: 35154:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 75 previous similar messages [3485230.813317] Lustre: oak-MDT0003: haven't heard from client 2848af66-1c85-467c-98f9-d8670393ea09 (at 10.50.1.21@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903a39529800, cur 1713749468 expire 1713749318 last 1713749241 [3485654.787262] LustreError: 36669:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107965008704 granted:486160868, total:353426758184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353426758184 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3485654.818155] LustreError: 36669:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 83 previous similar messages [3486258.932456] LustreError: 35148:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3486258.962437] LustreError: 35148:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 77 previous similar messages [3486868.849837] LustreError: 36658:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536794020 granted:559613176, total:353426758184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353426758184 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3486868.880507] LustreError: 36658:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 78 previous similar messages [3487504.324281] LustreError: 35161:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107934665712 granted:424654352, total:353426758184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353426758184 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3487504.354977] LustreError: 35161:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 85 previous similar messages [3488118.470343] LustreError: 36650:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852943888 granted:310379184, total:353426758184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353426758184 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3488118.501088] LustreError: 36650:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 83 previous similar messages [3488728.877344] LustreError: 35157:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852943888 granted:310379184, total:353426758184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353426758184 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3488728.907883] LustreError: 35157:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 78 previous similar messages [3489346.230994] LustreError: 36650:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3489346.261030] LustreError: 36650:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 81 previous similar messages [3489946.284715] LustreError: 35153:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3489946.314570] LustreError: 35153:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 77 previous similar messages [3490547.939755] LustreError: 31509:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536794020 granted:559613176, total:353426758184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353426758184 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3490547.970294] LustreError: 31509:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 75 previous similar messages [3491148.130111] LustreError: 36687:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107949056240 granted:483067692, total:353426758184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353426758184 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3491148.160795] LustreError: 36687:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 80 previous similar messages [3491755.775475] LustreError: 36661:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3491755.805635] LustreError: 36661:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 72 previous similar messages [3492371.673934] LustreError: 36664:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852943888 granted:310379184, total:353426758184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353426758184 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3492371.704690] LustreError: 36664:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 74 previous similar messages [3492974.819807] LustreError: 35147:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3492974.849652] LustreError: 35147:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 77 previous similar messages [3493579.492620] LustreError: 35144:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3493579.522474] LustreError: 35144:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 75 previous similar messages [3494188.911714] LustreError: 36669:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536794020 granted:559613176, total:353426758184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353426758184 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3494188.942254] LustreError: 36669:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 75 previous similar messages [3494789.738531] LustreError: 35196:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536794020 granted:559613176, total:353426758184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353426758184 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3494789.769085] LustreError: 35196:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 82 previous similar messages [3495389.751246] LustreError: 35181:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536794020 granted:559613176, total:353426758184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353426758184 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3495389.782025] LustreError: 35181:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 75 previous similar messages [3496007.393698] LustreError: 35164:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107949056240 granted:483067692, total:353426758184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353426758184 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3496007.393699] LustreError: 35170:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59926117672 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59926117672 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3496007.393701] LustreError: 35170:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 75 previous similar messages [3496607.517919] LustreError: 36671:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107814850800 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3496607.547596] LustreError: 36671:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 79 previous similar messages [3497212.816582] LustreError: 35195:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536794020 granted:559613176, total:353426758184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353426758184 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3497212.847129] LustreError: 35195:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 79 previous similar messages [3497816.214391] LustreError: 35153:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852943888 granted:310379184, total:353426758184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353426758184 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3497816.244928] LustreError: 35153:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 75 previous similar messages [3498222.186822] Lustre: oak-MDT0000: haven't heard from client 0b5bec42-5da6-4623-875e-322b74f0ee33 (at 10.51.15.4@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900f86739c00, cur 1713762460 expire 1713762310 last 1713762233 [3498222.208699] Lustre: Skipped 1 previous similar message [3498233.184740] Lustre: oak-MDT0003: haven't heard from client 0b5bec42-5da6-4623-875e-322b74f0ee33 (at 10.51.15.4@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901989697000, cur 1713762471 expire 1713762321 last 1713762244 [3498425.630758] LustreError: 36661:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107965008704 granted:486160868, total:353426758184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353426758184 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3498425.661300] LustreError: 36661:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 79 previous similar messages [3499029.317937] LustreError: 35226:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107949056240 granted:483067692, total:353426758184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353426758184 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3499029.317938] LustreError: 36676:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59926117672 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59926117672 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3499029.317941] LustreError: 36676:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 79 previous similar messages [3499642.827420] LustreError: 35152:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107965008704 granted:486160868, total:353426758184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353426758184 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3499642.857959] LustreError: 35152:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 82 previous similar messages [3500245.573080] LustreError: 35202:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107965008704 granted:486160868, total:353426758184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353426758184 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3500245.603628] LustreError: 35202:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 82 previous similar messages [3500854.899482] LustreError: 35181:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3500854.929687] LustreError: 35181:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 82 previous similar messages [3501466.795072] LustreError: 35167:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107949056240 granted:483067692, total:353426758184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353426758184 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3501466.825607] LustreError: 35167:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 79 previous similar messages [3502070.658993] LustreError: 35205:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852943888 granted:310379184, total:353426758184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353426758184 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3502070.689537] LustreError: 35205:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 83 previous similar messages [3502289.993610] Lustre: oak-MDT0003: haven't heard from client 887b7a89-6e38-4c88-a375-22fe4a0e354d (at 10.51.5.52@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90064f0c0000, cur 1713766528 expire 1713766378 last 1713766301 [3502677.487503] LustreError: 35157:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536794020 granted:559613176, total:353426758184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353426758184 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3502677.518379] LustreError: 35157:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 79 previous similar messages [3502980.961183] Lustre: oak-MDT0003: haven't heard from client 5b2125e1-13f9-49f9-95c9-3e346e01f9a7 (at 10.51.11.3@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9029e2008000, cur 1713767219 expire 1713767069 last 1713766992 [3502980.983063] Lustre: Skipped 1 previous similar message [3503277.499708] LustreError: 35143:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536794020 granted:559613176, total:353426758184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353426758184 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3503277.530252] LustreError: 35143:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 80 previous similar messages [3503896.861295] LustreError: 35205:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536794020 granted:559613176, total:353426758184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353426758184 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3503896.891993] LustreError: 35205:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 80 previous similar messages [3504497.738802] LustreError: 35184:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59926117672 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59926117672 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3504497.768824] LustreError: 35184:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 83 previous similar messages [3505099.816582] LustreError: 35166:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107934665712 granted:424654352, total:353426758184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353426758184 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3505099.847280] LustreError: 35166:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 73 previous similar messages [3505703.826708] LustreError: 36666:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107965008704 granted:486160868, total:353426758184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353426758184 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3505703.857391] LustreError: 36666:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 77 previous similar messages [3506308.858819] LustreError: 35224:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107934665712 granted:424654352, total:353426758184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353426758184 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3506308.889365] LustreError: 35224:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 76 previous similar messages [3506910.204846] LustreError: 35163:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107934665712 granted:424654352, total:353426758184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353426758184 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3506910.235386] LustreError: 35163:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 81 previous similar messages [3507533.099131] LustreError: 36699:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107934665712 granted:424654352, total:353426758184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353426758184 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3507533.129665] LustreError: 36699:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 74 previous similar messages [3508136.192953] LustreError: 36666:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107965008704 granted:486160868, total:353426758184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353426758184 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3508136.223489] LustreError: 36666:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 78 previous similar messages [3508737.643708] LustreError: 35207:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852943888 granted:310379184, total:353426758184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353426758184 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3508737.674272] LustreError: 35207:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 86 previous similar messages [3509341.740062] LustreError: 35158:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3509341.769892] LustreError: 35158:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 85 previous similar messages [3509565.651664] Lustre: oak-MDT0000: haven't heard from client 3f85c78b-5db4-44fe-9c2e-a849e60ed234 (at 10.51.16.18@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903b0621b000, cur 1713773804 expire 1713773654 last 1713773577 [3509565.673926] Lustre: Skipped 1 previous similar message [3509946.764673] LustreError: 35198:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3509946.794533] LustreError: 35198:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 85 previous similar messages [3510546.818342] LustreError: 35207:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3510546.848195] LustreError: 35207:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 94 previous similar messages [3510894.589251] Lustre: oak-MDT0003: haven't heard from client c0472ae4-54fc-4e52-a434-830ef348d781 (at 10.51.15.4@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901105705800, cur 1713775133 expire 1713774983 last 1713774906 [3510894.611128] Lustre: Skipped 7 previous similar messages [3510914.587972] Lustre: oak-MDT0000: haven't heard from client c0472ae4-54fc-4e52-a434-830ef348d781 (at 10.51.15.4@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9002fce58800, cur 1713775153 expire 1713775003 last 1713774926 [3511146.863198] LustreError: 31510:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3511146.893051] LustreError: 31510:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 78 previous similar messages [3511752.729643] LustreError: 35200:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3511752.759325] LustreError: 35200:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 83 previous similar messages [3511823.546844] Lustre: oak-MDT0003: haven't heard from client dc79669b-a5a6-4ce0-ad66-9fc7b90ff062 (at 10.51.13.9@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903b08741000, cur 1713776062 expire 1713775912 last 1713775835 [3512365.084657] LustreError: 36658:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107965008652 granted:486160868, total:353426758184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353426758184 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3512365.115209] LustreError: 36658:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 82 previous similar messages [3512643.507273] Lustre: oak-MDT0000: haven't heard from client 53610ef1-eb02-4c42-8318-826cb7285a16 (at 10.51.6.26@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903a84c44c00, cur 1713776882 expire 1713776732 last 1713776655 [3512643.529260] Lustre: Skipped 1 previous similar message [3512977.534082] LustreError: 35178:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536793948 granted:559613176, total:353426758184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353426758184 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3512977.564627] LustreError: 35178:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 83 previous similar messages [3513577.580516] LustreError: 31510:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536793948 granted:559613176, total:353426758184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353426758184 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3513577.611058] LustreError: 31510:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 84 previous similar messages [3514179.433382] LustreError: 35192:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852943888 granted:310379184, total:353426758184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353426758184 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3514179.463921] LustreError: 35192:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 82 previous similar messages [3514812.442716] LustreError: 35143:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3514812.472387] LustreError: 35143:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 86 previous similar messages [3515413.531743] LustreError: 35145:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3515413.561433] LustreError: 35145:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 89 previous similar messages [3516021.241310] LustreError: 35176:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3516021.270988] LustreError: 35176:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 76 previous similar messages [3516622.951689] LustreError: 35218:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107949056084 granted:483067692, total:353426758184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353426758184 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3516622.982227] LustreError: 35218:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 83 previous similar messages [3517222.963625] LustreError: 35170:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59926117672 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59926117672 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3517222.993643] LustreError: 35170:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 88 previous similar messages [3517831.997073] LustreError: 35166:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59926117672 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59926117672 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3517832.027087] LustreError: 35166:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 80 previous similar messages [3518435.879229] LustreError: 35146:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107965008652 granted:486160868, total:353426758184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353426758184 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3518435.879230] LustreError: 35204:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107814850800 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3518435.879234] LustreError: 35204:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 81 previous similar messages [3519052.612286] LustreError: 35155:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107814850800 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3519052.642101] LustreError: 35155:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 83 previous similar messages [3519654.960477] LustreError: 35227:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107934665656 granted:424654352, total:353426758184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353426758184 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3519654.991012] LustreError: 35227:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 79 previous similar messages [3520259.227864] LustreError: 35163:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107934665656 granted:424654352, total:353426758184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353426758184 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3520259.258403] LustreError: 35163:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 83 previous similar messages [3520861.069813] LustreError: 36650:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536793920 granted:559613176, total:353426758184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353426758184 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3520861.100362] LustreError: 36650:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 82 previous similar messages [3521463.339710] LustreError: 36670:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536793920 granted:559613176, total:353426758184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353426758184 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3521463.370369] LustreError: 36670:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 83 previous similar messages [3522066.229212] LustreError: 35148:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536793920 granted:559613176, total:353426758184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353426758184 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3522066.259757] LustreError: 35148:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 79 previous similar messages [3522667.493850] LustreError: 36656:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536793920 granted:559613176, total:353426758184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353426758184 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3522667.524706] LustreError: 36656:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 80 previous similar messages [3523267.540388] LustreError: 35176:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536793920 granted:559613176, total:353426758184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353426758184 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3523267.571255] LustreError: 35176:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 85 previous similar messages [3523867.586922] LustreError: 35156:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536793920 granted:559613176, total:353426758184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353426758184 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3523867.617460] LustreError: 35156:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 82 previous similar messages [3524477.595648] LustreError: 36685:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107934665656 granted:424654352, total:353426758184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353426758184 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3524477.626376] LustreError: 36685:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 83 previous similar messages [3525084.087390] LustreError: 35179:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536793920 granted:559613176, total:353426758184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353426758184 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3525084.117931] LustreError: 35179:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 85 previous similar messages [3525692.290077] LustreError: 35172:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59926117672 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59926117672 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3525692.320102] LustreError: 35172:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 82 previous similar messages [3526293.578404] LustreError: 36657:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852943888 granted:310379184, total:353426758184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353426758184 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3526293.608949] LustreError: 36657:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 75 previous similar messages [3526333.863383] Lustre: oak-MDT0003: haven't heard from client edea68dc-b257-437e-b342-3493007b07b9 (at 10.51.0.13@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901c8f9e2800, cur 1713790573 expire 1713790423 last 1713790346 [3526333.885298] Lustre: Skipped 1 previous similar message [3526346.863053] Lustre: oak-MDT0000: haven't heard from client edea68dc-b257-437e-b342-3493007b07b9 (at 10.51.0.13@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901f0d4f6000, cur 1713790586 expire 1713790436 last 1713790359 [3526900.265544] LustreError: 36691:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107934665656 granted:424654352, total:353426758184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353426758184 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3526900.296093] LustreError: 36691:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 84 previous similar messages [3527503.518565] LustreError: 35178:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852943888 granted:310379184, total:353426758184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353426758184 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3527503.549103] LustreError: 35178:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 83 previous similar messages [3528113.645468] LustreError: 36679:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107949056084 granted:483067692, total:353426758184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353426758184 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3528113.676310] LustreError: 36679:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 87 previous similar messages [3528149.782000] Lustre: oak-MDT0000: haven't heard from client 3e9a8c69-06f0-42d1-ae6a-f7a4067012bb (at 10.51.16.19@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900f3b22d000, cur 1713792389 expire 1713792239 last 1713792162 [3528713.669763] LustreError: 35207:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536793920 granted:559613176, total:353426758184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353426758184 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3528713.700307] LustreError: 35207:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 83 previous similar messages [3529320.300402] LustreError: 35176:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3529320.330075] LustreError: 35176:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 85 previous similar messages [3529920.398356] LustreError: 35204:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3529920.428165] LustreError: 35204:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 81 previous similar messages [3530530.321747] LustreError: 36657:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536793920 granted:559613176, total:353426758184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353426758184 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3530530.352288] LustreError: 36657:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 82 previous similar messages [3531141.242994] LustreError: 35203:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107814850800 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3531141.272670] LustreError: 35203:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 82 previous similar messages [3531765.684279] LustreError: 35223:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59926117672 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59926117672 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3531765.684281] LustreError: 36695:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107949056084 granted:483067692, total:353426758184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353426758184 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3531765.684283] LustreError: 36695:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 85 previous similar messages [3532365.727121] LustreError: 36679:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107949056084 granted:483067692, total:353426758184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353426758184 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3532365.757763] LustreError: 36679:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 81 previous similar messages [3532965.739980] LustreError: 36687:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107949056084 granted:483067692, total:353426758184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353426758184 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3532965.770674] LustreError: 36687:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 82 previous similar messages [3533565.786786] LustreError: 35237:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107949056084 granted:483067692, total:353426758184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353426758184 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3533565.817324] LustreError: 35237:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 80 previous similar messages [3534165.834747] LustreError: 36684:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59926117672 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59926117672 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3534165.834749] LustreError: 36687:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107949056084 granted:483067692, total:353426758184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353426758184 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3534165.834751] LustreError: 36687:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 79 previous similar messages [3534765.912892] LustreError: 36675:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59926117672 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59926117672 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3534765.943022] LustreError: 36675:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 81 previous similar messages [3535236.467882] Lustre: oak-MDT0000: haven't heard from client 4379aa9c-d1e1-4343-8394-69dfd7bb0d25 (at 10.51.16.1@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901a48019400, cur 1713799476 expire 1713799326 last 1713799249 [3535236.489754] Lustre: Skipped 1 previous similar message [3535367.855128] LustreError: 35162:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107934665656 granted:424654352, total:353426758184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353426758184 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3535367.885662] LustreError: 35162:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 86 previous similar messages [3535992.315110] LustreError: 35174:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107949056084 granted:483067692, total:353426758184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353426758184 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3535992.345642] LustreError: 35174:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 84 previous similar messages [3536496.388644] Lustre: oak-MDT0000: haven't heard from client 1dce4f0c-c95a-4799-9676-6a3d15ee216a (at 10.50.12.8@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903230a42400, cur 1713800736 expire 1713800586 last 1713800509 [3536496.410514] Lustre: Skipped 3 previous similar messages [3536592.387319] LustreError: 36689:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107949056084 granted:483067692, total:353426758184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353426758184 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3536592.417858] LustreError: 36689:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 86 previous similar messages [3536894.368735] Lustre: oak-MDT0003: haven't heard from client 06066051-63c9-490f-8b06-d32ffb26bc1f (at 10.50.8.49@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901f79458800, cur 1713801134 expire 1713800984 last 1713800907 [3536894.390613] Lustre: Skipped 1 previous similar message [3537199.032833] LustreError: 36664:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107965008652 granted:486160868, total:353426758184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353426758184 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3537199.063398] LustreError: 36664:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 85 previous similar messages [3537803.772400] LustreError: 36683:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107934665656 granted:424654352, total:353426758184 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353426758184 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3537803.802940] LustreError: 36683:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 83 previous similar messages [3538403.784648] LustreError: 35229:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107933150268 granted:424654352, total:353473948612 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353473948612 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3538403.815327] LustreError: 35229:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 1543 previous similar messages [3539040.323214] LustreError: 35203:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852205124 granted:310379184, total:353473948612 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353473948612 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3539040.353753] LustreError: 35203:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 85 previous similar messages [3539044.268967] Lustre: oak-MDT0003: haven't heard from client 395f4f64-9918-40ca-ad4d-1ed6ece00012 (at 10.51.2.7@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901586e75000, cur 1713803284 expire 1713803134 last 1713803057 [3539044.290757] Lustre: Skipped 29 previous similar messages [3539645.248203] LustreError: 36666:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3539645.277998] LustreError: 36666:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 90 previous similar messages [3540245.922037] LustreError: 35143:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069535749612 granted:559613176, total:353472900036 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353472900036 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3540245.952576] LustreError: 35143:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 329 previous similar messages [3540871.334203] LustreError: 36696:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107933150212 granted:424654352, total:353472900036 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353472900036 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3540871.364746] LustreError: 36696:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 94 previous similar messages [3541076.194682] Lustre: oak-MDT0000: haven't heard from client c646f641-4f53-453d-9164-9bc54798bdf1 (at 10.51.10.18@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900c37991800, cur 1713805316 expire 1713805166 last 1713805089 [3541076.216649] Lustre: Skipped 5 previous similar messages [3541085.173711] Lustre: oak-MDT0003: haven't heard from client c646f641-4f53-453d-9164-9bc54798bdf1 (at 10.51.10.18@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90073d993800, cur 1713805325 expire 1713805175 last 1713805098 [3541473.347586] LustreError: 35199:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107962716644 granted:486160868, total:353472900036 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353472900036 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3541473.378136] LustreError: 35199:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 255 previous similar messages [3542078.533059] LustreError: 35146:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3542078.562736] LustreError: 35146:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 195 previous similar messages [3542687.460036] LustreError: 36699:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59926117672 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59926117672 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3542687.460037] LustreError: 35221:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107948248692 granted:483067692, total:353472900036 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353472900036 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3542687.460039] LustreError: 35221:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 81 previous similar messages [3543287.536964] LustreError: 36681:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107948248692 granted:483067692, total:353472900036 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353472900036 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3543287.567500] LustreError: 36681:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 84 previous similar messages [3543783.047065] Lustre: oak-MDT0003: haven't heard from client a50f2b7d-f572-40c1-aa4e-8775b3b5d1ef (at 10.50.12.8@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903645b46000, cur 1713808023 expire 1713807873 last 1713807796 [3543893.356122] LustreError: 31508:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107933002456 granted:424654352, total:353472900036 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353472900036 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3543893.386666] LustreError: 31508:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 79 previous similar messages [3543979.037378] Lustre: oak-MDT0003: haven't heard from client aa844653-762a-4281-a24b-702dab7eebb6 (at 10.50.6.17@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900b73e03000, cur 1713808219 expire 1713808069 last 1713807992 [3543979.059257] Lustre: Skipped 1 previous similar message [3544373.021758] Lustre: oak-MDT0000: haven't heard from client 4c60c2f1-30b4-4749-ab89-6148aeeec393 (at 10.51.10.18@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902746129400, cur 1713808613 expire 1713808463 last 1713808386 [3544373.043721] Lustre: Skipped 1 previous similar message [3544384.019566] Lustre: oak-MDT0003: haven't heard from client 4c60c2f1-30b4-4749-ab89-6148aeeec393 (at 10.51.10.18@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903e881e0800, cur 1713808624 expire 1713808474 last 1713808397 [3544493.434724] LustreError: 35170:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107933002456 granted:424654352, total:353472900036 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353472900036 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3544493.465412] LustreError: 35170:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 82 previous similar messages [3545093.481330] LustreError: 35223:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107933002460 granted:424654352, total:353472900036 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353472900036 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3545093.511863] LustreError: 35223:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 92 previous similar messages [3545693.658867] LustreError: 36681:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107933002460 granted:424654352, total:353472900036 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353472900036 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3545693.689410] LustreError: 36681:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 70 previous similar messages [3546293.672127] LustreError: 36689:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107933132880 granted:424654352, total:353472900036 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353472900036 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3546293.702674] LustreError: 36689:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 122 previous similar messages [3546483.918926] Lustre: oak-MDT0000: haven't heard from client e8c8124f-ddee-4b2e-9ae2-49bd781356b3 (at 10.51.14.13@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90067e1af800, cur 1713810724 expire 1713810574 last 1713810497 [3546499.928148] Lustre: oak-MDT0003: haven't heard from client e8c8124f-ddee-4b2e-9ae2-49bd781356b3 (at 10.51.14.13@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90054e627800, cur 1713810740 expire 1713810590 last 1713810513 [3546894.544539] LustreError: 36671:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3546894.574392] LustreError: 36671:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 81 previous similar messages [3547494.554559] LustreError: 36664:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3547494.584416] LustreError: 36664:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 79 previous similar messages [3548110.249775] LustreError: 35197:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852186680 granted:310379184, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3548110.280464] LustreError: 35197:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 78 previous similar messages [3548339.828349] Lustre: oak-MDT0000: haven't heard from client f3e08bee-90bc-4a01-860d-ec2a949a852f (at 10.50.8.27@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900439424000, cur 1713812580 expire 1713812430 last 1713812353 [3548415.825182] Lustre: oak-MDT0000: haven't heard from client d4791916-bca7-4a63-92c3-d1d26a25de2d (at 10.51.5.47@o2ib3) in 225 seconds. I think it's dead, and I am evicting it. exp ffff904037378000, cur 1713812656 expire 1713812506 last 1713812431 [3548415.847054] Lustre: Skipped 23 previous similar messages [3548710.261346] LustreError: 35149:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852186680 granted:310379184, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3548710.291888] LustreError: 35149:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 84 previous similar messages [3549315.869104] LustreError: 35200:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107814850800 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3549315.898774] LustreError: 35200:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 91 previous similar messages [3549541.769843] Lustre: oak-MDT0000: haven't heard from client 398769d3-74dd-435b-9a9f-34ed674c1d06 (at 10.51.5.67@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903c80ea6800, cur 1713813782 expire 1713813632 last 1713813555 [3549541.791712] Lustre: Skipped 23 previous similar messages [3549617.768058] Lustre: oak-MDT0000: haven't heard from client 886da161-af2e-4570-b502-841e52f51e66 (at 10.51.14.18@o2ib3) in 216 seconds. I think it's dead, and I am evicting it. exp ffff902f95201000, cur 1713813858 expire 1713813708 last 1713813642 [3549617.790016] Lustre: Skipped 1 previous similar message [3549809.756698] Lustre: oak-MDT0000: haven't heard from client 13753edd-d643-4503-9b5b-f62850454df4 (at 10.51.2.65@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900dc49a6c00, cur 1713814050 expire 1713813900 last 1713813823 [3549809.778577] Lustre: Skipped 1 previous similar message [3549822.756393] Lustre: oak-MDT0003: haven't heard from client 13753edd-d643-4503-9b5b-f62850454df4 (at 10.51.2.65@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901ec9619800, cur 1713814063 expire 1713813913 last 1713813836 [3549885.754865] Lustre: oak-MDT0000: haven't heard from client edaa9c37-4ebc-4f7b-b7c1-0dfc3a4876c4 (at 10.51.8.9@o2ib3) in 195 seconds. I think it's dead, and I am evicting it. exp ffff901f7941e400, cur 1713814126 expire 1713813976 last 1713813931 [3549898.758191] Lustre: oak-MDT0003: haven't heard from client edaa9c37-4ebc-4f7b-b7c1-0dfc3a4876c4 (at 10.51.8.9@o2ib3) in 218 seconds. I think it's dead, and I am evicting it. exp ffff901f7941c400, cur 1713814139 expire 1713813989 last 1713813921 [3549919.156158] LustreError: 36697:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59926117672 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59926117672 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3549919.156160] LustreError: 36688:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107948231180 granted:483067692, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3549919.156162] LustreError: 36688:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 136 previous similar messages [3550522.054451] LustreError: 36662:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3550522.084124] LustreError: 36662:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 128 previous similar messages [3551134.985960] LustreError: 35199:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3551135.015634] LustreError: 35199:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 109 previous similar messages [3551735.029980] LustreError: 36666:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3551735.059652] LustreError: 36666:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 122 previous similar messages [3552335.308135] LustreError: 35151:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3552335.337815] LustreError: 35151:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 153 previous similar messages [3552932.607657] Lustre: oak-MDT0003: haven't heard from client 2b24e473-73b7-4e0e-8699-845d8599f609 (at 10.51.4.64@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9032c8b28800, cur 1713817173 expire 1713817023 last 1713816946 [3552936.139690] LustreError: 35204:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536472168 granted:559613176, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3552936.170232] LustreError: 35204:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 164 previous similar messages [3553537.268301] LustreError: 35158:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107962680764 granted:486160868, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3553537.298850] LustreError: 35158:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 139 previous similar messages [3553811.566655] Lustre: oak-MDT0003: haven't heard from client 9e2ff39e-bd3e-4393-a624-53b6a8c8deee (at 10.51.15.11@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903b8b477000, cur 1713818052 expire 1713817902 last 1713817825 [3553811.588609] Lustre: Skipped 7 previous similar messages [3554058.552055] Lustre: oak-MDT0003: haven't heard from client 261b3b74-47f0-4bb3-95d6-8dc3f2238f95 (at 10.51.4.22@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903fe4ef0000, cur 1713818299 expire 1713818149 last 1713818072 [3554058.573933] Lustre: Skipped 1 previous similar message [3554061.553896] Lustre: oak-MDT0000: haven't heard from client 261b3b74-47f0-4bb3-95d6-8dc3f2238f95 (at 10.51.4.22@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902bf9e21c00, cur 1713818302 expire 1713818152 last 1713818075 [3554142.730540] LustreError: 36696:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59926117672 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59926117672 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3554142.760561] LustreError: 36696:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 115 previous similar messages [3554774.332226] LustreError: 35143:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536472164 granted:559613176, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3554774.362769] LustreError: 35143:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 73 previous similar messages [3554904.510072] Lustre: oak-MDT0003: haven't heard from client 3b8b3c48-8e2f-4440-9fab-5cb0507f29f1 (at 10.50.9.46@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901d5ad83800, cur 1713819145 expire 1713818995 last 1713818918 [3555378.702116] LustreError: 35153:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3555378.732090] LustreError: 35153:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 78 previous similar messages [3555647.476270] Lustre: oak-MDT0003: haven't heard from client 964a238a-fdaf-4816-9af0-90e81804ba61 (at 10.51.11.11@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902fe5d56400, cur 1713819888 expire 1713819738 last 1713819661 [3555647.498236] Lustre: Skipped 1 previous similar message [3555986.079668] LustreError: 35172:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107933066300 granted:424654352, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3555986.110203] LustreError: 35172:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 75 previous similar messages [3556149.450654] Lustre: oak-MDT0000: haven't heard from client b6acbec5-2ee1-4e1a-9b93-9c97e7dd6981 (at 10.51.8.51@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900613eb8000, cur 1713820390 expire 1713820240 last 1713820163 [3556149.472530] Lustre: Skipped 1 previous similar message [3556586.125968] LustreError: 35227:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107933066300 granted:424654352, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3556586.156506] LustreError: 35227:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 90 previous similar messages [3557188.295218] LustreError: 36660:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3557188.325398] LustreError: 36660:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 80 previous similar messages [3557793.678954] LustreError: 35205:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852132284 granted:310379184, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3557793.709491] LustreError: 35205:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 84 previous similar messages [3558396.309353] LustreError: 35153:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3558396.339204] LustreError: 35153:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 82 previous similar messages [3558996.975393] LustreError: 36680:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59926117672 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59926117672 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3558996.975395] LustreError: 36700:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107948189376 granted:483067692, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3558996.975398] LustreError: 36700:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 83 previous similar messages [3559452.293629] Lustre: oak-MDT0000: haven't heard from client 41600002-cb7c-4840-a1a9-fa00eba9cf05 (at 10.51.2.6@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902c8920b400, cur 1713823693 expire 1713823543 last 1713823466 [3559452.315417] Lustre: Skipped 1 previous similar message [3559615.686155] LustreError: 35156:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3559615.716006] LustreError: 35156:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 78 previous similar messages [3559799.275337] Lustre: oak-MDT0000: haven't heard from client be21a7e0-141d-47cd-91d7-b1f091eebd9b (at 10.51.6.28@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903e09040c00, cur 1713824040 expire 1713823890 last 1713823813 [3559799.297216] Lustre: Skipped 23 previous similar messages [3560215.700814] LustreError: 36671:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3560215.730661] LustreError: 36671:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 84 previous similar messages [3560333.253136] Lustre: oak-MDT0000: haven't heard from client ad35675f-2594-4a24-b874-66b9f3746747 (at 10.51.11.11@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9025e3cb3000, cur 1713824574 expire 1713824424 last 1713824347 [3560333.275115] Lustre: Skipped 1 previous similar message [3560643.237290] Lustre: oak-MDT0000: haven't heard from client 2abce909-074e-443b-82f3-cac65d16fc6d (at 10.51.8.37@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900e3a510800, cur 1713824884 expire 1713824734 last 1713824657 [3560643.259167] Lustre: Skipped 1 previous similar message [3560824.063849] LustreError: 35223:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107933059748 granted:424654352, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3560824.094556] LustreError: 35223:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 84 previous similar messages [3561424.840732] LustreError: 36681:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107933059748 granted:424654352, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3561424.871404] LustreError: 36681:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 78 previous similar messages [3562047.303594] LustreError: 31510:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3562047.333268] LustreError: 31510:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 79 previous similar messages [3562352.156675] Lustre: oak-MDT0000: haven't heard from client b9a084a4-06e3-4af9-8515-866a3a5739ba (at 10.51.12.6@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9031b5520000, cur 1713826593 expire 1713826443 last 1713826366 [3562352.178547] Lustre: Skipped 1 previous similar message [3562428.150556] Lustre: oak-MDT0003: haven't heard from client d1923acb-43fb-4748-9da1-76c4b7dccf2b (at 10.51.15.2@o2ib3) in 158 seconds. I think it's dead, and I am evicting it. exp ffff901cd39f9c00, cur 1713826669 expire 1713826519 last 1713826511 [3562428.172432] Lustre: Skipped 1 previous similar message [3562649.639050] LustreError: 35178:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3562649.668900] LustreError: 35178:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 79 previous similar messages [3563098.122480] Lustre: oak-MDT0000: haven't heard from client cb7a0675-376d-477b-a366-98719ed1a7ba (at 10.50.7.65@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900c3799c000, cur 1713827339 expire 1713827189 last 1713827112 [3563098.144359] Lustre: Skipped 1 previous similar message [3563249.649123] LustreError: 36655:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3563249.678970] LustreError: 36655:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 87 previous similar messages [3563589.098139] Lustre: oak-MDT0000: haven't heard from client 665ee7d9-d732-4127-a85b-62d6b09b3de5 (at 10.51.17.18@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9002ad705800, cur 1713827830 expire 1713827680 last 1713827603 [3563589.120099] Lustre: Skipped 1 previous similar message [3563861.383970] LustreError: 35226:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107948230844 granted:483067692, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3563861.414510] LustreError: 35226:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 134 previous similar messages [3564288.062487] Lustre: oak-MDT0000: haven't heard from client 134b1362-f14a-4417-b147-557fb6149bc4 (at 10.50.9.45@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903a476cbc00, cur 1713828529 expire 1713828379 last 1713828302 [3564288.084538] Lustre: Skipped 1 previous similar message [3564467.491295] LustreError: 36690:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107948230844 granted:483067692, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3564467.491296] LustreError: 35162:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59926117672 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59926117672 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3564467.491299] LustreError: 35162:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 80 previous similar messages [3564945.031839] Lustre: oak-MDT0000: haven't heard from client faae43f3-aa6f-4d8c-8a0f-0380751e2b11 (at 10.51.3.54@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90147cd07400, cur 1713829186 expire 1713829036 last 1713828959 [3564945.053713] Lustre: Skipped 1 previous similar message [3565072.491306] LustreError: 36660:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107962682248 granted:486160868, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3565072.521842] LustreError: 36660:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 136 previous similar messages [3565088.008994] LustreError: 44813:0:(mdt_open.c:1217:mdt_cross_open()) oak-MDT0003: [0x32c0020b7d:0xe139:0x0] doesn't exist!: rc = -14 [3565360.011895] Lustre: oak-MDT0000: haven't heard from client 497dd0c3-c320-4b7a-b441-f264a51a414a (at 10.50.6.25@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9020cd29b000, cur 1713829601 expire 1713829451 last 1713829374 [3565360.033778] Lustre: Skipped 5 previous similar messages [3565606.008032] Lustre: oak-MDT0000: haven't heard from client 4591a2bf-358a-4d8c-ae67-1a7feadd57a3 (at 10.51.6.40@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903be479a800, cur 1713829847 expire 1713829697 last 1713829620 [3565606.029911] Lustre: Skipped 1 previous similar message [3565675.145476] LustreError: 36657:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3565675.175150] LustreError: 36657:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 175 previous similar messages [3566285.495415] LustreError: 35183:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3566285.525261] LustreError: 35183:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 134 previous similar messages [3566886.451697] LustreError: 35148:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107814850800 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3566886.481380] LustreError: 35148:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 92 previous similar messages [3567487.116222] LustreError: 35192:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107814850800 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3567487.145900] LustreError: 35192:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 80 previous similar messages [3567606.907968] Lustre: oak-MDT0000: haven't heard from client ab6b23cf-61f0-4bf5-ba2d-5c208183ccf7 (at 10.51.1.25@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9006a0d3e400, cur 1713831848 expire 1713831698 last 1713831621 [3567606.929842] Lustre: Skipped 1 previous similar message [3567682.907338] Lustre: oak-MDT0000: haven't heard from client a059f719-0c9f-4913-affe-1045347d075a (at 10.51.15.5@o2ib3) in 221 seconds. I think it's dead, and I am evicting it. exp ffff902655a7ec00, cur 1713831924 expire 1713831774 last 1713831703 [3567682.929220] Lustre: Skipped 1 previous similar message [3567693.905509] Lustre: oak-MDT0003: haven't heard from client a059f719-0c9f-4913-affe-1045347d075a (at 10.51.15.5@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902f620f4000, cur 1713831935 expire 1713831785 last 1713831708 [3568087.249391] LustreError: 35184:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107948145936 granted:483067692, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3568087.279928] LustreError: 35184:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 79 previous similar messages [3568613.862906] Lustre: oak-MDT0003: haven't heard from client 130ec399-cb41-4381-a311-f66b049c2b82 (at 10.50.7.65@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902cb71aa400, cur 1713832855 expire 1713832705 last 1713832628 [3568692.762918] LustreError: 35158:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107814850800 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3568692.792593] LustreError: 35158:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 93 previous similar messages [3569299.072705] LustreError: 35221:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107948147152 granted:483067692, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3569299.103241] LustreError: 35221:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 111 previous similar messages [3569324.839316] Lustre: oak-MDT0000: haven't heard from client 8e83a51d-af97-4dfa-bcce-d91264ca8d1d (at 10.51.11.19@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903adab61000, cur 1713833566 expire 1713833416 last 1713833339 [3569324.861277] Lustre: Skipped 3 previous similar messages [3569337.846771] Lustre: oak-MDT0003: haven't heard from client 8e83a51d-af97-4dfa-bcce-d91264ca8d1d (at 10.51.11.19@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901ccc9a3400, cur 1713833579 expire 1713833429 last 1713833352 [3569904.171648] LustreError: 35147:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3569904.201324] LustreError: 35147:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 164 previous similar messages [3570507.486770] LustreError: 35182:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536490896 granted:559613176, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3570507.517314] LustreError: 35182:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 187 previous similar messages [3571118.304228] LustreError: 36653:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852099076 granted:310379184, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3571118.334769] LustreError: 36653:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 206 previous similar messages [3571310.738245] Lustre: oak-MDT0000: haven't heard from client 4addce28-908e-4e61-a9e1-811f01e331ac (at 10.51.4.18@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903302c46c00, cur 1713835552 expire 1713835402 last 1713835325 [3571745.824571] LustreError: 35157:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852079592 granted:310379184, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3571745.855112] LustreError: 35157:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 169 previous similar messages [3572347.487014] LustreError: 35208:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3572347.516866] LustreError: 35208:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 83 previous similar messages [3572948.983376] LustreError: 35200:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536466528 granted:559613176, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3572949.013916] LustreError: 35200:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 83 previous similar messages [3573334.647637] Lustre: oak-MDT0003: haven't heard from client a05aeb2a-2467-4ea4-b33a-5bc4cf5e5f73 (at 10.51.8.71@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900e6af8c000, cur 1713837576 expire 1713837426 last 1713837349 [3573334.669511] Lustre: Skipped 1 previous similar message [3573549.030881] LustreError: 36665:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536466528 granted:559613176, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3573549.061423] LustreError: 36665:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 84 previous similar messages [3574167.797371] LustreError: 35153:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852079148 granted:310379184, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3574167.827914] LustreError: 35153:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 122 previous similar messages [3574706.584586] Lustre: oak-MDT0000: haven't heard from client 5486530c-afeb-484d-8a5e-3e70e9d5332c (at 10.51.6.68@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90220a347c00, cur 1713838948 expire 1713838798 last 1713838721 [3574706.606463] Lustre: Skipped 9 previous similar messages [3574768.840236] LustreError: 35165:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107948122860 granted:483067692, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3574768.870774] LustreError: 35165:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 138 previous similar messages [3575372.352952] LustreError: 36654:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852075936 granted:310379184, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3575372.383494] LustreError: 36654:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 85 previous similar messages [3575976.480433] LustreError: 36656:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3575976.510107] LustreError: 36656:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 90 previous similar messages [3576251.512159] Lustre: oak-MDT0000: haven't heard from client 55837f25-181a-4fa9-89e5-81e787379bd6 (at 10.50.16.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902acd2ac400, cur 1713840493 expire 1713840343 last 1713840266 [3576251.534035] Lustre: Skipped 1 previous similar message [3576576.540800] LustreError: 35148:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536463704 granted:559613176, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3576576.571343] LustreError: 35148:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 78 previous similar messages [3577187.767673] LustreError: 36659:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536462952 granted:559613176, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3577187.798212] LustreError: 36659:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 85 previous similar messages [3577790.458836] LustreError: 35149:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536466528 granted:559613176, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3577790.489379] LustreError: 35149:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 131 previous similar messages [3578402.420342] LustreError: 35195:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852079580 granted:310379184, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3578402.450889] LustreError: 35195:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 80 previous similar messages [3579031.146948] LustreError: 35144:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3579031.176613] LustreError: 35144:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 83 previous similar messages [3579645.977274] LustreError: 35207:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536467840 granted:559613176, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3579646.007827] LustreError: 35207:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 94 previous similar messages [3580190.328869] Lustre: oak-MDT0003: haven't heard from client adcac6e8-6599-4b27-af4f-b8bc692ec7a2 (at 10.51.10.48@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9037fd5aec00, cur 1713844432 expire 1713844282 last 1713844205 [3580190.350838] Lustre: Skipped 1 previous similar message [3580245.991327] LustreError: 35156:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536467840 granted:559613176, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3580246.021869] LustreError: 35156:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 188 previous similar messages [3580846.003242] LustreError: 35157:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536467840 granted:559613176, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3580846.033778] LustreError: 35157:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 83 previous similar messages [3581446.050419] LustreError: 35152:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536467840 granted:559613176, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3581446.080970] LustreError: 35152:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 77 previous similar messages [3582052.258434] LustreError: 31510:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3582052.288110] LustreError: 31510:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 76 previous similar messages [3582653.477758] LustreError: 35192:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852092024 granted:310379184, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3582653.508297] LustreError: 35192:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 86 previous similar messages [3583254.623820] LustreError: 36662:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3583254.653680] LustreError: 36662:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 78 previous similar messages [3583854.635056] LustreError: 36664:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3583854.664908] LustreError: 36664:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 79 previous similar messages [3584461.280367] LustreError: 36697:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107948130792 granted:483067692, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3584461.310906] LustreError: 36697:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 84 previous similar messages [3585068.617184] LustreError: 35167:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59926117672 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59926117672 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3585068.647202] LustreError: 35167:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 81 previous similar messages [3585256.093863] Lustre: oak-MDT0000: haven't heard from client aa3f96fd-377d-4fca-bfbe-5b877becb8ce (at 10.51.6.57@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900b3e80bc00, cur 1713849498 expire 1713849348 last 1713849271 [3585256.115745] Lustre: Skipped 1 previous similar message [3585266.102070] Lustre: oak-MDT0003: haven't heard from client aa3f96fd-377d-4fca-bfbe-5b877becb8ce (at 10.51.6.57@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901f52bbdc00, cur 1713849508 expire 1713849358 last 1713849281 [3585544.079473] Lustre: oak-MDT0000: haven't heard from client 8e4a757e-2b2c-4ac0-9f62-3ca04c89a643 (at 10.51.6.25@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901ccc83fc00, cur 1713849786 expire 1713849636 last 1713849559 [3585680.152042] LustreError: 35154:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3585680.181897] LustreError: 35154:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 78 previous similar messages [3585933.062182] Lustre: oak-MDT0000: haven't heard from client 594be47c-c093-4f61-a3b3-cdceac992565 (at 10.51.5.26@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901e0f9fec00, cur 1713850175 expire 1713850025 last 1713849948 [3585933.084055] Lustre: Skipped 1 previous similar message [3586087.260681] LustreError: 44845:0:(mdt_open.c:1217:mdt_cross_open()) oak-MDT0003: [0x32c0020bbb:0x1ce2d:0x0] doesn't exist!: rc = -14 [3586141.051556] Lustre: oak-MDT0000: haven't heard from client 724e0879-e4b7-420c-9e5f-8973dc6703ba (at 10.51.15.10@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903d27078c00, cur 1713850383 expire 1713850233 last 1713850156 [3586141.073520] Lustre: Skipped 1 previous similar message [3586298.686110] LustreError: 36671:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536467840 granted:559613176, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3586298.716652] LustreError: 36671:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 379 previous similar messages [3586333.043907] Lustre: oak-MDT0000: haven't heard from client 260b80c5-d07c-41c3-8e8d-48dff919aee3 (at 10.51.5.37@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903b88c26400, cur 1713850575 expire 1713850425 last 1713850348 [3586333.065784] Lustre: Skipped 1 previous similar message [3586472.973975] LustreError: 44880:0:(mdt_open.c:1217:mdt_cross_open()) oak-MDT0003: [0x32c0020cbd:0x2aea:0x0] doesn't exist!: rc = -14 [3586914.497285] LustreError: 35178:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107962621232 granted:486160868, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3586914.527828] LustreError: 35178:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 172 previous similar messages [3587520.930595] LustreError: 35144:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3587520.960441] LustreError: 35144:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 216 previous similar messages [3588132.718811] LustreError: 36654:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3588132.748658] LustreError: 36654:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 77 previous similar messages [3588461.599987] LustreError: 36605:0:(mdt_open.c:1217:mdt_cross_open()) oak-MDT0003: [0x32c0020cbe:0x5609:0x0] doesn't exist!: rc = -14 [3588685.933991] Lustre: oak-MDT0000: haven't heard from client 5b64267b-db0f-4479-92ea-e821a79b7f23 (at 10.51.18.9@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90305611e800, cur 1713852928 expire 1713852778 last 1713852701 [3588685.955870] Lustre: Skipped 1 previous similar message [3588749.342979] LustreError: 36658:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852092024 granted:310379184, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3588749.373521] LustreError: 36658:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 79 previous similar messages [3589133.805012] LustreError: 44898:0:(mdt_open.c:1217:mdt_cross_open()) oak-MDT0003: [0x32c0020cbe:0xfdd3:0x0] doesn't exist!: rc = -14 [3589351.874559] LustreError: 35154:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3589351.904412] LustreError: 35154:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 82 previous similar messages [3589956.404897] LustreError: 36683:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107948130792 granted:483067692, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3589956.435440] LustreError: 36683:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 72 previous similar messages [3590343.862484] Lustre: oak-MDT0003: haven't heard from client 1ca74f7e-948b-4f98-918d-abb2d5d92879 (at 10.51.14.22@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9031925df800, cur 1713854586 expire 1713854436 last 1713854359 [3590343.884444] Lustre: Skipped 1 previous similar message [3590409.876972] LustreError: 44940:0:(mdt_open.c:1217:mdt_cross_open()) oak-MDT0003: [0x32c0020cbf:0x4936:0x0] doesn't exist!: rc = -14 [3590409.889052] LustreError: 44940:0:(mdt_open.c:1217:mdt_cross_open()) Skipped 2 previous similar messages [3590572.117857] LustreError: 35206:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3590572.147528] LustreError: 35206:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 71 previous similar messages [3591177.828742] LustreError: 35192:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3591177.858590] LustreError: 35192:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 81 previous similar messages [3591779.064886] LustreError: 35203:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107962621232 granted:486160868, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3591779.095436] LustreError: 35203:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 75 previous similar messages [3592379.552258] LustreError: 35153:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3592379.582113] LustreError: 35153:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 76 previous similar messages [3592992.549889] LustreError: 35199:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3592992.579741] LustreError: 35199:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 84 previous similar messages [3593039.732623] Lustre: oak-MDT0003: haven't heard from client 2f08993e-0529-487b-9b45-7a285afbf93a (at 10.51.15.9@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9032e81e9c00, cur 1713857282 expire 1713857132 last 1713857055 [3593039.754497] Lustre: Skipped 1 previous similar message [3593607.859095] LustreError: 35180:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536467840 granted:559613176, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3593607.889630] LustreError: 35180:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 83 previous similar messages [3594227.031617] LustreError: 31509:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107814850800 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3594227.061299] LustreError: 31509:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 80 previous similar messages [3594307.853115] LustreError: 31502:0:(mdt_open.c:1217:mdt_cross_open()) oak-MDT0003: [0x32c0020cc3:0x80e7:0x0] doesn't exist!: rc = -14 [3594307.865107] LustreError: 31502:0:(mdt_open.c:1217:mdt_cross_open()) Skipped 2 previous similar messages [3594783.041270] LustreError: 44813:0:(mdt_open.c:1217:mdt_cross_open()) oak-MDT0003: [0x32c0020cc3:0xf61d:0x0] doesn't exist!: rc = -14 [3594834.316411] LustreError: 36667:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3594834.346399] LustreError: 36667:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 94 previous similar messages [3595441.173695] LustreError: 35206:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852092024 granted:310379184, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3595441.204240] LustreError: 35206:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 69 previous similar messages [3595932.604140] Lustre: oak-MDT0003: haven't heard from client b734d9f5-b62c-41e5-94e7-39b8e88b5ade (at 10.51.8.55@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901f0c523800, cur 1713860175 expire 1713860025 last 1713859948 [3595932.626019] Lustre: Skipped 1 previous similar message [3596058.182256] LustreError: 35171:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59926117672 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59926117672 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3596058.212282] LustreError: 35171:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 76 previous similar messages [3596661.426456] LustreError: 36662:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852092024 granted:310379184, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3596661.456998] LustreError: 36662:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 76 previous similar messages [3597278.616767] LustreError: 36658:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852091780 granted:310379184, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3597278.647305] LustreError: 36658:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 96 previous similar messages [3597879.359496] LustreError: 35176:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536467800 granted:559613176, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3597879.390041] LustreError: 35176:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 80 previous similar messages [3598133.499645] Lustre: oak-MDT0000: haven't heard from client 0cba3ed0-7c9f-498b-b8a3-0308c1536828 (at 10.51.15.6@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901dfbf87c00, cur 1713862376 expire 1713862226 last 1713862149 [3598133.521523] Lustre: Skipped 1 previous similar message [3598138.505405] Lustre: oak-MDT0003: haven't heard from client 0cba3ed0-7c9f-498b-b8a3-0308c1536828 (at 10.51.15.6@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9003600f0400, cur 1713862381 expire 1713862231 last 1713862154 [3598498.080891] LustreError: 35165:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107933075844 granted:424654352, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3598498.111426] LustreError: 35165:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 79 previous similar messages [3598710.834803] LustreError: 44951:0:(mdt_open.c:1217:mdt_cross_open()) oak-MDT0003: [0x32c0020cc5:0x14127:0x0] doesn't exist!: rc = -14 [3599100.087791] LustreError: 35156:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3599100.117636] LustreError: 35156:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 85 previous similar messages [3599716.711676] LustreError: 35149:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107962621232 granted:486160868, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3599716.742218] LustreError: 35149:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 84 previous similar messages [3600318.861235] LustreError: 35178:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852091780 granted:310379184, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3600318.891778] LustreError: 35178:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 88 previous similar messages [3600387.401444] Lustre: oak-MDT0000: haven't heard from client 16846db7-72db-44f3-b0aa-7e8d8ea368cf (at 10.51.15.4@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900909d09800, cur 1713864630 expire 1713864480 last 1713864403 [3600406.397651] Lustre: oak-MDT0003: haven't heard from client 16846db7-72db-44f3-b0aa-7e8d8ea368cf (at 10.51.15.4@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901993d67400, cur 1713864649 expire 1713864499 last 1713864422 [3600920.951035] LustreError: 35185:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107948130764 granted:483067692, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3600920.981693] LustreError: 35185:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 80 previous similar messages [3601523.775310] LustreError: 36650:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3601523.805161] LustreError: 36650:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 96 previous similar messages [3601926.325868] Lustre: oak-MDT0003: haven't heard from client 30c2b9e2-56c1-4542-be8c-c5f41fea7097 (at 10.50.10.1@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900516be2400, cur 1713866169 expire 1713866019 last 1713865942 [3602130.204708] LustreError: 35145:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107814850800 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3602130.234391] LustreError: 35145:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 82 previous similar messages [3602621.295255] Lustre: oak-MDT0000: haven't heard from client 0e6f4ba3-b7a2-47eb-bdf3-e6467d2368ad (at 10.51.1.30@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9020fa063800, cur 1713866864 expire 1713866714 last 1713866637 [3602621.317302] Lustre: Skipped 1 previous similar message [3602730.250380] LustreError: 35192:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107962621232 granted:486160868, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3602730.280920] LustreError: 35192:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 85 previous similar messages [3603330.297923] LustreError: 35203:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107962621232 granted:486160868, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3603330.328470] LustreError: 35203:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 82 previous similar messages [3603931.161592] LustreError: 35179:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536467792 granted:559613176, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3603931.192140] LustreError: 35179:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 80 previous similar messages [3604533.456073] LustreError: 35155:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3604533.485916] LustreError: 35155:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 84 previous similar messages [3605142.186465] Lustre: oak-MDT0000: haven't heard from client ae99dc53-4735-40b7-9bbb-70f42c9e99f3 (at 10.51.1.12@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9020adae5000, cur 1713869385 expire 1713869235 last 1713869158 [3605142.208339] Lustre: Skipped 11 previous similar messages [3605145.195394] LustreError: 35176:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852091780 granted:310379184, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3605145.225937] LustreError: 35176:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 87 previous similar messages [3605745.243281] LustreError: 35199:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852091780 granted:310379184, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3605745.273829] LustreError: 35199:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 85 previous similar messages [3606364.263112] LustreError: 35143:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536467792 granted:559613176, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3606364.293658] LustreError: 35143:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 87 previous similar messages [3606964.310872] LustreError: 35200:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536467792 granted:559613176, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3606964.341413] LustreError: 35200:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 83 previous similar messages [3607573.490229] LustreError: 35152:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536467792 granted:559613176, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3607573.520769] LustreError: 35152:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 84 previous similar messages [3608173.546027] LustreError: 36671:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536467792 granted:559613176, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3608173.576564] LustreError: 36671:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 79 previous similar messages [3608788.887346] LustreError: 36665:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3608788.917195] LustreError: 36665:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 81 previous similar messages [3609389.164980] LustreError: 35221:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107933075844 granted:424654352, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3609389.195521] LustreError: 35221:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 68 previous similar messages [3610005.790119] LustreError: 35220:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59926117672 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59926117672 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3610005.820137] LustreError: 35220:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 74 previous similar messages [3610606.173316] LustreError: 35175:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107933075844 granted:424654352, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3610606.203857] LustreError: 35175:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 79 previous similar messages [3611209.369238] LustreError: 35199:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852091780 granted:310379184, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3611209.399932] LustreError: 35199:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 77 previous similar messages [3611791.879611] Lustre: oak-MDT0000: haven't heard from client 5a9ee06f-3684-4644-9d17-38838f9bd4ce (at 10.50.8.50@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902f6233bc00, cur 1713876035 expire 1713875885 last 1713875808 [3611791.901502] Lustre: Skipped 3 previous similar messages [3611827.203656] LustreError: 36697:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107948130764 granted:483067692, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3611827.234191] LustreError: 36697:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 77 previous similar messages [3612428.031350] LustreError: 31509:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107962621232 granted:486160868, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3612428.061895] LustreError: 31509:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 80 previous similar messages [3613046.886323] LustreError: 35218:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107933075844 granted:424654352, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3613046.916865] LustreError: 35218:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 78 previous similar messages [3613651.994592] LustreError: 35218:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107933075844 granted:424654352, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3613652.025135] LustreError: 35218:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 81 previous similar messages [3614252.043380] LustreError: 35219:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107933075844 granted:424654352, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3614252.074051] LustreError: 35219:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 80 previous similar messages [3614852.091025] LustreError: 35161:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107933075844 granted:424654352, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3614852.121565] LustreError: 35161:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 80 previous similar messages [3615459.363844] LustreError: 36670:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3615459.393516] LustreError: 36670:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 81 previous similar messages [3616059.858116] LustreError: 35227:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107933075844 granted:424654352, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3616059.888650] LustreError: 35227:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 85 previous similar messages [3616660.066208] LustreError: 36654:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852091780 granted:310379184, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3616660.096752] LustreError: 36654:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 84 previous similar messages [3617270.077575] LustreError: 36657:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536467792 granted:559613176, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3617270.108115] LustreError: 36657:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 85 previous similar messages [3617873.508071] LustreError: 36665:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107814850800 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3617873.537751] LustreError: 36665:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 84 previous similar messages [3618476.186666] LustreError: 36653:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536467792 granted:559613176, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3618476.217207] LustreError: 36653:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 85 previous similar messages [3619076.199502] LustreError: 35201:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536467792 granted:559613176, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3619076.230049] LustreError: 35201:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 82 previous similar messages [3619679.980150] LustreError: 35157:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3619680.009996] LustreError: 35157:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 83 previous similar messages [3620282.803411] LustreError: 35156:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3620282.833088] LustreError: 35156:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 86 previous similar messages [3620896.344696] LustreError: 35184:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59926117672 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59926117672 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3620896.374719] LustreError: 35184:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 88 previous similar messages [3621496.939105] LustreError: 35161:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107933075844 granted:424654352, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3621496.969735] LustreError: 35161:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 80 previous similar messages [3621899.419477] Lustre: oak-MDT0000: haven't heard from client 4fba7060-6538-46d0-a5c6-58c59f12b57c (at 10.50.16.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902b7d8ca400, cur 1713886143 expire 1713885993 last 1713885916 [3621899.441361] Lustre: Skipped 1 previous similar message [3622098.986364] LustreError: 36689:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59926117672 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59926117672 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3622099.016379] LustreError: 36689:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 83 previous similar messages [3622130.407874] Lustre: oak-MDT0000: haven't heard from client 601b85ac-6317-40ad-97bd-99ae9dfcb066 (at 10.50.9.45@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903b47664400, cur 1713886374 expire 1713886224 last 1713886147 [3622130.429750] Lustre: Skipped 15 previous similar messages [3622302.409689] Lustre: oak-MDT0000: haven't heard from client fb85c3e3-7219-4f2f-8220-bbecd31cc006 (at 10.51.5.26@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9012d7cc1000, cur 1713886546 expire 1713886396 last 1713886319 [3622302.431566] Lustre: Skipped 35 previous similar messages [3622698.997697] LustreError: 36681:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107948130764 granted:483067692, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3622699.028234] LustreError: 36681:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 86 previous similar messages [3623025.369903] Lustre: oak-MDT0000: haven't heard from client f93517b5-d0de-40bc-8509-61379a8d9ccc (at 10.51.1.55@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903c15ee3800, cur 1713887269 expire 1713887119 last 1713887042 [3623025.391949] Lustre: Skipped 1 previous similar message [3623299.010081] LustreError: 35227:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454530798048 granted:126893260, total:59926117672 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59926117672 time:0 qunit: 65536 edquot:0 may_rel:0 revoke:0 default:no [3623299.040101] LustreError: 35227:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 87 previous similar messages [3623489.342714] LNet: Dropping PUT from 12345-10.50.4.2@o2ib2 portal 26 match 1797065183425024 offset 0 length 520: 4 [3623489.342787] Lustre: oak-MDT0003: Client 70360d7e-0ee3-4ef2-928d-4a9efe3f0a2e (at 10.50.4.2@o2ib2) reconnecting [3623489.343028] LustreError: 137-5: oak-MDT0001_UUID: not available for connect from 10.50.4.2@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server. [3623489.343030] LustreError: Skipped 42 previous similar messages [3623489.386609] LNet: Skipped 21 previous similar messages [3623511.343820] Lustre: oak-MDT0000: haven't heard from client 2f50ddf1-7198-4313-b236-343886de1b11 (at 10.51.9.25@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9040fab92000, cur 1713887755 expire 1713887605 last 1713887528 [3623511.365702] Lustre: Skipped 1 previous similar message [3623598.564988] Lustre: oak-MDT0003: Client aaa0e444-1590-4e94-bc8a-88fc0d2c3614 (at 10.50.4.3@o2ib2) reconnecting [3623598.575168] Lustre: Skipped 2 previous similar messages [3623865.328246] Lustre: oak-MDT0000: haven't heard from client 0331c7b4-c244-49eb-8b5e-82509ad728b9 (at 10.51.10.61@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90070019c000, cur 1713888109 expire 1713887959 last 1713887882 [3623865.350205] Lustre: Skipped 5 previous similar messages [3623899.021571] LustreError: 36676:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107948130764 granted:483067692, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3623899.052121] LustreError: 36676:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 83 previous similar messages [3624488.298803] Lustre: oak-MDT0000: haven't heard from client de7294e2-1450-4370-b6c0-d221c90407e6 (at 10.51.13.6@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900e3a503400, cur 1713888732 expire 1713888582 last 1713888505 [3624488.320680] Lustre: Skipped 25 previous similar messages [3624503.224073] LustreError: 35199:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536467792 granted:559613176, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3624503.254614] LustreError: 35199:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 84 previous similar messages [3624670.290257] Lustre: oak-MDT0000: haven't heard from client cbba709e-45f8-49a0-a6be-0b1f14dfa343 (at 10.50.8.50@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902c25ca6400, cur 1713888914 expire 1713888764 last 1713888687 [3624670.312138] Lustre: Skipped 1 previous similar message [3625069.299832] Lustre: oak-MDT0003: haven't heard from client f5d80f7b-1181-4e0c-929a-e650bd31fa13 (at 10.50.9.47@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902c25d05c00, cur 1713889313 expire 1713889163 last 1713889086 [3625069.321707] Lustre: Skipped 3 previous similar messages [3625089.276376] Lustre: oak-MDT0000: haven't heard from client f5d80f7b-1181-4e0c-929a-e650bd31fa13 (at 10.50.9.47@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9039e0b58000, cur 1713889333 expire 1713889183 last 1713889106 [3625089.298255] Lustre: Skipped 2 previous similar messages [3625119.191223] LustreError: 36682:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107933075844 granted:424654352, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3625119.221765] LustreError: 36682:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 82 previous similar messages [3625145.270541] Lustre: oak-MDT0003: haven't heard from client a83c3e20-c8a6-40f5-8ad8-f7e7120153a8 (at 10.51.5.46@o2ib3) in 225 seconds. I think it's dead, and I am evicting it. exp ffff901190f50800, cur 1713889389 expire 1713889239 last 1713889164 [3625145.292420] Lustre: Skipped 6 previous similar messages [3625641.246227] Lustre: oak-MDT0000: haven't heard from client 874fe3b7-23e1-4c2f-85ac-f04b7f5a5af7 (at 10.51.14.13@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902092dc5000, cur 1713889885 expire 1713889735 last 1713889658 [3625641.268624] Lustre: Skipped 3 previous similar messages [3625719.239734] LustreError: 36693:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107933075844 granted:424654352, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3625719.270273] LustreError: 36693:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 86 previous similar messages [3626321.875286] LustreError: 35174:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107933075844 granted:424654352, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3626321.905820] LustreError: 35174:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 78 previous similar messages [3626368.212421] Lustre: oak-MDT0003: haven't heard from client 9c9ae746-4c1d-49d5-8d1c-dbaef264f966 (at 10.51.13.12@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9002eb4bc800, cur 1713890612 expire 1713890462 last 1713890385 [3626368.234385] Lustre: Skipped 1 previous similar message [3626380.211792] Lustre: oak-MDT0000: haven't heard from client 9c9ae746-4c1d-49d5-8d1c-dbaef264f966 (at 10.51.13.12@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9002fce1cc00, cur 1713890624 expire 1713890474 last 1713890397 [3626791.201458] Lustre: oak-MDT0000: haven't heard from client 6fa492f7-d4b3-4c8c-b8c6-842ce7226117 (at 10.50.8.20@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902928b53000, cur 1713891035 expire 1713890885 last 1713890808 [3626922.888967] LustreError: 35176:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536467792 granted:559613176, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3626922.919513] LustreError: 35176:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 89 previous similar messages [3627033.187250] Lustre: oak-MDT0000: haven't heard from client 87454275-fca5-4b27-a8ba-7852eda639ce (at 10.50.7.67@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9037ac31d000, cur 1713891277 expire 1713891127 last 1713891050 [3627033.209126] Lustre: Skipped 1 previous similar message [3627478.462776] LustreError: 33277:0:(qmt_lock.c:425:qmt_lvbo_update()) $$$ failed to release quota space on glimpse 0!=1929510965347609316 : rc = -22 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:59797960376 time:0 qunit: 262144 edquot:0 may_rel:0 revoke:0 default:no [3627478.490979] LustreError: 33277:0:(qmt_lock.c:425:qmt_lvbo_update()) Skipped 3 previous similar messages [3627522.901317] LustreError: 36656:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536467792 granted:559613176, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3627522.931855] LustreError: 36656:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 86 previous similar messages [3628122.913655] LustreError: 35197:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536467792 granted:559613176, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3628122.944199] LustreError: 35197:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 93 previous similar messages [3628735.124042] LustreError: 36688:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0209_UUID release: 16517233108345225224 granted:230227968, total:26693632680 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:26693632680 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3628735.154405] LustreError: 36688:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 95 previous similar messages [3629357.527891] LustreError: 36662:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536467792 granted:559613176, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3629357.558439] LustreError: 36662:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 96 previous similar messages [3629963.760114] LustreError: 31509:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852091780 granted:310379184, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3629963.790659] LustreError: 31509:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 94 previous similar messages [3630170.038622] Lustre: oak-MDT0000: haven't heard from client c11fb6e2-45e3-4523-9e75-30904e0c7e4b (at 10.50.1.14@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900e6830ac00, cur 1713894414 expire 1713894264 last 1713894187 [3630170.060499] Lustre: Skipped 1 previous similar message [3630573.218026] LustreError: 35205:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536467808 granted:559613176, total:353471851460 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353471851460 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3630573.248564] LustreError: 35205:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 115 previous similar messages [3630987.001838] Lustre: oak-MDT0000: haven't heard from client d10375b8-e0b8-4db5-ab83-893bcc20a9f2 (at 10.51.4.68@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903fe4ef1000, cur 1713895231 expire 1713895081 last 1713895004 [3630987.023723] Lustre: Skipped 3 previous similar messages [3631062.997418] Lustre: oak-MDT0000: haven't heard from client 7da13ece-13dc-4407-bb6f-c4ddade7263a (at 10.51.4.67@o2ib3) in 218 seconds. I think it's dead, and I am evicting it. exp ffff901dd1652400, cur 1713895307 expire 1713895157 last 1713895089 [3631063.019291] Lustre: Skipped 3 previous similar messages [3631182.172124] LustreError: 36662:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852091772 granted:310379184, total:353470802884 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353470802884 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3631182.202666] LustreError: 36662:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 105 previous similar messages [3631227.988047] Lustre: oak-MDT0000: haven't heard from client 92aa3ba5-6eb9-4ca8-9cda-bbeb29adbd5a (at 10.50.9.47@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903a84d74400, cur 1713895472 expire 1713895322 last 1713895245 [3631228.009919] Lustre: Skipped 1 previous similar message [3631295.382147] LustreError: 44867:0:(mdt_open.c:1217:mdt_cross_open()) oak-MDT0003: [0x32c0020ce0:0x127c1:0x0] doesn't exist!: rc = -14 [3631601.971981] Lustre: oak-MDT0003: haven't heard from client 4e12a15a-678e-4b08-8b7f-40e87e41765a (at 10.51.4.6@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903b7ce2d800, cur 1713895846 expire 1713895696 last 1713895619 [3631601.993771] Lustre: Skipped 1 previous similar message [3631782.187733] LustreError: 35200:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852091772 granted:310379184, total:353470802884 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353470802884 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3631782.218624] LustreError: 35200:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 92 previous similar messages [3632382.200466] LustreError: 36664:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852091772 granted:310379184, total:353470802884 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353470802884 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3632382.231010] LustreError: 36664:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 95 previous similar messages [3632983.448705] LustreError: 36685:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454514086368 granted:126893260, total:29568200696 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:29568200696 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3632983.448708] LustreError: 36676:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107948130764 granted:483067692, total:353470802884 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353470802884 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3632983.448712] LustreError: 36676:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 89 previous similar messages [3633451.887726] Lustre: oak-MDT0003: haven't heard from client 4a6a4585-5966-4bf9-9046-f40c9e9fa9f8 (at 10.51.13.10@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9020788d8400, cur 1713897696 expire 1713897546 last 1713897469 [3633451.909693] Lustre: Skipped 3 previous similar messages [3633463.887180] Lustre: oak-MDT0000: haven't heard from client 4a6a4585-5966-4bf9-9046-f40c9e9fa9f8 (at 10.51.13.10@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9002a7d26400, cur 1713897708 expire 1713897558 last 1713897481 [3633583.491357] LustreError: 36692:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454514086368 granted:126893260, total:29568200696 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:29568200696 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3633583.521645] LustreError: 36692:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 93 previous similar messages [3634191.733655] LustreError: 36686:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107933075832 granted:424654352, total:353470802884 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353470802884 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3634191.764287] LustreError: 36686:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 98 previous similar messages [3634793.550139] LustreError: 36690:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0209_UUID release: 16517233108345090036 granted:230227968, total:29852900736 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:29852900736 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3634793.580501] LustreError: 36690:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 186 previous similar messages [3635393.742080] LustreError: 35173:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454512525432 granted:126893260, total:29884360652 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:29884360652 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3635393.772351] LustreError: 35173:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 1398 previous similar messages [3635807.779701] Lustre: oak-MDT0000: haven't heard from client 6be664ea-2104-4156-8f53-c0b5ca525371 (at 10.51.10.30@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903fc0ddac00, cur 1713900052 expire 1713899902 last 1713899825 [3635994.436440] LustreError: 36695:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0209_UUID release: 16517233108337812836 granted:230227968, total:29900092632 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:29900092632 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3635994.466801] LustreError: 36695:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 1400 previous similar messages [3636218.762447] Lustre: oak-MDT0000: haven't heard from client 43e5b07b-e62b-4724-a948-5cfe523b225b (at 10.50.2.10@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903c581d2000, cur 1713900463 expire 1713900313 last 1713900236 [3636218.784320] Lustre: Skipped 1 previous similar message [3636595.908513] LustreError: 35168:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0209_UUID release: 16517233108335161492 granted:230227968, total:29915822956 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:29915822956 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3636595.938874] LustreError: 35168:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 2357 previous similar messages [3637125.717941] Lustre: oak-MDT0003: haven't heard from client 95756833-14d6-44a4-b7d8-f0d08ec331de (at 10.51.10.33@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901a8198f800, cur 1713901370 expire 1713901220 last 1713901143 [3637125.739900] Lustre: Skipped 1 previous similar message [3637196.233465] LustreError: 35218:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0209_UUID release: 16517233108333053964 granted:230227968, total:29931555820 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:29931555820 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3637196.263829] LustreError: 35218:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 705 previous similar messages [3637798.240878] LustreError: 36691:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0209_UUID release: 16517233108330081684 granted:230227968, total:29947288040 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:29947288040 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3637798.271238] LustreError: 36691:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 1046 previous similar messages [3638398.229447] LustreError: 35224:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0209_UUID release: 16517233108325838220 granted:230227968, total:29978748828 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:29978748828 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3638398.259814] LustreError: 35224:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 1152 previous similar messages [3638550.664913] Lustre: oak-MDT0003: haven't heard from client fe9ba85a-0e41-4510-adf7-21f482358b35 (at 10.50.1.36@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901ccc83b800, cur 1713902795 expire 1713902645 last 1713902568 [3638550.686783] Lustre: Skipped 1 previous similar message [3638998.848017] LustreError: 35169:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0209_UUID release: 16517233108325478068 granted:230227968, total:30023906232 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:30023906232 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3638998.878384] LustreError: 35169:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 182 previous similar messages [3639126.627942] Lustre: oak-MDT0003: haven't heard from client 254e69bb-1f3e-4ce6-8691-9af319269b6c (at 10.50.8.20@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901ab89be400, cur 1713903371 expire 1713903221 last 1713903144 [3639126.649814] Lustre: Skipped 1 previous similar message [3639336.619774] Lustre: oak-MDT0000: haven't heard from client c2f81594-4bb8-4c4e-b592-efcebf4fac45 (at 10.50.17.46@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902078800c00, cur 1713903581 expire 1713903431 last 1713903354 [3639336.641733] Lustre: Skipped 1 previous similar message [3639412.615037] Lustre: oak-MDT0000: haven't heard from client ffafaa92-31f2-4966-9e6b-185080665cc7 (at 10.50.4.70@o2ib2) in 180 seconds. I think it's dead, and I am evicting it. exp ffff902f7d8fa000, cur 1713903657 expire 1713903507 last 1713903477 [3639412.636918] Lustre: Skipped 1 previous similar message [3639488.611242] Lustre: oak-MDT0000: haven't heard from client 95b99525-4d36-42e3-a471-6ba5002e2d99 (at 10.51.10.36@o2ib3) in 206 seconds. I think it's dead, and I am evicting it. exp ffff90272f6c1000, cur 1713903733 expire 1713903583 last 1713903527 [3639488.633212] Lustre: Skipped 1 previous similar message [3639612.552080] LustreError: 36692:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454509505672 granted:126893260, total:30071094428 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:30071094428 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3639612.582365] LustreError: 36692:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 472 previous similar messages [3639962.589606] Lustre: oak-MDT0000: haven't heard from client a7e68144-f47b-4ea2-b903-013ab2823308 (at 10.50.1.18@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9006080d4000, cur 1713904207 expire 1713904057 last 1713903980 [3639962.611492] Lustre: Skipped 3 previous similar messages [3640219.224151] LustreError: 36664:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107962524728 granted:486160868, total:353470802884 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353470802884 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3640219.254692] LustreError: 36664:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 104 previous similar messages [3640821.225041] LustreError: 36655:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536445784 granted:559613176, total:353470802884 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353470802884 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3640821.255608] LustreError: 36655:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 121 previous similar messages [3640993.544495] Lustre: oak-MDT0000: haven't heard from client 3bbad175-e714-4d78-8f1e-a802bf9af8f4 (at 10.51.15.16@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900bb2306400, cur 1713905238 expire 1713905088 last 1713905011 [3640993.566460] Lustre: Skipped 7 previous similar messages [3641424.226676] LustreError: 35180:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107962523268 granted:486160868, total:353470802884 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353470802884 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3641424.257224] LustreError: 35180:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 264 previous similar messages [3642043.804499] LustreError: 31507:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454509505672 granted:126893260, total:30071094428 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:30071094428 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3642043.834777] LustreError: 31507:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 904 previous similar messages [3642632.471200] Lustre: oak-MDT0003: haven't heard from client e07ef152-306d-40f2-97e4-81ca400afb7b (at 10.51.15.16@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9002a7d37000, cur 1713906877 expire 1713906727 last 1713906650 [3642632.493278] Lustre: Skipped 1 previous similar message [3642648.151247] LustreError: 36654:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852062840 granted:310379184, total:353470802884 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353470802884 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3642648.181789] LustreError: 36654:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 249 previous similar messages [3643259.145798] LustreError: 35170:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0209_UUID release: 16517233108325478068 granted:230227968, total:30071094428 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:30071094428 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3643259.176166] LustreError: 35170:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 90 previous similar messages [3643808.413130] Lustre: oak-MDT0000: haven't heard from client c082640e-8632-4880-8c78-1e9896045a54 (at 10.50.4.32@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9031ebd80000, cur 1713908053 expire 1713907903 last 1713907826 [3643808.435007] Lustre: Skipped 3 previous similar messages [3643875.659753] LustreError: 35207:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852062840 granted:310379184, total:353462414276 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353462414276 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3643875.690293] LustreError: 35207:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 91 previous similar messages [3643972.405861] Lustre: oak-MDT0000: haven't heard from client 40febbdb-4087-4e45-8f47-d7849d7d284b (at 10.51.5.57@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903a297ab000, cur 1713908217 expire 1713908067 last 1713907990 [3643972.427742] Lustre: Skipped 1 previous similar message [3644401.385352] Lustre: oak-MDT0000: haven't heard from client 08970bd2-972d-4f82-9f42-03bfb624999c (at 10.51.8.15@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901190f4e000, cur 1713908646 expire 1713908496 last 1713908419 [3644401.407226] Lustre: Skipped 1 previous similar message [3644477.126893] LustreError: 36653:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536444148 granted:559613176, total:353044940452 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353044940452 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3644477.157438] LustreError: 36653:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 87 previous similar messages [3645089.113342] LustreError: 36698:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107933045852 granted:424654352, total:353044940452 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353044940452 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3645089.143876] LustreError: 36698:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 96 previous similar messages [3645689.134706] LustreError: 35218:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107933045852 granted:424654352, total:353044940452 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353044940452 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3645689.165247] LustreError: 35218:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 94 previous similar messages [3646289.146960] LustreError: 35225:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107933045852 granted:424654352, total:353044940452 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353044940452 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3646289.177499] LustreError: 35225:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 97 previous similar messages [3646907.853884] LustreError: 35181:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536444148 granted:559613176, total:353044940452 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353044940452 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3646907.884436] LustreError: 35181:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 91 previous similar messages [3647252.258312] Lustre: oak-MDT0000: haven't heard from client 672a5996-e788-45e9-99d7-86ddbddb0c18 (at 10.51.10.50@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9028c757f400, cur 1713911497 expire 1713911347 last 1713911270 [3647252.280271] Lustre: Skipped 1 previous similar message [3647507.901031] LustreError: 35202:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536444148 granted:559613176, total:353044940452 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353044940452 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3647507.931732] LustreError: 35202:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 99 previous similar messages [3648107.948175] LustreError: 35180:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536444148 granted:559613176, total:353043891876 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353043891876 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3648107.978715] LustreError: 35180:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 92 previous similar messages [3648582.192235] Lustre: oak-MDT0000: haven't heard from client 036a2797-efbc-42ac-975c-5849cd8bcbc7 (at 10.51.5.68@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9002a7d54c00, cur 1713912827 expire 1713912677 last 1713912600 [3648582.214118] Lustre: Skipped 1 previous similar message [3648725.830416] LustreError: 35152:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3648725.860264] LustreError: 35152:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 93 previous similar messages [3649042.171975] Lustre: oak-MDT0000: haven't heard from client 3ba7e842-11e1-4599-a3af-6bbe9e28ef5d (at 10.51.10.34@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903bf676f400, cur 1713913287 expire 1713913137 last 1713913060 [3649042.193943] Lustre: Skipped 1 previous similar message [3649339.531902] LustreError: 36665:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536444148 granted:559613176, total:353043891876 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353043891876 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3649339.562442] LustreError: 36665:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 97 previous similar messages [3649940.582949] LustreError: 36657:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852062848 granted:310379184, total:353043891876 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353043891876 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3649940.613524] LustreError: 36657:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 86 previous similar messages [3649992.127748] Lustre: oak-MDT0000: haven't heard from client 61c0d100-cc7a-41b7-8e4e-b4907a84d270 (at 10.51.1.42@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900d5492d800, cur 1713914237 expire 1713914087 last 1713914010 [3649992.149627] Lustre: Skipped 1 previous similar message [3650068.127545] Lustre: oak-MDT0000: haven't heard from client 04dfc940-7542-4ce4-8842-5e7c5693fd39 (at 10.51.15.4@o2ib3) in 160 seconds. I think it's dead, and I am evicting it. exp ffff901e29163800, cur 1713914313 expire 1713914163 last 1713914153 [3650068.149419] Lustre: Skipped 1 previous similar message [3650148.121110] Lustre: oak-MDT0003: haven't heard from client 04dfc940-7542-4ce4-8842-5e7c5693fd39 (at 10.51.15.4@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901e41af9000, cur 1713914393 expire 1713914243 last 1713914166 [3650541.838647] LustreError: 35185:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107933017404 granted:424654352, total:353043891876 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353043891876 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3650541.869182] LustreError: 35185:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 115 previous similar messages [3650739.093250] Lustre: oak-MDT0000: haven't heard from client 61c9660e-3082-4738-87ad-361a65edc53a (at 10.51.10.32@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902b82bc4800, cur 1713914984 expire 1713914834 last 1713914757 [3650815.089278] Lustre: oak-MDT0000: haven't heard from client 785f4b74-2862-473d-a860-a8511f2150ae (at 10.51.3.67@o2ib3) in 163 seconds. I think it's dead, and I am evicting it. exp ffff9028aab68c00, cur 1713915060 expire 1713914910 last 1713914897 [3650815.111149] Lustre: Skipped 1 previous similar message [3651144.210450] LustreError: 36666:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3651144.240307] LustreError: 36666:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 90 previous similar messages [3651758.931971] LustreError: 36690:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454509505672 granted:126893260, total:30071094428 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:30071094428 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3651758.962248] LustreError: 36690:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 104 previous similar messages [3652365.676980] LustreError: 36655:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107852032972 granted:310379184, total:353043891876 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353043891876 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3652365.707524] LustreError: 36655:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 111 previous similar messages [3652968.739096] LustreError: 35157:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107851974800 granted:310379184, total:353043891876 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353043891876 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3652968.769644] LustreError: 35157:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 245 previous similar messages [3653262.975068] Lustre: oak-MDT0000: haven't heard from client e72b132e-a2a3-4bb1-9ce9-2b996e63084e (at 10.50.1.44@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9018cf368c00, cur 1713917508 expire 1713917358 last 1713917281 [3653262.996945] Lustre: Skipped 1 previous similar message [3653570.143303] LustreError: 35177:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107851836084 granted:310379184, total:353043891876 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353043891876 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3653570.173845] LustreError: 35177:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 265 previous similar messages [3653602.958388] Lustre: oak-MDT0003: haven't heard from client 87f779f9-2c5c-479d-b0af-7121be904f2e (at 10.50.4.27@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901087571000, cur 1713917848 expire 1713917698 last 1713917621 [3653602.980269] Lustre: Skipped 1 previous similar message [3654171.324209] LustreError: 35177:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107961548436 granted:486160868, total:353043891876 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353043891876 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3654171.354753] LustreError: 35177:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 245 previous similar messages [3654775.252167] LustreError: 35197:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107851797620 granted:310379184, total:353043891876 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353043891876 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3654775.282712] LustreError: 35197:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 248 previous similar messages [3655000.894716] Lustre: oak-MDT0000: haven't heard from client 857add44-2576-4cf3-843a-0e288ab45736 (at 10.50.16.13@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9002fcef0c00, cur 1713919246 expire 1713919096 last 1713919019 [3655000.916677] Lustre: Skipped 1 previous similar message [3655380.578584] LustreError: 35208:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107814850800 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3655380.608266] LustreError: 35208:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 201 previous similar messages [3655494.871407] Lustre: oak-MDT0003: haven't heard from client e284fe8d-40c5-4e22-9fbd-083a18269c22 (at 10.51.9.67@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9024f69e4800, cur 1713919740 expire 1713919590 last 1713919513 [3655494.893277] Lustre: Skipped 1 previous similar message [3655988.180681] LustreError: 36669:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107814850800 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3655988.180682] LustreError: 31510:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107961548436 granted:486160868, total:353043891876 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353043891876 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3655988.180684] LustreError: 31510:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 106 previous similar messages [3656599.548414] LustreError: 35145:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107851777508 granted:310379184, total:353043891876 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353043891876 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3656599.578953] LustreError: 35145:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 131 previous similar messages [3657207.216176] LustreError: 36683:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107932997884 granted:424654352, total:353043891876 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353043891876 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3657207.246846] LustreError: 36683:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 111 previous similar messages [3657808.141759] LustreError: 36681:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107932993388 granted:424654352, total:353043891876 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353043891876 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3657808.172309] LustreError: 36681:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 155 previous similar messages [3657975.047012] LustreError: 32486:0:(mdt_open.c:1217:mdt_cross_open()) oak-MDT0003: [0x32c0020d1c:0x37f2:0x0] doesn't exist!: rc = -14 [3657976.309099] LustreError: 44921:0:(mdt_open.c:1217:mdt_cross_open()) oak-MDT0003: [0x32c0020d1c:0x3900:0x0] doesn't exist!: rc = -14 [3658410.725145] LustreError: 35195:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3658410.754997] LustreError: 35195:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 136 previous similar messages [3658974.707906] Lustre: oak-MDT0000: haven't heard from client c374ae92-bc39-44bf-900a-a3d6fa7f676a (at 10.51.10.28@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9031925d8800, cur 1713923220 expire 1713923070 last 1713922993 [3658974.729869] Lustre: Skipped 1 previous similar message [3659010.737203] LustreError: 35180:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3659010.767049] LustreError: 35180:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 94 previous similar messages [3659050.705290] Lustre: oak-MDT0000: haven't heard from client 52eec7ee-d963-49e9-ab25-e6f4d88f16e3 (at 10.51.15.19@o2ib3) in 195 seconds. I think it's dead, and I am evicting it. exp ffff902800483c00, cur 1713923296 expire 1713923146 last 1713923101 [3659050.727385] Lustre: Skipped 1 previous similar message [3659617.550742] LustreError: 36676:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0209_UUID release: 16517233108325478068 granted:230227968, total:30071094428 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:30071094428 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3659617.581100] LustreError: 36676:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 98 previous similar messages [3660245.738091] LustreError: 36693:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0209_UUID release: 16517233108325478068 granted:230227968, total:30071094428 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:30071094428 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3660245.768456] LustreError: 36693:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 111 previous similar messages [3660850.047402] LustreError: 36669:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107814850800 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3660850.077075] LustreError: 36669:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 84 previous similar messages [3661451.888903] LustreError: 36670:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3661451.918581] LustreError: 36670:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 117 previous similar messages [3662056.773452] LustreError: 36658:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107851767636 granted:310379184, total:352885733496 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352885733496 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3662056.804000] LustreError: 36658:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 206 previous similar messages [3662545.550746] Lustre: oak-MDT0003: haven't heard from client 4e03e317-4c71-46bc-920d-97860126ce9d (at 10.51.11.22@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9007196b1400, cur 1713926791 expire 1713926641 last 1713926564 [3662545.572706] Lustre: Skipped 1 previous similar message [3662556.541960] Lustre: oak-MDT0000: haven't heard from client 4e03e317-4c71-46bc-920d-97860126ce9d (at 10.51.11.22@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9007196a6400, cur 1713926802 expire 1713926652 last 1713926575 [3662556.563924] Lustre: Skipped 1 previous similar message [3662661.305596] LustreError: 36654:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107851767544 granted:310379184, total:352885733496 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352885733496 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3662661.336134] LustreError: 36654:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 244 previous similar messages [3662850.535092] Lustre: oak-MDT0000: haven't heard from client 581b4ece-03bc-44ed-9997-4447357db139 (at 10.50.6.30@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902c3b72d000, cur 1713927096 expire 1713926946 last 1713926869 [3662850.556966] Lustre: Skipped 1 previous similar message [3663286.422165] LustreError: 35222:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107932972812 granted:424654352, total:352885733496 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352885733496 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3663286.452701] LustreError: 35222:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 198 previous similar messages [3663889.700111] LustreError: 35195:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107961470976 granted:486160868, total:352885733496 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352885733496 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3663889.730661] LustreError: 35195:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 87 previous similar messages [3664494.482376] LustreError: 35177:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3664494.512056] LustreError: 35177:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 84 previous similar messages [3665109.182778] LustreError: 36664:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3665109.212453] LustreError: 36664:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 89 previous similar messages [3665710.718057] LustreError: 36656:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107961446368 granted:486160868, total:352885733496 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352885733496 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3665710.748607] LustreError: 36656:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 78 previous similar messages [3666312.788364] LustreError: 35207:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3666312.818034] LustreError: 35207:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 77 previous similar messages [3666913.380536] LustreError: 35207:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107851766932 granted:310379184, total:352885733496 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352885733496 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3666913.411090] LustreError: 35207:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 88 previous similar messages [3667515.079206] LustreError: 36683:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0209_UUID release: 16517233108325478068 granted:230227968, total:30071094428 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:30071094428 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3667515.109573] LustreError: 36683:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 103 previous similar messages [3668120.006641] LustreError: 35207:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3668120.036317] LustreError: 35207:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 82 previous similar messages [3668724.855569] LustreError: 35153:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3668724.885244] LustreError: 35153:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 79 previous similar messages [3669331.497480] LustreError: 35207:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3669331.527155] LustreError: 35207:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 80 previous similar messages [3669456.224026] Lustre: oak-MDT0000: haven't heard from client c2c8233a-8d88-4ea6-9cef-9395d57de55a (at 10.0.3.225@o2ib5) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9004e689fc00, cur 1713933702 expire 1713933552 last 1713933475 [3669456.245909] Lustre: Skipped 1 previous similar message [3669936.711360] LustreError: 35182:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069536271300 granted:559613176, total:352885733496 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352885733496 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3669936.742019] LustreError: 35182:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 86 previous similar messages [3670544.586227] LustreError: 35198:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3670544.615903] LustreError: 35198:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 84 previous similar messages [3670657.170671] Lustre: oak-MDT0000: haven't heard from client fc67995a-4574-43e5-a6a9-3a451066f336 (at 10.51.6.15@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90143eb4d800, cur 1713934903 expire 1713934753 last 1713934676 [3670657.192556] Lustre: Skipped 1 previous similar message [3670673.170280] Lustre: oak-MDT0003: haven't heard from client fc67995a-4574-43e5-a6a9-3a451066f336 (at 10.51.6.15@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9020fc30c400, cur 1713934919 expire 1713934769 last 1713934692 [3671148.630930] LustreError: 35224:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107932744000 granted:424654352, total:352885733496 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352885733496 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3671148.661720] LustreError: 35224:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 118 previous similar messages [3671189.578564] LNet: Tx -> 10.0.3.225@o2ib5 cookie 0x59ec5a22a sending 1 waiting 0: failed 13 [3671189.579238] LNet: Rx from 10.0.3.225@o2ib5 failed: 5 [3671189.579241] LNet: Closing conn to 10.0.3.225@o2ib5: error -5(waiting) [3671724.118149] Lustre: oak-MDT0003: haven't heard from client 7539cfe4-bab6-4ede-9a2f-9363b6c10ceb (at 10.51.10.35@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900b67cb2000, cur 1713935970 expire 1713935820 last 1713935743 [3671750.875503] LustreError: 36672:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3671750.905180] LustreError: 36672:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 80 previous similar messages [3672267.104968] Lustre: oak-MDT0003: haven't heard from client 6e8c10f1-cb96-447c-9989-afdbafde1249 (at 10.50.10.35@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900f3887d000, cur 1713936513 expire 1713936363 last 1713936286 [3672267.126929] Lustre: Skipped 1 previous similar message [3672270.094068] Lustre: oak-MDT0000: haven't heard from client e2f3113b-2cbf-4db8-b9bd-51f7b2a65ba8 (at 10.51.3.64@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90147cd01800, cur 1713936516 expire 1713936366 last 1713936289 [3672270.115943] Lustre: Skipped 1 previous similar message [3672355.800509] LustreError: 35181:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107851766932 granted:310379184, total:352885733496 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352885733496 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3672355.831049] LustreError: 35181:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 92 previous similar messages [3672957.061011] Lustre: oak-MDT0000: haven't heard from client 0bc840df-ce88-4f94-a3da-7d10ab8be306 (at 10.51.1.66@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901265e5b800, cur 1713937203 expire 1713937053 last 1713936976 [3672957.082891] Lustre: Skipped 1 previous similar message [3672965.576988] LustreError: 35157:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107961446368 granted:486160868, total:352885733496 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352885733496 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3672965.607529] LustreError: 35157:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 95 previous similar messages [3672972.066533] Lustre: oak-MDT0003: haven't heard from client 0bc840df-ce88-4f94-a3da-7d10ab8be306 (at 10.51.1.66@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901fc004d800, cur 1713937218 expire 1713937068 last 1713936991 [3673569.433339] LustreError: 31509:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107851755200 granted:310379184, total:352885733496 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352885733496 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3673569.463876] LustreError: 31509:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 94 previous similar messages [3674184.269853] LustreError: 35201:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3674184.299526] LustreError: 35201:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 98 previous similar messages [3674667.985786] Lustre: oak-MDT0000: haven't heard from client 574ddeb9-4453-444c-963d-d13f742c6738 (at 10.51.7.54@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900370293800, cur 1713938914 expire 1713938764 last 1713938687 [3674794.119735] LustreError: 35181:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107851537264 granted:310379184, total:352885733496 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352885733496 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3674794.150281] LustreError: 35181:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 142 previous similar messages [3675400.228592] LustreError: 36672:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107851537264 granted:310379184, total:352885733496 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352885733496 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3675400.259135] LustreError: 36672:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 137 previous similar messages [3676002.790971] LustreError: 31510:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107961446016 granted:486160868, total:352885733496 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352885733496 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3676002.821515] LustreError: 31510:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 82 previous similar messages [3676620.940294] LustreError: 35154:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3676620.969970] LustreError: 35154:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 81 previous similar messages [3676940.876780] Lustre: oak-MDT0000: haven't heard from client f80e0cd5-a579-4829-897c-95c78c5db172 (at 10.51.13.21@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903733416000, cur 1713941187 expire 1713941037 last 1713940960 [3676940.898745] Lustre: Skipped 1 previous similar message [3677229.918227] LustreError: 35153:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3677229.948524] LustreError: 35153:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 83 previous similar messages [3677417.999996] LustreError: 33277:0:(qmt_lock.c:425:qmt_lvbo_update()) $$$ failed to release quota space on glimpse 0!=122992722186456 : rc = -22 qmt:oak-QMT0000 pool:dt-0x0 id:9826 enforced:1 hard:134217728 soft:0 granted:134206344 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:3510885 default:no [3677474.853968] Lustre: oak-MDT0000: haven't heard from client 51c0d6a5-1745-4d1e-be5c-b93cf81b10df (at 10.50.7.2@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90186bf33400, cur 1713941721 expire 1713941571 last 1713941494 [3677474.875760] Lustre: Skipped 1 previous similar message [3677479.857547] Lustre: oak-MDT0003: haven't heard from client 51c0d6a5-1745-4d1e-be5c-b93cf81b10df (at 10.50.7.2@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9002a7d32000, cur 1713941726 expire 1713941576 last 1713941499 [3677831.081083] LustreError: 35235:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0209_UUID release: 16517233108325478068 granted:230227968, total:30071094428 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:30071094428 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3677831.111448] LustreError: 35235:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 80 previous similar messages [3678440.029551] LustreError: 35196:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107961445404 granted:486160868, total:352885733496 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352885733496 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3678440.060089] LustreError: 35196:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 85 previous similar messages [3678895.787582] Lustre: oak-MDT0000: haven't heard from client 075fb5eb-8e89-4e97-8429-ed96e51d2501 (at 10.51.15.4@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900e3780dc00, cur 1713943142 expire 1713942992 last 1713942915 [3678909.794527] Lustre: oak-MDT0003: haven't heard from client 075fb5eb-8e89-4e97-8429-ed96e51d2501 (at 10.51.15.4@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902029730800, cur 1713943156 expire 1713943006 last 1713942929 [3679060.945608] LustreError: 36697:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0209_UUID release: 16517233108325478068 granted:230227968, total:30071094428 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:30071094428 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3679060.975978] LustreError: 36697:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 85 previous similar messages [3679465.761860] Lustre: oak-MDT0000: haven't heard from client d26dc93c-71b5-40ed-a99e-a0221fa6d2c9 (at 10.51.7.56@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90046dada400, cur 1713943712 expire 1713943562 last 1713943485 [3679671.557055] LustreError: 36655:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3679671.586732] LustreError: 36655:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 86 previous similar messages [3680215.726131] Lustre: oak-MDT0000: haven't heard from client ddbb6752-4e18-49aa-bf01-9a8d51cf1cf2 (at 10.51.10.31@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90119db9c400, cur 1713944462 expire 1713944312 last 1713944235 [3680215.748231] Lustre: Skipped 1 previous similar message [3680277.783943] LustreError: 36692:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454509505672 granted:126893260, total:30071094428 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:30071094428 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3680277.814227] LustreError: 36692:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 77 previous similar messages [3680892.130118] LustreError: 35146:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020c_UUID release: 18446621080987365160 granted:1028, total:134206344 qmt:oak-QMT0000 pool:dt-0x0 id:9826 enforced:1 hard:134217728 soft:0 granted:134206344 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:3510885 default:no [3680892.159615] LustreError: 35146:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 81 previous similar messages [3681497.258208] LustreError: 36671:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3681497.287888] LustreError: 36671:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 96 previous similar messages [3682104.839948] LustreError: 35145:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3682104.869626] LustreError: 35145:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 94 previous similar messages [3682730.976855] LustreError: 35221:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107947169136 granted:483067692, total:352885733496 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352885733496 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3682731.007464] LustreError: 35221:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 87 previous similar messages [3683331.653868] LustreError: 36686:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107932521436 granted:424654352, total:352885733496 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352885733496 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3683331.684406] LustreError: 36686:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 108 previous similar messages [3683933.953642] LustreError: 36688:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454509505672 granted:126893260, total:30071094428 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:30071094428 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3683933.953643] LustreError: 35229:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107947169096 granted:483067692, total:352885733496 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352885733496 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3683933.953646] LustreError: 35229:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 92 previous similar messages [3684184.544606] Lustre: oak-MDT0003: haven't heard from client 551f7525-db28-43d8-a931-83e8737e903c (at 10.51.8.35@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902060034c00, cur 1713948431 expire 1713948281 last 1713948204 [3684184.566481] Lustre: Skipped 1 previous similar message [3684540.525414] LustreError: 36658:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3684540.555090] LustreError: 36658:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 92 previous similar messages [3685010.506849] Lustre: oak-MDT0003: haven't heard from client ce47d448-24fb-48e2-9fa4-eefabf8593db (at 10.51.10.29@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903856b1fc00, cur 1713949257 expire 1713949107 last 1713949030 [3685010.528815] Lustre: Skipped 1 previous similar message [3685143.506828] LustreError: 35203:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107851537184 granted:310379184, total:352917193056 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352917193056 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3685143.537500] LustreError: 35203:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 1348 previous similar messages [3685743.995087] LustreError: 35207:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107849772828 granted:310379184, total:352917193056 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352917193056 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3685744.025629] LustreError: 35207:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 513 previous similar messages [3686344.094903] LustreError: 35146:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069532682656 granted:559613176, total:352932922420 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352932922420 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3686344.125441] LustreError: 35146:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 515 previous similar messages [3686886.421282] Lustre: oak-MDT0003: haven't heard from client eccce069-47cf-4feb-8b62-f8d4c44c9f06 (at 10.51.15.4@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9020788dbc00, cur 1713951133 expire 1713950983 last 1713950906 [3686886.443162] Lustre: Skipped 1 previous similar message [3686897.426221] Lustre: oak-MDT0000: haven't heard from client eccce069-47cf-4feb-8b62-f8d4c44c9f06 (at 10.51.15.4@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900884511000, cur 1713951144 expire 1713950994 last 1713950917 [3686945.090990] LustreError: 36669:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069529410756 granted:559613176, total:352948652288 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352948652288 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3686945.121538] LustreError: 36669:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 1322 previous similar messages [3687545.422257] LustreError: 36655:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107956123624 granted:486160868, total:352964382512 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352964382512 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3687545.452807] LustreError: 36655:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 494 previous similar messages [3688145.849556] LustreError: 35200:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069526787036 granted:559613176, total:352995846268 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:352995846268 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3688145.880098] LustreError: 35200:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 442 previous similar messages [3688753.831310] LustreError: 36660:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3688753.861160] LustreError: 36660:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 351 previous similar messages [3689356.245556] LustreError: 36667:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020c_UUID release: 18446621080987365160 granted:1028, total:134206344 qmt:oak-QMT0000 pool:dt-0x0 id:9826 enforced:1 hard:134217728 soft:0 granted:134206344 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:3510885 default:no [3689356.275057] LustreError: 36667:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 381 previous similar messages [3689960.571039] LustreError: 35181:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107847654896 granted:310379184, total:353090240312 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353090240312 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3689960.601584] LustreError: 35181:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 736 previous similar messages [3690561.271424] LustreError: 31508:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107932521400 granted:424654352, total:353090240312 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353090240312 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3690561.301965] LustreError: 31508:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 92 previous similar messages [3691161.633046] LustreError: 35227:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107945145628 granted:483067692, total:353090240312 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353090240312 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3691161.663585] LustreError: 35227:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 1200 previous similar messages [3691765.145630] LustreError: 36656:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020c_UUID release: 18446621080987365160 granted:1028, total:134206344 qmt:oak-QMT0000 pool:dt-0x0 id:9826 enforced:1 hard:134217728 soft:0 granted:134206344 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:3510885 default:no [3691765.175140] LustreError: 36656:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 248 previous similar messages [3692368.352236] LustreError: 35207:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3692368.382088] LustreError: 35207:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 91 previous similar messages [3692970.696813] LustreError: 35151:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020c_UUID release: 18446621080987365160 granted:1028, total:134206344 qmt:oak-QMT0000 pool:dt-0x0 id:9826 enforced:1 hard:134217728 soft:0 granted:134206344 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:3510885 default:no [3692970.726495] LustreError: 35151:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 95 previous similar messages [3693576.621290] LustreError: 35203:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107814850800 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3693576.650963] LustreError: 35203:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 99 previous similar messages [3694176.903387] LustreError: 35204:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107845444076 granted:310379184, total:353136396876 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353136396876 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3694176.933929] LustreError: 35204:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 594 previous similar messages [3694778.608427] LustreError: 35203:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3694778.638104] LustreError: 35203:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 199 previous similar messages [3695378.943136] LustreError: 35171:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107942831084 granted:483067692, total:353183586352 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353183586352 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3695378.973670] LustreError: 35171:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 916 previous similar messages [3695555.026362] Lustre: oak-MDT0000: haven't heard from client 1a7f9618-9e5f-4c98-b644-0503d0765d1f (at 10.51.7.34@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9018d6e8dc00, cur 1713959802 expire 1713959652 last 1713959575 [3695982.874609] LustreError: 35176:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3695982.904289] LustreError: 35176:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 1642 previous similar messages [3696584.092896] LustreError: 36681:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107929783040 granted:424654352, total:353261188948 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353261188948 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3696584.123436] LustreError: 36681:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 89 previous similar messages [3697184.772992] LustreError: 36681:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107928234724 granted:424654352, total:353276921028 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353276921028 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3697184.803533] LustreError: 36681:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 1850 previous similar messages [3697785.414156] LustreError: 35157:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107814850800 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3697785.443829] LustreError: 35157:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 4483 previous similar messages [3698385.835865] LustreError: 36677:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107926243604 granted:424654352, total:353369204180 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353369204180 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3698385.866401] LustreError: 36677:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 2945 previous similar messages [3698986.766417] LustreError: 36661:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069522533456 granted:559613176, total:353432128976 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353432128976 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3698986.796959] LustreError: 36661:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 625 previous similar messages [3699586.827232] LustreError: 36685:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107921646840 granted:424654352, total:353463590584 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353463590584 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3699586.857773] LustreError: 36685:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 1264 previous similar messages [3700191.096418] LustreError: 35144:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3700191.126264] LustreError: 35144:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 943 previous similar messages [3700799.083664] LustreError: 36656:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069522533448 granted:559613176, total:353525466476 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353525466476 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3700799.114206] LustreError: 36656:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 96 previous similar messages [3701400.558118] LustreError: 35154:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069522533448 granted:559613176, total:353571621228 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353571621228 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3701400.588659] LustreError: 35154:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 99 previous similar messages [3702000.762694] LustreError: 35201:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069522533448 granted:559613176, total:353571621228 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353571621228 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3702000.793427] LustreError: 35201:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 750 previous similar messages [3702601.106421] LustreError: 35224:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454509505672 granted:126893260, total:30071094428 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:30071094428 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3702601.136702] LustreError: 35224:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 742 previous similar messages [3703212.145927] LustreError: 36678:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454509505672 granted:126893260, total:30071094428 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:30071094428 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3703212.176205] LustreError: 36678:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 96 previous similar messages [3703821.270487] LustreError: 36666:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107944510604 granted:486160868, total:353618809968 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353618809968 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3703821.301033] LustreError: 36666:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 89 previous similar messages [3704421.384086] LustreError: 36670:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020c_UUID release: 18446621080987365160 granted:1028, total:134206344 qmt:oak-QMT0000 pool:dt-0x0 id:9826 enforced:1 hard:134217728 soft:0 granted:134206344 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:3510885 default:no [3704421.413591] LustreError: 36670:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 1297 previous similar messages [3705022.754149] LustreError: 35225:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0209_UUID release: 16517233108325478068 granted:230227968, total:30071094428 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:30071094428 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3705022.784519] LustreError: 35225:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 855 previous similar messages [3705258.584911] Lustre: oak-MDT0000: haven't heard from client acf6ba95-acd7-47fb-9347-54f063b2cca9 (at 10.51.3.5@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903b7ee46800, cur 1713969506 expire 1713969356 last 1713969279 [3705258.606697] Lustre: Skipped 5 previous similar messages [3705266.584512] Lustre: oak-MDT0003: haven't heard from client acf6ba95-acd7-47fb-9347-54f063b2cca9 (at 10.51.3.5@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903a6139c400, cur 1713969514 expire 1713969364 last 1713969287 [3705266.606330] Lustre: Skipped 2 previous similar messages [3705623.362522] LustreError: 35145:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107939221920 granted:486160868, total:353664958176 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353664958176 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3705623.393063] LustreError: 35145:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 659 previous similar messages [3706237.117102] LustreError: 35197:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3706237.146952] LustreError: 35197:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 143 previous similar messages [3706272.616427] Lustre: oak-MDT0000: Client 57264216-9fcb-41c4-a4d3-7b816e6ecbdf (at 10.51.18.1@o2ib3) reconnecting [3706272.626684] Lustre: Skipped 1 previous similar message [3706284.814314] Lustre: oak-MDT0003: Client a383c06e-cbec-4027-9218-5c41810556b2 (at 10.51.9.49@o2ib3) reconnecting [3706301.275734] LustreError: 137-5: oak-MDT0002_UUID: not available for connect from 10.51.0.2@o2ib3 (no target). If you are running an HA pair check that the target is mounted on the other server. [3706307.973449] Lustre: oak-MDT0000: Client fdfeec99-23c0-4f2f-954f-a17eaef9bc5d (at 10.51.3.25@o2ib3) reconnecting [3706307.983989] LustreError: 137-5: oak-MDT0002_UUID: not available for connect from 10.51.5.1@o2ib3 (no target). If you are running an HA pair check that the target is mounted on the other server. [3706307.985346] LNet: Dropping PUT from 12345-10.51.1.25@o2ib3 portal 26 match 1797085887997504 offset 0 length 520: 4 [3706308.012976] LustreError: Skipped 1 previous similar message [3706308.502298] LNet: Dropping PUT from 12345-10.51.5.25@o2ib3 portal 26 match 1797111518951424 offset 0 length 520: 4 [3706308.512820] LNet: Skipped 3 previous similar messages [3706308.988972] LustreError: 137-5: oak-MDT0001_UUID: not available for connect from 10.51.4.65@o2ib3 (no target). If you are running an HA pair check that the target is mounted on the other server. [3706309.006422] LustreError: Skipped 12 previous similar messages [3706309.509286] LNet: Dropping PUT from 12345-10.51.1.41@o2ib3 portal 26 match 1793366788809152 offset 0 length 520: 4 [3706309.519808] LNet: Skipped 7 previous similar messages [3706310.193002] Lustre: oak-MDT0003: Client ed9e04ba-c87e-4fb8-8608-671131f3af90 (at 10.51.9.1@o2ib3) reconnecting [3706310.203176] Lustre: Skipped 46 previous similar messages [3706311.061313] LustreError: 137-5: oak-MDT0001_UUID: not available for connect from 10.51.15.17@o2ib3 (no target). If you are running an HA pair check that the target is mounted on the other server. [3706311.078853] LustreError: Skipped 15 previous similar messages [3706312.281680] LNet: Dropping PUT from 12345-10.51.3.40@o2ib3 portal 26 match 1796971588676608 offset 0 length 520: 4 [3706312.292194] LNet: Skipped 7 previous similar messages [3706316.698803] LustreError: 137-5: oak-MDT0002_UUID: not available for connect from 10.51.1.15@o2ib3 (no target). If you are running an HA pair check that the target is mounted on the other server. [3706316.716256] LustreError: Skipped 8 previous similar messages [3706317.651864] Lustre: oak-MDT0003: Client 876fc126-7fc7-4612-96e2-e88d9be29ef3 (at 10.51.6.25@o2ib3) reconnecting [3706317.662121] Lustre: Skipped 23 previous similar messages [3706320.244317] LNet: Dropping PUT from 12345-10.51.7.41@o2ib3 portal 26 match 1793358552435456 offset 0 length 520: 4 [3706320.254835] LNet: Skipped 4 previous similar messages [3706325.968224] LustreError: 137-5: oak-MDT0002_UUID: not available for connect from 10.51.5.49@o2ib3 (no target). If you are running an HA pair check that the target is mounted on the other server. [3706325.985678] LustreError: Skipped 6 previous similar messages [3706327.137932] Lustre: oak-MDT0003: Client 61edec1b-1d52-45ab-97e8-5038e0eaeb3f (at 10.51.4.40@o2ib3) reconnecting [3706327.148190] Lustre: Skipped 1 previous similar message [3706328.352439] LNet: Dropping PUT from 12345-10.51.9.65@o2ib3 portal 26 match 1796853987598912 offset 0 length 520: 4 [3706343.786341] LustreError: 137-5: oak-MDT0002_UUID: not available for connect from 10.51.16.1@o2ib3 (no target). If you are running an HA pair check that the target is mounted on the other server. [3706343.803798] LustreError: Skipped 21 previous similar messages [3706343.827470] Lustre: oak-MDT0000: Client f1ea759f-698c-404c-bcbb-ddbf5194e880 (at 10.51.16.1@o2ib3) reconnecting [3706343.837723] Lustre: Skipped 66 previous similar messages [3706344.703910] LNet: Dropping PUT from 12345-10.51.6.41@o2ib3 portal 26 match 1796611045031232 offset 0 length 520: 4 [3706344.714433] LNet: Skipped 9 previous similar messages [3706380.406067] Lustre: oak-MDT0003: Client d7793df2-0c8e-4be5-816d-d7b187a822be (at 10.51.3.64@o2ib3) reconnecting [3706380.416330] Lustre: Skipped 12 previous similar messages [3706504.531599] Lustre: oak-MDT0000: haven't heard from client be538544-56f4-47bb-9133-60425a41d4de (at 10.51.4.55@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9006a08ac000, cur 1713970752 expire 1713970602 last 1713970525 [3706504.553474] Lustre: Skipped 2 previous similar messages [3706504.853324] Lustre: oak-MDT0003: Client f78980b5-2a48-4275-8b39-5c46c8234554 (at 10.51.4.17@o2ib3) reconnecting [3706504.863582] Lustre: Skipped 1 previous similar message [3706676.517179] Lustre: oak-MDT0000: haven't heard from client 7453d13f-eb33-4120-aec6-a3927ffad357 (at 10.51.8.72@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903cb3684c00, cur 1713970924 expire 1713970774 last 1713970697 [3706676.539059] Lustre: Skipped 7 previous similar messages [3706837.129776] LustreError: 35154:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3706837.159631] LustreError: 35154:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 100 previous similar messages [3707315.488925] Lustre: oak-MDT0000: haven't heard from client 1f416b5a-b10b-473f-9cb5-cd32a30ef41b (at 10.51.8.8@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90353578bc00, cur 1713971563 expire 1713971413 last 1713971336 [3707315.510711] Lustre: Skipped 1 previous similar message [3707319.519873] Lustre: oak-MDT0003: haven't heard from client 1f416b5a-b10b-473f-9cb5-cd32a30ef41b (at 10.51.8.8@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903cb3686c00, cur 1713971567 expire 1713971417 last 1713971340 [3707319.541657] Lustre: Skipped 1 previous similar message [3707437.189567] LustreError: 36689:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107917747116 granted:424654352, total:353743611540 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353743611540 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3707437.220104] LustreError: 36689:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 271 previous similar messages [3707686.650901] LustreError: 44813:0:(mdt_open.c:1217:mdt_cross_open()) oak-MDT0003: [0x32c0020d63:0x14cdc:0x0] doesn't exist!: rc = -14 [3708037.329048] LustreError: 35149:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069519184964 granted:559613176, total:353804432920 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353804432920 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3708037.359731] LustreError: 35149:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 733 previous similar messages [3708637.349637] LustreError: 35157:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107835570788 granted:310379184, total:353820166796 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353820166796 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3708637.380177] LustreError: 35157:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 1250 previous similar messages [3709009.999099] LustreError: 44973:0:(mdt_open.c:1217:mdt_cross_open()) oak-MDT0003: [0x32c0020d6d:0x262b:0x0] doesn't exist!: rc = -14 [3709010.011086] LustreError: 44973:0:(mdt_open.c:1217:mdt_cross_open()) Skipped 1 previous similar message [3709237.976743] LustreError: 35173:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107914252840 granted:424654352, total:353867358820 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353867358820 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3709238.007611] LustreError: 35173:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 972 previous similar messages [3709838.154795] LustreError: 36697:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107933471432 granted:483067692, total:353914548868 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353914548868 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3709838.185331] LustreError: 36697:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 363 previous similar messages [3710162.355725] Lustre: oak-MDT0003: haven't heard from client 25058274-650c-408e-8b3f-6cb6004483b4 (at 10.51.2.72@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90299c98d800, cur 1713974410 expire 1713974260 last 1713974183 [3710162.377594] Lustre: Skipped 1 previous similar message [3710438.509847] LustreError: 35177:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069517129552 granted:559613176, total:353930278584 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353930278584 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3710438.540511] LustreError: 35177:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 627 previous similar messages [3711044.428939] LustreError: 31507:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454509505672 granted:126893260, total:30071094428 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:30071094428 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3711044.428941] LustreError: 35236:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107931090856 granted:483067692, total:353930278584 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353930278584 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3711044.428945] LustreError: 35236:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 722 previous similar messages [3711592.289211] Lustre: oak-MDT0003: haven't heard from client 30a71819-dd41-4600-a8cb-50cd668e4499 (at 10.51.2.59@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900a9281ec00, cur 1713975840 expire 1713975690 last 1713975613 [3711592.311093] Lustre: Skipped 3 previous similar messages [3711645.611133] LustreError: 36664:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069514425688 granted:559613176, total:353993207168 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:353993207168 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3711645.641858] LustreError: 36664:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 99 previous similar messages [3712246.117842] LustreError: 35163:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107928416148 granted:483067692, total:354024668196 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:354024668196 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3712246.148384] LustreError: 35163:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 722 previous similar messages [3712440.249171] Lustre: oak-MDT0000: haven't heard from client 691a4b8a-d858-4522-83a8-74f1b2510752 (at 10.51.2.49@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900348182400, cur 1713976688 expire 1713976538 last 1713976461 [3712440.271049] Lustre: Skipped 1 previous similar message [3712847.309416] LustreError: 35179:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107832462484 granted:310379184, total:354040399160 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:354040399160 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3712847.339961] LustreError: 35179:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 376 previous similar messages [3713447.538865] LustreError: 36662:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069510204436 granted:559613176, total:354056129448 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:354056129448 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3713447.569406] LustreError: 36662:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 1434 previous similar messages [3714050.195452] LustreError: 35157:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107830936116 granted:310379184, total:354056129448 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:354056129448 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3714050.225992] LustreError: 35157:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 997 previous similar messages [3714092.172918] Lustre: oak-MDT0003: haven't heard from client 53ae3788-8521-4358-b64e-421b6dc17020 (at 10.50.7.67@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90402ba76c00, cur 1713978340 expire 1713978190 last 1713978113 [3714092.194795] Lustre: Skipped 1 previous similar message [3714326.162050] Lustre: oak-MDT0003: haven't heard from client 8daa6422-1527-434e-bf2d-22f14fb3c54f (at 10.50.8.1@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9010ae24cc00, cur 1713978574 expire 1713978424 last 1713978347 [3714326.183832] Lustre: Skipped 1 previous similar message [3714485.155987] Lustre: oak-MDT0000: haven't heard from client 444288d2-7ffd-421e-9bc6-52a62a5ba820 (at 10.50.1.53@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903fe4ef7400, cur 1713978733 expire 1713978583 last 1713978506 [3714485.177866] Lustre: Skipped 1 previous similar message [3714650.198134] LustreError: 35172:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107926702644 granted:483067692, total:354070817228 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:354070817228 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3714650.228767] LustreError: 35172:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 535 previous similar messages [3714946.133806] Lustre: oak-MDT0000: haven't heard from client 9775a998-3514-487a-a0c0-44cd2f3f6650 (at 10.51.9.66@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9020b49f7800, cur 1713979194 expire 1713979044 last 1713978967 [3714946.155687] Lustre: Skipped 1 previous similar message [3715254.983269] LustreError: 35144:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3715255.012946] LustreError: 35144:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 334 previous similar messages [3715855.047874] LustreError: 36687:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107922693924 granted:483067692, total:354116957116 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:354116957116 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3715855.078409] LustreError: 36687:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 770 previous similar messages [3716465.316297] LustreError: 35156:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020c_UUID release: 18446621080987365160 granted:1028, total:134206344 qmt:oak-QMT0000 pool:dt-0x0 id:9826 enforced:1 hard:134217728 soft:0 granted:134206344 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:3510885 default:no [3716465.346302] LustreError: 35156:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 637 previous similar messages [3716604.060215] Lustre: oak-MDT0003: haven't heard from client 68bce30f-9282-40fe-a99d-3a404e2d4959 (at 10.51.16.11@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901105704000, cur 1713980852 expire 1713980702 last 1713980625 [3716604.082172] Lustre: Skipped 3 previous similar messages [3717076.838306] LustreError: 31508:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0209_UUID release: 16517233108325478068 granted:230227968, total:30071094428 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:30071094428 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3717076.868678] LustreError: 31508:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 91 previous similar messages [3717191.026743] Lustre: oak-MDT0000: haven't heard from client 34f4a785-66ea-414f-b507-baaa8feca603 (at 10.50.13.13@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903ae9711400, cur 1713981439 expire 1713981289 last 1713981212 [3717191.048980] Lustre: Skipped 1 previous similar message [3717677.487642] LustreError: 35201:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107830941528 granted:310379184, total:354179883876 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:354179883876 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3717677.518185] LustreError: 35201:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 103 previous similar messages [3718300.307359] LustreError: 36659:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3718300.337062] LustreError: 36659:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 455 previous similar messages [3718900.728849] LustreError: 35237:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107913490184 granted:424654352, total:354357128132 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:354357128132 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3718900.759388] LustreError: 35237:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 815 previous similar messages [3719127.523687] LustreError: 44790:0:(mdt_open.c:1217:mdt_cross_open()) oak-MDT0003: [0x32c0020d8f:0x868c:0x0] doesn't exist!: rc = -14 [3719505.302632] LustreError: 36693:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454509505672 granted:126893260, total:30071094428 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:30071094428 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3719505.302633] LustreError: 36683:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107920229528 granted:483067692, total:354372862344 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:354372862344 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3719505.302636] LustreError: 36683:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 94 previous similar messages [3719663.912607] Lustre: oak-MDT0000: haven't heard from client d4adbedf-cc20-46ca-b256-bb2ec1b9d1ab (at 10.51.13.16@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90059a8aa000, cur 1713983912 expire 1713983762 last 1713983685 [3719663.934573] Lustre: Skipped 1 previous similar message [3720105.940279] LustreError: 31507:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0209_UUID release: 16517233108325478068 granted:230227968, total:30071094428 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:30071094428 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3720105.970647] LustreError: 31507:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 82 previous similar messages [3720707.449867] LustreError: 35163:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0209_UUID release: 16517233108325478068 granted:230227968, total:30071094428 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:30071094428 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3720707.480234] LustreError: 35163:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 87 previous similar messages [3721312.040473] LustreError: 35183:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107936342216 granted:486160868, total:354514433868 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:354514433868 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3721312.071016] LustreError: 35183:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 158 previous similar messages [3721913.595367] LustreError: 35200:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3721913.625221] LustreError: 35200:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 82 previous similar messages [3722514.196382] LustreError: 35196:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107936342216 granted:486160868, total:354544850856 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:354544850856 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3722514.226917] LustreError: 35196:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 86 previous similar messages [3723139.184121] LustreError: 35181:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3723139.213972] LustreError: 35181:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 88 previous similar messages [3723751.238564] LustreError: 36658:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107936342216 granted:486160868, total:354639236288 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:354639236288 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3723751.269114] LustreError: 36658:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 86 previous similar messages [3724362.240895] LustreError: 36680:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0209_UUID release: 16517233108325478068 granted:230227968, total:30071094428 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:30071094428 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3724362.271259] LustreError: 36680:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 89 previous similar messages [3724552.682478] Lustre: oak-MDT0003: haven't heard from client d0948315-2007-40e2-84fc-d62feaec171d (at 10.51.2.44@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9031b5524800, cur 1713988801 expire 1713988651 last 1713988574 [3724552.704359] Lustre: Skipped 1 previous similar message [3724962.636863] LustreError: 35221:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0209_UUID release: 16517233108325478068 granted:230227968, total:30071094428 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:30071094428 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3724962.667229] LustreError: 35221:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 87 previous similar messages [3725562.652943] LustreError: 35203:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069508643704 granted:559613176, total:354686428300 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:354686428300 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3725562.683492] LustreError: 35203:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 189 previous similar messages [3725712.641758] Lustre: oak-MDT0000: haven't heard from client b0de144c-8184-461f-a0a3-94f6842f8b25 (at 10.50.10.2@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902f49c76c00, cur 1713989961 expire 1713989811 last 1713989734 [3725712.663642] Lustre: Skipped 1 previous similar message [3725721.633031] Lustre: oak-MDT0003: haven't heard from client b0de144c-8184-461f-a0a3-94f6842f8b25 (at 10.50.10.2@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9027827c5800, cur 1713989970 expire 1713989820 last 1713989743 [3726162.899717] LustreError: 35201:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069505562112 granted:559613176, total:354686428300 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:354686428300 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3726162.930265] LustreError: 35201:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 853 previous similar messages [3726750.580583] Lustre: oak-MDT0000: haven't heard from client cda23cf1-f0ba-47bb-bd42-0894e8937aa3 (at 10.50.10.2@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9040cd7cf400, cur 1713990999 expire 1713990849 last 1713990772 [3726767.465280] LustreError: 35175:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107919996568 granted:483067692, total:354748300820 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:354748300820 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3726767.495824] LustreError: 35175:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 135 previous similar messages [3727372.536592] LustreError: 35149:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3727372.566284] LustreError: 35149:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 86 previous similar messages [3727613.330181] LustreError: 44920:0:(mdt_open.c:1217:mdt_cross_open()) oak-MDT0003: [0x32c0020da4:0x1851:0x0] doesn't exist!: rc = -14 [3727613.342170] LustreError: 44920:0:(mdt_open.c:1217:mdt_cross_open()) Skipped 1 previous similar message [3727888.529080] Lustre: oak-MDT0003: haven't heard from client eb6b0a68-e5db-46b7-bf85-3776c84dcb28 (at 10.51.16.5@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903b4ea06c00, cur 1713992137 expire 1713991987 last 1713991910 [3727888.550976] Lustre: Skipped 3 previous similar messages [3727985.502999] LustreError: 35205:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020c_UUID release: 18446621080987365160 granted:1028, total:134206344 qmt:oak-QMT0000 pool:dt-0x0 id:9826 enforced:1 hard:134217728 soft:0 granted:134206344 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:3510885 default:no [3727985.532498] LustreError: 35205:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 89 previous similar messages [3728602.124187] LustreError: 35201:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3728602.154035] LustreError: 35201:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 88 previous similar messages [3728768.485397] Lustre: oak-MDT0003: haven't heard from client 22223d35-c478-40a8-9962-e31ae9d24de9 (at 10.51.10.27@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902092db0c00, cur 1713993017 expire 1713992867 last 1713992790 [3728768.507517] Lustre: Skipped 1 previous similar message [3729209.895884] LustreError: 36661:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107830941468 granted:310379184, total:354873100832 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:354873100832 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3729209.926428] LustreError: 36661:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 91 previous similar messages [3729832.184777] LustreError: 35155:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020c_UUID release: 18446621080987365160 granted:1028, total:134206344 qmt:oak-QMT0000 pool:dt-0x0 id:9826 enforced:1 hard:134217728 soft:0 granted:134206344 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:3510885 default:no [3729832.214282] LustreError: 35155:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 91 previous similar messages [3730047.426714] Lustre: oak-MDT0000: haven't heard from client f7b5054c-993d-4f34-bc65-67bcae828172 (at 10.51.16.5@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902092d87400, cur 1713994296 expire 1713994146 last 1713994069 [3730047.448594] Lustre: Skipped 1 previous similar message [3730438.893540] LustreError: 35147:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020c_UUID release: 18446621080987365160 granted:1028, total:134206344 qmt:oak-QMT0000 pool:dt-0x0 id:9826 enforced:1 hard:134217728 soft:0 granted:134206344 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:3510885 default:no [3730438.923042] LustreError: 35147:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 93 previous similar messages [3731041.850341] LustreError: 35147:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020c_UUID release: 18446621080987365160 granted:1028, total:134206344 qmt:oak-QMT0000 pool:dt-0x0 id:9826 enforced:1 hard:134217728 soft:0 granted:134206344 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:3510885 default:no [3731041.879839] LustreError: 35147:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 92 previous similar messages [3731653.537242] LustreError: 35172:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107913450172 granted:424654352, total:355028310964 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355028310964 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3731653.567782] LustreError: 35172:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 84 previous similar messages [3732253.604521] LustreError: 36669:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069505562040 granted:559613176, total:355028310964 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355028310964 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3732253.635062] LustreError: 36669:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 97 previous similar messages [3732868.819448] LustreError: 36691:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0209_UUID release: 16517233108325478068 granted:230227968, total:30071094428 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:30071094428 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3732868.849805] LustreError: 36691:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 96 previous similar messages [3733061.704513] LustreError: 44947:0:(mdt_open.c:1217:mdt_cross_open()) oak-MDT0003: [0x32c0020dac:0x1abe0:0x0] doesn't exist!: rc = -14 [3733436.114548] LustreError: 44949:0:(mdt_open.c:1217:mdt_cross_open()) oak-MDT0003: [0x32c0020dac:0x1fc6d:0x0] doesn't exist!: rc = -14 [3733471.151908] LustreError: 35153:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3733471.181577] LustreError: 35153:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 95 previous similar messages [3734072.146437] LustreError: 36698:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0209_UUID release: 16517233108325478068 granted:230227968, total:30071094428 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:30071094428 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3734072.176984] LustreError: 36698:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 92 previous similar messages [3734679.792994] LustreError: 35227:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107919785312 granted:483067692, total:355168847560 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355168847560 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3734679.823699] LustreError: 35227:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 211 previous similar messages [3735261.292174] LustreError: 44880:0:(mdt_open.c:1217:mdt_cross_open()) oak-MDT0003: [0x32c0020de4:0x104fd:0x0] doesn't exist!: rc = -14 [3735261.304247] LustreError: 44880:0:(mdt_open.c:1217:mdt_cross_open()) Skipped 1 previous similar message [3735285.240978] LustreError: 36691:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107919785088 granted:483067692, total:355230729064 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355230729064 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3735285.271522] LustreError: 36691:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 97 previous similar messages [3735894.642629] LustreError: 36657:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020c_UUID release: 18446621080987365160 granted:1028, total:134206344 qmt:oak-QMT0000 pool:dt-0x0 id:9826 enforced:1 hard:134217728 soft:0 granted:134206344 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:3510885 default:no [3735894.672129] LustreError: 36657:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 166 previous similar messages [3736495.329356] LustreError: 36666:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3736495.359205] LustreError: 36666:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 96 previous similar messages [3737119.945529] LustreError: 35175:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107913450152 granted:424654352, total:355261163376 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355261163376 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3737119.976213] LustreError: 35175:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 101 previous similar messages [3737723.220768] LustreError: 35185:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0209_UUID release: 16517233108325478068 granted:230227968, total:30071094428 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:30071094428 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3737723.251134] LustreError: 35185:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 96 previous similar messages [3738348.269751] LustreError: 35184:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107913450152 granted:424654352, total:355261163376 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355261163376 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3738348.300436] LustreError: 35184:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 93 previous similar messages [3738403.035585] Lustre: oak-MDT0000: haven't heard from client a7c007b0-d8ca-480b-94d8-97269bc8630c (at 10.51.15.4@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900f5bb50800, cur 1714002652 expire 1714002502 last 1714002425 [3738403.057468] Lustre: Skipped 1 previous similar message [3738948.281636] LustreError: 36687:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107913450152 granted:424654352, total:355261163376 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355261163376 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3738948.312174] LustreError: 36687:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 97 previous similar messages [3739552.737653] LustreError: 35203:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3739552.767331] LustreError: 35203:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 101 previous similar messages [3740155.959446] LustreError: 35148:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107830939800 granted:310379184, total:355261163376 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355261163376 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3740155.989993] LustreError: 35148:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 97 previous similar messages [3740278.947656] Lustre: oak-MDT0000: haven't heard from client 3c4c7750-5869-45ba-ac13-f6ce1641ba6c (at 10.51.5.54@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901a42dbe000, cur 1714004528 expire 1714004378 last 1714004301 [3740278.969529] Lustre: Skipped 1 previous similar message [3740288.946505] Lustre: oak-MDT0003: haven't heard from client 3c4c7750-5869-45ba-ac13-f6ce1641ba6c (at 10.51.5.54@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900e24925000, cur 1714004538 expire 1714004388 last 1714004311 [3740288.968378] Lustre: Skipped 7 previous similar messages [3740760.560739] LustreError: 35164:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107913450152 granted:424654352, total:355261163376 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355261163376 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3740760.591273] LustreError: 35164:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 91 previous similar messages [3741376.416254] LustreError: 36664:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107830939800 granted:310379184, total:355261163376 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355261163376 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3741376.446789] LustreError: 36664:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 91 previous similar messages [3741976.429057] LustreError: 35149:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107830939800 granted:310379184, total:355261163376 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355261163376 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3741976.459597] LustreError: 35149:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 89 previous similar messages [3742268.854445] Lustre: oak-MDT0003: haven't heard from client df811f26-2f74-4502-ad24-0ea786bb2ebe (at 10.51.1.54@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901517e85000, cur 1714006518 expire 1714006368 last 1714006291 [3742268.876314] Lustre: Skipped 7 previous similar messages [3742344.859397] Lustre: oak-MDT0000: haven't heard from client 1f6239e6-b0aa-4c94-b917-6d001dba7ded (at 10.51.16.12@o2ib3) in 204 seconds. I think it's dead, and I am evicting it. exp ffff902c35a1ac00, cur 1714006594 expire 1714006444 last 1714006390 [3742344.881362] Lustre: Skipped 1 previous similar message [3742585.558527] LustreError: 36665:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107830939800 granted:310379184, total:355261163376 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355261163376 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3742585.589077] LustreError: 36665:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 89 previous similar messages [3743185.578030] LustreError: 35219:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107913450152 granted:424654352, total:355261163376 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355261163376 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3743185.608567] LustreError: 35219:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 94 previous similar messages [3743764.784355] Lustre: oak-MDT0000: haven't heard from client 1900ed0a-ea79-4a47-a2c1-82036f6cbc33 (at 10.50.10.71@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9020f8fe2400, cur 1714008014 expire 1714007864 last 1714007787 [3743764.806321] Lustre: Skipped 1 previous similar message [3743796.312949] LustreError: 35200:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069505562008 granted:559613176, total:355261163376 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355261163376 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3743796.343628] LustreError: 35200:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 96 previous similar messages [3744101.769560] Lustre: oak-MDT0000: haven't heard from client 6db91e6a-8684-4a4d-927f-337c0bc31dea (at 10.51.16.2@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902fe8d57c00, cur 1714008351 expire 1714008201 last 1714008124 [3744101.791437] Lustre: Skipped 1 previous similar message [3744397.040425] LustreError: 35219:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0209_UUID release: 16517233108325478068 granted:230227968, total:30071094428 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:30071094428 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3744397.070795] LustreError: 35219:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 93 previous similar messages [3744423.753354] Lustre: oak-MDT0000: haven't heard from client 49009184-969d-441b-b095-0e5b0f529b6b (at 10.50.5.41@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9024acd22c00, cur 1714008673 expire 1714008523 last 1714008446 [3744423.775229] Lustre: Skipped 1 previous similar message [3744597.746164] Lustre: oak-MDT0000: haven't heard from client 3443c67f-133f-490c-b1e2-09057fa1b9d7 (at 10.51.16.2@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902dedba1000, cur 1714008847 expire 1714008697 last 1714008620 [3744597.768034] Lustre: Skipped 3 previous similar messages [3744998.751908] LustreError: 36666:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069505562008 granted:559613176, total:355261163376 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355261163376 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3744998.782447] LustreError: 36666:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 89 previous similar messages [3745600.249831] LustreError: 35146:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020c_UUID release: 18446621080987365160 granted:1028, total:134206344 qmt:oak-QMT0000 pool:dt-0x0 id:9826 enforced:1 hard:134217728 soft:0 granted:134206344 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:3510885 default:no [3745600.279333] LustreError: 35146:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 92 previous similar messages [3746056.688609] Lustre: oak-MDT0003: haven't heard from client 1504c502-7f96-4efb-8d75-993cec8acead (at 10.51.1.61@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9028c7579c00, cur 1714010306 expire 1714010156 last 1714010079 [3746056.710488] Lustre: Skipped 1 previous similar message [3746065.688459] Lustre: oak-MDT0000: haven't heard from client 1504c502-7f96-4efb-8d75-993cec8acead (at 10.51.1.61@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90292ff0c400, cur 1714010315 expire 1714010165 last 1714010088 [3746209.734339] LustreError: 36668:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107830939800 granted:310379184, total:355261163376 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355261163376 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3746209.764883] LustreError: 36668:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 94 previous similar messages [3746552.655004] Lustre: oak-MDT0000: haven't heard from client c9dc0c26-1db3-46af-962e-8e2759924876 (at 10.50.1.56@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901e72f1e000, cur 1714010802 expire 1714010652 last 1714010575 [3746811.749131] LustreError: 35199:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020c_UUID release: 18446621080987365160 granted:1028, total:134206344 qmt:oak-QMT0000 pool:dt-0x0 id:9826 enforced:1 hard:134217728 soft:0 granted:134206344 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:3510885 default:no [3746811.778982] LustreError: 35199:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 91 previous similar messages [3746814.642335] Lustre: oak-MDT0000: haven't heard from client 36354a41-ca14-468d-bb8e-4accfa6ce513 (at 10.51.9.6@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900bc0d9a400, cur 1714011064 expire 1714010914 last 1714010837 [3746814.664125] Lustre: Skipped 1 previous similar message [3747415.965805] LustreError: 36650:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069505562008 granted:559613176, total:355261163376 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355261163376 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3747415.996347] LustreError: 36650:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 89 previous similar messages [3748019.612592] LustreError: 35183:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107830939800 granted:310379184, total:355261163376 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355261163376 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3748019.643129] LustreError: 35183:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 95 previous similar messages [3748630.931976] LustreError: 35203:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107830939800 granted:310379184, total:355261163376 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355261163376 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3748630.962519] LustreError: 35203:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 97 previous similar messages [3749234.942240] LustreError: 36677:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107913450152 granted:424654352, total:355261163376 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355261163376 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3749234.972776] LustreError: 36677:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 96 previous similar messages [3749720.504991] Lustre: oak-MDT0003: haven't heard from client d9bf61b5-1296-407f-89c8-2fcd6e20689a (at 10.51.8.42@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903b43f9e000, cur 1714013970 expire 1714013820 last 1714013743 [3749720.526870] Lustre: Skipped 1 previous similar message [3749840.980811] LustreError: 35220:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107913450152 granted:424654352, total:355261163376 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355261163376 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3749841.011354] LustreError: 35220:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 91 previous similar messages [3750458.542928] LustreError: 35226:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107919785312 granted:483067692, total:355261163376 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355261163376 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3750458.573469] LustreError: 35226:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 92 previous similar messages [3751069.235353] LustreError: 35172:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0209_UUID release: 16517233108325478068 granted:230227968, total:30071094428 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:30071094428 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3751069.266091] LustreError: 35172:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 90 previous similar messages [3751670.059851] LustreError: 35203:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069505562008 granted:559613176, total:355261163376 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355261163376 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3751670.090394] LustreError: 35203:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 85 previous similar messages [3752275.417408] LustreError: 35207:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107830939800 granted:310379184, total:355252406484 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355252406484 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3752275.447960] LustreError: 35207:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 94 previous similar messages [3752850.358538] Lustre: oak-MDT0000: haven't heard from client a28f7a9c-bb51-4f50-885c-e572937bffa4 (at 10.51.9.53@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9022f7184800, cur 1714017100 expire 1714016950 last 1714016873 [3752850.380413] Lustre: Skipped 1 previous similar message [3752883.891732] LustreError: 44827:0:(mdt_open.c:1217:mdt_cross_open()) oak-MDT0003: [0x32c0020e0b:0x67e2:0x0] doesn't exist!: rc = -14 [3752886.734884] LustreError: 35147:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107830939800 granted:310379184, total:355252406484 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355252406484 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3752886.765587] LustreError: 35147:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 91 previous similar messages [3753348.336112] Lustre: oak-MDT0000: haven't heard from client caa8075d-9baf-4caa-a7ad-37f7c9d7795d (at 10.51.5.10@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9018cf36c400, cur 1714017598 expire 1714017448 last 1714017371 [3753348.357990] Lustre: Skipped 7 previous similar messages [3753496.224277] LustreError: 36654:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069505561960 granted:559613176, total:355252406484 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355252406484 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3753496.254823] LustreError: 36654:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 91 previous similar messages [3753753.318096] Lustre: oak-MDT0000: haven't heard from client 7d8a8dc0-9d08-411e-80e9-b1493dedc1d8 (at 10.51.7.49@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90224da71000, cur 1714018003 expire 1714017853 last 1714017776 [3753753.339974] Lustre: Skipped 17 previous similar messages [3753766.318791] Lustre: oak-MDT0003: haven't heard from client 7d8a8dc0-9d08-411e-80e9-b1493dedc1d8 (at 10.51.7.49@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90360eeaf400, cur 1714018016 expire 1714017866 last 1714017789 [3753766.340671] Lustre: Skipped 1 previous similar message [3754100.677808] LustreError: 35234:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107919785312 granted:483067692, total:355252406484 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355252406484 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3754100.708346] LustreError: 35234:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 98 previous similar messages [3754443.287582] Lustre: oak-MDT0000: haven't heard from client c6c57547-a84b-472e-9766-c5aec83eab48 (at 10.51.3.30@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901d579a1000, cur 1714018693 expire 1714018543 last 1714018466 [3754443.309611] Lustre: Skipped 1 previous similar message [3754709.509305] LustreError: 35226:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454509505672 granted:126893260, total:30071094428 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:30071094428 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3754709.539586] LustreError: 35226:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 92 previous similar messages [3755309.588779] LustreError: 35204:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107830939800 granted:310379184, total:355252406484 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355252406484 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3755309.619343] LustreError: 35204:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 92 previous similar messages [3755910.966400] LustreError: 36685:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107919785312 granted:483067692, total:355252406484 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355252406484 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3755910.996944] LustreError: 36685:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 93 previous similar messages [3755973.213886] Lustre: oak-MDT0003: haven't heard from client ab045e4f-be6a-4ede-a786-4a7525b1a93f (at 10.51.2.54@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903797e8ec00, cur 1714020223 expire 1714020073 last 1714019996 [3755973.236079] Lustre: Skipped 1 previous similar message [3756513.535142] LustreError: 35168:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107919785312 granted:483067692, total:355252406484 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355252406484 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3756513.535143] LustreError: 36677:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454509505672 granted:126893260, total:30071094428 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:30071094428 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3756513.535146] LustreError: 36677:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 93 previous similar messages [3757122.885005] LustreError: 35198:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020c_UUID release: 18446621080987365160 granted:1028, total:134206344 qmt:oak-QMT0000 pool:dt-0x0 id:9826 enforced:1 hard:134217728 soft:0 granted:134206344 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:3510885 default:no [3757122.914507] LustreError: 35198:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 98 previous similar messages [3757725.426023] LustreError: 35195:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3757725.455700] LustreError: 35195:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 94 previous similar messages [3758338.949876] LustreError: 36680:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0209_UUID release: 16517233108325478068 granted:230227968, total:30071094428 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:30071094428 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3758338.980252] LustreError: 36680:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 98 previous similar messages [3758889.080131] Lustre: oak-MDT0000: haven't heard from client d097b6fd-7810-4759-95e4-1498fce435b0 (at 10.51.9.22@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901f795c6400, cur 1714023139 expire 1714022989 last 1714022912 [3758889.102009] Lustre: Skipped 1 previous similar message [3758941.081604] LustreError: 35205:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069505561952 granted:559613176, total:355252406484 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355252406484 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3758941.112147] LustreError: 35205:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 93 previous similar messages [3759377.054816] Lustre: oak-MDT0000: haven't heard from client 5f4766f7-7904-423d-8746-859d165ee49e (at 10.51.2.51@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9002cf80b800, cur 1714023627 expire 1714023477 last 1714023400 [3759377.076691] Lustre: Skipped 1 previous similar message [3759544.131989] LustreError: 35177:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069505561952 granted:559613176, total:355252406484 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355252406484 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3759544.162536] LustreError: 35177:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 99 previous similar messages [3760156.593654] LustreError: 35206:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3760156.623482] LustreError: 35206:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 98 previous similar messages [3760757.649183] LustreError: 35175:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107913450152 granted:424654352, total:355252406484 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355252406484 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3760757.679713] LustreError: 35175:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 98 previous similar messages [3761068.976357] Lustre: oak-MDT0000: haven't heard from client b5e79138-897c-4511-9028-d20496cfe4ce (at 10.50.14.13@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902c3b6df400, cur 1714025319 expire 1714025169 last 1714025092 [3761068.998324] Lustre: Skipped 1 previous similar message [3761073.991821] Lustre: oak-MDT0003: haven't heard from client b5e79138-897c-4511-9028-d20496cfe4ce (at 10.50.14.13@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9032aa687400, cur 1714025324 expire 1714025174 last 1714025097 [3761359.197756] LustreError: 35197:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3761359.227608] LustreError: 35197:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 96 previous similar messages [3761978.528866] LustreError: 36691:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0209_UUID release: 16517233108325478068 granted:230227968, total:30071094428 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:30071094428 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3761978.559232] LustreError: 36691:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 98 previous similar messages [3762580.411554] LustreError: 35182:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3762580.441229] LustreError: 35182:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 107 previous similar messages [3763203.881675] LustreError: 35153:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107936226068 granted:486160868, total:355252406484 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355252406484 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3763203.912387] LustreError: 35153:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 99 previous similar messages [3763380.876404] Lustre: oak-MDT0000: haven't heard from client bfead361-5c88-4cb1-b684-ecfa9a476e87 (at 10.51.11.2@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900e6e50fc00, cur 1714027631 expire 1714027481 last 1714027404 [3763804.338022] LustreError: 36653:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3763804.367693] LustreError: 36653:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 92 previous similar messages [3764416.292452] LustreError: 35168:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107913450152 granted:424654352, total:355252406484 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355252406484 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3764416.322997] LustreError: 35168:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 104 previous similar messages [3765019.279599] LustreError: 35173:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107919785312 granted:483067692, total:355252406484 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355252406484 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3765019.310137] LustreError: 35173:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 102 previous similar messages [3765630.054312] LustreError: 36678:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107913450152 granted:424654352, total:355252406484 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355252406484 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3765630.084847] LustreError: 36678:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 99 previous similar messages [3766231.330500] LustreError: 35218:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107913450152 granted:424654352, total:355252406484 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355252406484 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3766231.361302] LustreError: 35218:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 102 previous similar messages [3766840.385348] LustreError: 35185:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0209_UUID release: 16517233108325478068 granted:230227968, total:30071094428 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:30071094428 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3766840.415714] LustreError: 35185:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 106 previous similar messages [3767454.130380] LustreError: 35178:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020c_UUID release: 18446621080987365160 granted:1028, total:134206344 qmt:oak-QMT0000 pool:dt-0x0 id:9826 enforced:1 hard:134217728 soft:0 granted:134206344 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:3510885 default:no [3767454.159882] LustreError: 35178:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 102 previous similar messages [3767556.678052] Lustre: oak-MDT0003: haven't heard from client 06e5f7ee-7856-440d-8eb2-769ff2f8c488 (at 10.51.14.5@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9012416c5000, cur 1714031807 expire 1714031657 last 1714031580 [3767556.700039] Lustre: Skipped 1 previous similar message [3768089.536915] LustreError: 36661:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069505561932 granted:559613176, total:355252406484 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355252406484 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3768089.567460] LustreError: 36661:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 101 previous similar messages [3768628.629910] Lustre: oak-MDT0000: haven't heard from client 97fa3e7b-4f60-4530-ad14-1e4f74ee4629 (at 10.51.13.15@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903d1b4b3400, cur 1714032879 expire 1714032729 last 1714032652 [3768628.651873] Lustre: Skipped 1 previous similar message [3768636.629228] Lustre: oak-MDT0003: haven't heard from client 97fa3e7b-4f60-4530-ad14-1e4f74ee4629 (at 10.51.13.15@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902f869f9400, cur 1714032887 expire 1714032737 last 1714032660 [3768691.584201] LustreError: 35205:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069505561932 granted:559613176, total:355252406484 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355252406484 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3768691.614744] LustreError: 35205:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 101 previous similar messages [3769291.791241] LustreError: 35146:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3769291.820911] LustreError: 35146:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 96 previous similar messages [3769900.406069] LustreError: 35200:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3769900.435914] LustreError: 35200:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 111 previous similar messages [3770510.307932] LustreError: 35183:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069505561888 granted:559613176, total:355252406484 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355252406484 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3770510.338472] LustreError: 35183:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 94 previous similar messages [3771115.977580] LustreError: 36693:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107913450072 granted:424654352, total:355252406484 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355252406484 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3771116.008119] LustreError: 36693:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 103 previous similar messages [3771722.293450] LustreError: 36654:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069505561884 granted:559613176, total:355252406484 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355252406484 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3771722.323988] LustreError: 36654:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 105 previous similar messages [3772325.689673] LustreError: 35224:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107919785280 granted:483067692, total:355267097188 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355267097188 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3772325.720214] LustreError: 35224:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 103 previous similar messages [3772927.611297] LustreError: 35153:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107936226032 granted:486160868, total:355267097188 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355267097188 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3772927.641841] LustreError: 35153:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 96 previous similar messages [3773540.365770] LustreError: 35225:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0209_UUID release: 16517233108325478068 granted:230227968, total:30071094428 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:30071094428 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3773540.396132] LustreError: 35225:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 98 previous similar messages [3773905.384045] Lustre: oak-MDT0000: haven't heard from client 11452f8d-7fe1-4eff-a446-5b09868579ce (at 10.50.2.21@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900fcba8ec00, cur 1714038156 expire 1714038006 last 1714037929 [3774142.354455] LustreError: 35181:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020c_UUID release: 18446621080987365160 granted:1028, total:134206344 qmt:oak-QMT0000 pool:dt-0x0 id:9826 enforced:1 hard:134217728 soft:0 granted:134206344 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:3510885 default:no [3774142.383961] LustreError: 35181:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 101 previous similar messages [3774743.852456] LustreError: 35237:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107913450072 granted:424654352, total:355267097188 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355267097188 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3774743.882998] LustreError: 35237:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 97 previous similar messages [3775348.228017] LustreError: 35145:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107830939800 granted:310379184, total:355267097188 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355267097188 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3775348.258553] LustreError: 35145:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 103 previous similar messages [3775554.308101] Lustre: oak-MDT0003: haven't heard from client f5f2cc45-ad68-45a1-8773-b521c72086e7 (at 10.51.7.12@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901d5aca2400, cur 1714039805 expire 1714039655 last 1714039578 [3775554.329972] Lustre: Skipped 1 previous similar message [3775948.350533] LustreError: 35183:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020c_UUID release: 18446621080987365160 granted:1028, total:134206344 qmt:oak-QMT0000 pool:dt-0x0 id:9826 enforced:1 hard:134217728 soft:0 granted:134206344 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:3510885 default:no [3775948.380034] LustreError: 35183:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 102 previous similar messages [3776554.476578] LustreError: 35153:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069505561876 granted:559613176, total:355267097188 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355267097188 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3776554.507123] LustreError: 35153:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 95 previous similar messages [3777160.265646] LustreError: 35200:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3777160.295496] LustreError: 35200:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 101 previous similar messages [3777760.277287] LustreError: 36659:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3777760.307130] LustreError: 36659:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 93 previous similar messages [3778287.181907] Lustre: oak-MDT0000: haven't heard from client 8391c876-73b5-4dfe-85a9-182337c0280d (at 10.51.4.3@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90148a1ec400, cur 1714042538 expire 1714042388 last 1714042311 [3778287.203696] Lustre: Skipped 1 previous similar message [3778367.802243] LustreError: 35196:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107936226032 granted:486160868, total:355267097188 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355267097188 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3778367.832782] LustreError: 35196:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 106 previous similar messages [3778975.981163] LustreError: 35199:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020c_UUID release: 18446621080987365160 granted:1028, total:134206344 qmt:oak-QMT0000 pool:dt-0x0 id:9826 enforced:1 hard:134217728 soft:0 granted:134206344 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:3510885 default:no [3778976.010667] LustreError: 35199:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 101 previous similar messages [3779576.796028] LustreError: 35224:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107913450072 granted:424654352, total:355267097188 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355267097188 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3779576.826559] LustreError: 35224:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 107 previous similar messages [3780179.699568] LustreError: 36657:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3780179.729242] LustreError: 36657:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 96 previous similar messages [3780780.330407] LustreError: 35234:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0209_UUID release: 16517233108325478064 granted:230227968, total:30071094428 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:30071094428 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3780780.360769] LustreError: 35234:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 102 previous similar messages [3780980.058138] Lustre: oak-MDT0000: haven't heard from client 461cc5ae-b16c-47e9-878f-e68a79ca3ef1 (at 10.50.3.51@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9038fcba4800, cur 1714045231 expire 1714045081 last 1714045004 [3780980.080016] Lustre: Skipped 1 previous similar message [3781392.009069] LustreError: 36699:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454509505672 granted:126893260, total:30071094428 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:30071094428 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3781392.009070] LustreError: 35161:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107919785280 granted:483067692, total:355282829208 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355282829208 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3781392.009073] LustreError: 35161:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 94 previous similar messages [3781992.051427] LustreError: 35222:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107919785280 granted:483067692, total:355282829208 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355282829208 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3781992.081968] LustreError: 35222:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 96 previous similar messages [3782592.108618] LustreError: 35166:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107913450072 granted:424654352, total:355282829208 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355282829208 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3782592.139154] LustreError: 35166:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 99 previous similar messages [3783203.577724] LustreError: 35143:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107830939800 granted:310379184, total:355282829208 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355282829208 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3783203.608393] LustreError: 35143:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 101 previous similar messages [3783807.817450] LustreError: 36661:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3783807.847298] LustreError: 36661:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 95 previous similar messages [3784416.059836] LustreError: 36667:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020c_UUID release: 18446621080987365160 granted:1028, total:134206344 qmt:oak-QMT0000 pool:dt-0x0 id:9826 enforced:1 hard:134217728 soft:0 granted:134206344 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:3510885 default:no [3784416.089346] LustreError: 36667:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 99 previous similar messages [3784422.898168] Lustre: oak-MDT0000: haven't heard from client 3719b005-a832-4633-9dfb-ab17cc7b9f99 (at 10.51.17.9@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902deda6e000, cur 1714048674 expire 1714048524 last 1714048447 [3784422.920043] Lustre: Skipped 7 previous similar messages [3785019.700919] LustreError: 35143:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107830939800 granted:310379184, total:355282829208 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355282829208 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3785019.731460] LustreError: 35143:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 101 previous similar messages [3785619.714142] LustreError: 36667:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107830939800 granted:310379184, total:355282829208 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355282829208 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3785619.744686] LustreError: 36667:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 97 previous similar messages [3786143.819465] Lustre: oak-MDT0003: haven't heard from client 4ba59845-8663-442b-9404-33dd2660aedc (at 10.50.8.17@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901ccc82b800, cur 1714050395 expire 1714050245 last 1714050168 [3786143.841338] Lustre: Skipped 1 previous similar message [3786148.818580] Lustre: oak-MDT0000: haven't heard from client 4ba59845-8663-442b-9404-33dd2660aedc (at 10.50.8.17@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901ccc86d000, cur 1714050400 expire 1714050250 last 1714050173 [3786219.930608] LustreError: 35227:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107913450072 granted:424654352, total:355282829208 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355282829208 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3786219.961151] LustreError: 35227:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 94 previous similar messages [3786824.058217] LustreError: 35202:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3786824.087894] LustreError: 35202:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 99 previous similar messages [3787424.789668] LustreError: 35151:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069505561852 granted:559613176, total:355282829208 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355282829208 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3787424.820212] LustreError: 35151:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 100 previous similar messages [3788030.321905] LustreError: 35144:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020c_UUID release: 18446621080987365160 granted:1028, total:134206344 qmt:oak-QMT0000 pool:dt-0x0 id:9826 enforced:1 hard:134217728 soft:0 granted:134206344 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:3510885 default:no [3788030.351416] LustreError: 35144:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 106 previous similar messages [3788043.730522] Lustre: oak-MDT0000: haven't heard from client c462af35-b658-4632-9e05-3f1cb0d142b1 (at 10.51.1.5@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90248606f000, cur 1714052295 expire 1714052145 last 1714052068 [3788630.980067] LustreError: 35235:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107913450072 granted:424654352, total:355282829208 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355282829208 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3788631.010605] LustreError: 35235:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 96 previous similar messages [3789236.945828] LustreError: 35201:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069505561852 granted:559613176, total:355282829208 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355282829208 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3789236.976367] LustreError: 35201:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 109 previous similar messages [3789841.149695] LustreError: 36667:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107830939800 granted:310379184, total:355282829208 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355282829208 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3789841.180236] LustreError: 36667:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 96 previous similar messages [3790441.370211] LustreError: 35179:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069505561848 granted:559613176, total:355282829208 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355282829208 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3790441.400756] LustreError: 35179:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 97 previous similar messages [3791046.075488] LustreError: 35167:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454509505672 granted:126893260, total:30071094428 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:30071094428 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3791046.075489] LustreError: 36683:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107919785280 granted:483067692, total:355282829208 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355282829208 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3791046.075492] LustreError: 36683:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 99 previous similar messages [3791633.569135] Lustre: oak-MDT0000: haven't heard from client 0218abe1-a7ae-4bb2-bb37-5dbbe04cfe83 (at 10.51.11.18@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900e682a4800, cur 1714055885 expire 1714055735 last 1714055658 [3791633.591096] Lustre: Skipped 1 previous similar message [3791648.156848] LustreError: 35158:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069505561844 granted:559613176, total:355282829208 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355282829208 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3791648.187386] LustreError: 35158:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 101 previous similar messages [3791969.550391] Lustre: oak-MDT0000: haven't heard from client 97eb5d59-f2d4-4789-bd13-0095d0306211 (at 10.50.13.12@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9005871a8800, cur 1714056221 expire 1714056071 last 1714055994 [3791969.572356] Lustre: Skipped 1 previous similar message [3792255.373100] LustreError: 36667:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3792255.402947] LustreError: 36667:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 91 previous similar messages [3792859.946210] LustreError: 36659:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3792859.976055] LustreError: 36659:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 94 previous similar messages [3793177.504439] Lustre: oak-MDT0000: haven't heard from client 90a23426-9a7d-41b4-8536-c15b9b1e0e65 (at 10.51.1.46@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90073d953400, cur 1714057429 expire 1714057279 last 1714057202 [3793177.526317] Lustre: Skipped 1 previous similar message [3793194.517850] Lustre: oak-MDT0003: haven't heard from client 90a23426-9a7d-41b4-8536-c15b9b1e0e65 (at 10.51.1.46@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9040fb397c00, cur 1714057446 expire 1714057296 last 1714057219 [3793466.015120] LustreError: 35152:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107830939800 granted:310379184, total:355282829208 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355282829208 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3793466.045665] LustreError: 35152:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 103 previous similar messages [3794074.851285] LustreError: 35174:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107913450072 granted:424654352, total:355298559264 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355298559264 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3794074.881822] LustreError: 35174:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 101 previous similar messages [3794675.333834] LustreError: 35148:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3794675.363505] LustreError: 35148:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 95 previous similar messages [3795277.154942] LustreError: 35237:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107913450072 granted:424654352, total:355298559264 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355298559264 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3795277.185589] LustreError: 35237:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 101 previous similar messages [3795881.074032] LustreError: 36655:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069505561836 granted:559613176, total:355298559264 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355298559264 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3795881.104578] LustreError: 36655:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 98 previous similar messages [3796495.624295] LustreError: 36687:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0209_UUID release: 16517233108325478064 granted:230227968, total:30071094428 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:30071094428 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3796495.654656] LustreError: 36687:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 99 previous similar messages [3797099.172327] LustreError: 35208:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069505561836 granted:559613176, total:355298559264 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355298559264 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3797099.202869] LustreError: 35208:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 90 previous similar messages [3797707.890021] LustreError: 36669:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107936226032 granted:486160868, total:355298559264 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355298559264 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3797707.920562] LustreError: 36669:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 102 previous similar messages [3798308.922565] LustreError: 35197:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020c_UUID release: 18446621080987365160 granted:1028, total:134206344 qmt:oak-QMT0000 pool:dt-0x0 id:9826 enforced:1 hard:134217728 soft:0 granted:134206344 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:3510885 default:no [3798308.952062] LustreError: 35197:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 102 previous similar messages [3798373.260472] Lustre: oak-MDT0000: haven't heard from client fa8d924b-2cf7-41d0-ae4a-e6d434b2867e (at 10.50.7.38@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900343ba3c00, cur 1714062625 expire 1714062475 last 1714062398 [3798922.277373] LustreError: 35144:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107830939800 granted:310379184, total:355298559264 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355298559264 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3798922.307909] LustreError: 35144:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 94 previous similar messages [3799529.274375] LustreError: 35143:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020c_UUID release: 18446621080987365160 granted:1028, total:134206344 qmt:oak-QMT0000 pool:dt-0x0 id:9826 enforced:1 hard:134217728 soft:0 granted:134206344 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:3510885 default:no [3799529.304000] LustreError: 35143:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 97 previous similar messages [3799655.037388] LustreError: 31496:0:(ldlm_lockd.c:261:expired_lock_main()) ### lock callback timer expired after 112s: evicting client at 10.210.12.8@tcp1 ns: mdt-oak-MDT0003_UUID lock: ffff903eb3a03840/0x1e62cae8f1373594 lrc: 4/0,0 mode: PR/PR res: [0x32c00202e3:0xc8:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x60200400000020 nid: 10.210.12.8@tcp1 remote: 0xf2bf6ad2c92716f9 expref: 18 pid: 44812 timeout: 3799828 lvb_type: 0 [3799677.195309] Lustre: oak-MDT0003: haven't heard from client 497516a5-2729-8277-d494-775516f7d55b (at 10.210.12.9@tcp1) in 163 seconds. I think it's dead, and I am evicting it. exp ffff9019c831f800, cur 1714063929 expire 1714063779 last 1714063766 [3799677.217187] Lustre: Skipped 1 previous similar message [3800133.963362] LustreError: 35147:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107830939800 granted:310379184, total:355298559264 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355298559264 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3800133.993913] LustreError: 35147:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 95 previous similar messages [3800736.996698] LustreError: 36682:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107913450072 granted:424654352, total:355298559264 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355298559264 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3800737.027232] LustreError: 36682:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 99 previous similar messages [3801259.121930] Lustre: oak-MDT0003: haven't heard from client 834d76eb-7c87-4a88-a3d3-000477876223 (at 10.51.10.67@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9006e72e9c00, cur 1714065511 expire 1714065361 last 1714065284 [3801259.144048] Lustre: Skipped 2 previous similar messages [3801338.608705] LustreError: 35180:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069505561828 granted:559613176, total:355298559264 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355298559264 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3801338.639248] LustreError: 35180:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 91 previous similar messages [3801943.450558] LustreError: 35173:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107913450072 granted:424654352, total:355298559264 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355298559264 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3801943.481101] LustreError: 35173:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 99 previous similar messages [3801954.093178] Lustre: oak-MDT0003: haven't heard from client eb725ad4-e73c-49ca-b125-a3d3b7031a60 (at 10.51.3.24@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901b3ca1e400, cur 1714066206 expire 1714066056 last 1714065979 [3801954.115233] Lustre: Skipped 1 previous similar message [3802544.463647] LustreError: 36699:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107919785280 granted:483067692, total:355298559264 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355298559264 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3802544.494189] LustreError: 36699:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 95 previous similar messages [3803158.584505] LustreError: 36657:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107936226032 granted:486160868, total:355298559264 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355298559264 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3803158.615051] LustreError: 36657:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 101 previous similar messages [3803762.220761] LustreError: 36655:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107936226032 granted:486160868, total:355298559264 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355298559264 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3803762.251298] LustreError: 36655:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 105 previous similar messages [3804370.854440] LustreError: 35172:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107913450072 granted:424654352, total:355298559264 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355298559264 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3804370.884980] LustreError: 35172:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 95 previous similar messages [3804971.654195] LustreError: 35163:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107913450072 granted:424654352, total:355298559264 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355298559264 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3804971.684738] LustreError: 35163:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 92 previous similar messages [3805064.945328] Lustre: oak-MDT0000: haven't heard from client 97a1fa98-002b-4b9a-aa7e-bd61595ced3e (at 10.51.9.19@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903635caf400, cur 1714069317 expire 1714069167 last 1714069090 [3805064.967205] Lustre: Skipped 1 previous similar message [3805347.932298] Lustre: oak-MDT0003: haven't heard from client 91bd7f21-6aa1-42dc-b88e-7bccded4f449 (at 10.50.3.69@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903a4a9c5c00, cur 1714069600 expire 1714069450 last 1714069373 [3805347.954167] Lustre: Skipped 7 previous similar messages [3805552.923635] Lustre: oak-MDT0000: haven't heard from client 0cc834ae-a4c7-4375-88b4-110b4873c174 (at 10.50.7.38@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902086d33400, cur 1714069805 expire 1714069655 last 1714069578 [3805552.945512] Lustre: Skipped 11 previous similar messages [3805571.753508] LustreError: 35219:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107919785280 granted:483067692, total:355298559264 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355298559264 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3805571.784048] LustreError: 35219:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 97 previous similar messages [3806158.893713] Lustre: oak-MDT0000: haven't heard from client 40bf71c1-971d-403a-97eb-799637d17f54 (at 10.50.4.32@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903f7c52c000, cur 1714070411 expire 1714070261 last 1714070184 [3806158.915586] Lustre: Skipped 19 previous similar messages [3806172.517907] LustreError: 35221:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454509505672 granted:126893260, total:30071094428 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:30071094428 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3806172.517908] LustreError: 36689:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107919785280 granted:483067692, total:355298559264 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355298559264 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3806172.517911] LustreError: 36689:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 104 previous similar messages [3806173.900611] Lustre: oak-MDT0003: haven't heard from client 40bf71c1-971d-403a-97eb-799637d17f54 (at 10.50.4.32@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902de3dea000, cur 1714070426 expire 1714070276 last 1714070199 [3806173.922488] Lustre: Skipped 7 previous similar messages [3806234.898068] Lustre: oak-MDT0000: haven't heard from client b6aaf080-75ce-466c-b80c-111aef24addc (at 10.50.4.9@o2ib2) in 211 seconds. I think it's dead, and I am evicting it. exp ffff9006734bc000, cur 1714070487 expire 1714070337 last 1714070276 [3806234.919863] Lustre: Skipped 7 previous similar messages [3806772.596928] LustreError: 35161:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454509505672 granted:126893260, total:30071094428 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:30071094428 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3806772.627211] LustreError: 35161:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 102 previous similar messages [3807373.707154] LustreError: 35218:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107913450072 granted:424654352, total:355298559264 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355298559264 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3807373.737691] LustreError: 35218:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 99 previous similar messages [3807974.933425] LustreError: 36650:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069505561812 granted:559613176, total:355298559264 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355298559264 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3807974.963968] LustreError: 36650:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 94 previous similar messages [3808131.803140] Lustre: oak-MDT0000: haven't heard from client 06e15ef7-33c8-4574-87c1-112c9ec3826c (at 10.51.7.70@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9007196f3000, cur 1714072384 expire 1714072234 last 1714072157 [3808131.825017] Lustre: Skipped 5 previous similar messages [3808577.753354] LustreError: 36655:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107830939800 granted:310379184, total:355314289600 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355314289600 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3808577.783899] LustreError: 36655:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 100 previous similar messages [3809195.060627] LustreError: 36665:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107830939800 granted:310379184, total:355314289600 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355314289600 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3809195.091168] LustreError: 36665:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 106 previous similar messages [3809797.362330] LustreError: 35229:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107913450072 granted:424654352, total:355314289600 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355314289600 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3809797.392865] LustreError: 35229:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 106 previous similar messages [3810308.711379] Lustre: oak-MDT0000: haven't heard from client 8df49d9e-0c78-429a-9587-d732b3ada88f (at 10.51.17.7@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90307644a000, cur 1714074561 expire 1714074411 last 1714074334 [3810308.733254] Lustre: Skipped 1 previous similar message [3810318.706329] Lustre: oak-MDT0003: haven't heard from client 8df49d9e-0c78-429a-9587-d732b3ada88f (at 10.51.17.7@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903b3cdad400, cur 1714074571 expire 1714074421 last 1714074344 [3810407.286767] LustreError: 35207:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069505561804 granted:559613176, total:355314289600 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355314289600 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3810407.317492] LustreError: 35207:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 97 previous similar messages [3811014.525844] LustreError: 36682:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0209_UUID release: 16517233108325478064 granted:230227968, total:30071094428 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:30071094428 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3811014.556208] LustreError: 36682:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 101 previous similar messages [3811626.391592] LustreError: 35147:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020c_UUID release: 18446621080987365160 granted:1028, total:134206344 qmt:oak-QMT0000 pool:dt-0x0 id:9826 enforced:1 hard:134217728 soft:0 granted:134206344 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:3510885 default:no [3811626.421102] LustreError: 35147:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 97 previous similar messages [3812245.329265] LustreError: 36697:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0209_UUID release: 16517233108325478064 granted:230227968, total:30071094428 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:30071094428 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3812245.359911] LustreError: 36697:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 102 previous similar messages [3812603.598436] Lustre: oak-MDT0000: haven't heard from client 17b7b8c6-0ab4-4b63-b3d3-8ce7f0c01170 (at 10.51.7.67@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9023c4bc8c00, cur 1714076856 expire 1714076706 last 1714076629 [3812846.222777] LustreError: 36670:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069505561796 granted:559613176, total:355314289600 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355314289600 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3812846.253320] LustreError: 36670:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 101 previous similar messages [3813452.559550] LustreError: 36656:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3813452.589224] LustreError: 36656:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 101 previous similar messages [3814053.083827] LustreError: 35178:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069505561792 granted:559613176, total:355313241024 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355313241024 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3814053.114370] LustreError: 35178:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 108 previous similar messages [3814654.579964] LustreError: 35153:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3814654.609792] LustreError: 35153:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 99 previous similar messages [3814735.642641] LNet: Closing conn to 10.0.3.101@o2ib5: error -116 [3814735.648675] LNet: 35749:0:(o2iblnd_cb.c:2638:kiblnd_passive_connect()) Conn stale 10.0.3.101@o2ib5 version 12/12 incarnation 1711694573056543/1714078977102776 [3815254.986166] LustreError: 36675:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107919366220 granted:483067692, total:355313241024 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355313241024 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3815255.016828] LustreError: 36675:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 208 previous similar messages [3815859.451780] LustreError: 36662:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069505561788 granted:559613176, total:355313241024 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355313241024 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3815859.482317] LustreError: 36662:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 217 previous similar messages [3816117.239471] LustreError: 44820:0:(mdt_open.c:1217:mdt_cross_open()) oak-MDT0003: [0x32c0020c91:0x13147:0x0] doesn't exist!: rc = -14 [3816467.460839] LustreError: 36655:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3816467.490519] LustreError: 36655:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 304 previous similar messages [3816854.396861] Lustre: oak-MDT0003: haven't heard from client 33b58f49-b0f3-4f2d-8b8d-7dd1952f85e2 (at 10.50.10.65@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9004da865800, cur 1714081107 expire 1714080957 last 1714080880 [3816854.418827] Lustre: Skipped 1 previous similar message [3817069.428656] LustreError: 35147:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069505560972 granted:559613176, total:355313241024 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355313241024 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3817069.459330] LustreError: 35147:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 131 previous similar messages [3817075.387448] Lustre: oak-MDT0000: haven't heard from client c36058c1-dd13-436c-8560-a760ed6a5ad1 (at 10.50.8.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903d258e9400, cur 1714081328 expire 1714081178 last 1714081101 [3817075.409238] Lustre: Skipped 9 previous similar messages [3817246.378501] Lustre: oak-MDT0000: haven't heard from client ecb4cfaa-c312-4c90-be38-16da0e766437 (at 10.50.1.10@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900e3a50ec00, cur 1714081499 expire 1714081349 last 1714081272 [3817246.400381] Lustre: Skipped 1 previous similar message [3817252.378208] Lustre: oak-MDT0003: haven't heard from client ecb4cfaa-c312-4c90-be38-16da0e766437 (at 10.50.1.10@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901e72e88800, cur 1714081505 expire 1714081355 last 1714081278 [3817672.443453] LustreError: 35176:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107830939904 granted:310379184, total:355328973744 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355328973744 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3817672.473997] LustreError: 35176:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 98 previous similar messages [3818276.455094] LustreError: 36665:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020c_UUID release: 18446621080987365160 granted:1028, total:134206344 qmt:oak-QMT0000 pool:dt-0x0 id:9826 enforced:1 hard:134217728 soft:0 granted:134206344 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:3510885 default:no [3818276.484596] LustreError: 36665:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 95 previous similar messages [3818875.302709] Lustre: oak-MDT0003: haven't heard from client 265af49c-0da0-4824-85d9-b2516bc9cea6 (at 10.50.13.2@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9003684cd000, cur 1714083128 expire 1714082978 last 1714082901 [3818880.790999] LustreError: 35155:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107830939904 granted:310379184, total:355344705884 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355344705884 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3818880.821694] LustreError: 35155:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 106 previous similar messages [3818890.306335] Lustre: oak-MDT0000: haven't heard from client 265af49c-0da0-4824-85d9-b2516bc9cea6 (at 10.50.13.2@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901d5ad6a400, cur 1714083143 expire 1714082993 last 1714082916 [3818890.329947] LustreError: 36604:0:(client.c:1256:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff90080bc3b180 x1793468154865088/t0(0) o104->oak-MDT0000@10.50.13.2@o2ib2:15/16 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' [3818890.351825] LustreError: 36604:0:(client.c:1256:ptlrpc_import_delay_req()) Skipped 2 previous similar messages [3819493.611474] LustreError: 35147:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3819493.641320] LustreError: 35147:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 477 previous similar messages [3820095.832127] LustreError: 35180:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107830208380 granted:310379184, total:355359387548 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355359387548 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3820095.862666] LustreError: 35180:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 102 previous similar messages [3820696.662082] LustreError: 36659:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3820696.691928] LustreError: 36659:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 474 previous similar messages [3820732.217378] Lustre: oak-MDT0003: haven't heard from client 05bf45c3-5c25-4704-828e-454bc2c38ae9 (at 10.50.5.13@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903adff13000, cur 1714084985 expire 1714084835 last 1714084758 [3821297.225965] LustreError: 35179:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069505560936 granted:559613176, total:355359387548 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355359387548 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3821297.256504] LustreError: 35179:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 126 previous similar messages [3821899.122561] LustreError: 36692:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107918979188 granted:483067692, total:355359387548 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355359387548 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3821899.153101] LustreError: 36692:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 418 previous similar messages [3822500.392864] LustreError: 35200:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107936225932 granted:486160868, total:355359387548 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355359387548 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3822500.423402] LustreError: 35200:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 269 previous similar messages [3822701.123326] Lustre: oak-MDT0003: haven't heard from client 94ad8bfe-3bac-47b9-85a9-f5a481098b6b (at 10.51.3.7@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903302c45400, cur 1714086954 expire 1714086804 last 1714086727 [3822701.145108] Lustre: Skipped 1 previous similar message [3822703.123258] Lustre: oak-MDT0000: haven't heard from client 141e74be-fe4f-4f22-b895-ec8f89711efa (at 10.51.3.1@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9032b3c0e000, cur 1714086956 expire 1714086806 last 1714086729 [3822703.145043] Lustre: Skipped 1 previous similar message [3823101.337634] LustreError: 36670:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069504709296 granted:559613176, total:355359387548 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355359387548 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3823101.368184] LustreError: 36670:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 518 previous similar messages [3823161.601283] LNet: Tx -> 10.0.3.119@o2ib5 cookie 0x5ca0fcdd1 sending 1 waiting 0: failed 12 [3823161.609731] LNet: Closing conn to 10.0.3.119@o2ib5: error -5(waiting) [3823161.609734] LNet: Rx from 10.0.3.119@o2ib5 failed: 5 [3823252.098424] Lustre: oak-MDT0003: haven't heard from client 4e6b8d98-6c09-4382-821f-631bb0600f72 (at 10.0.3.119@o2ib5) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901271580c00, cur 1714087505 expire 1714087355 last 1714087278 [3823252.120294] Lustre: Skipped 1 previous similar message [3823701.729205] LustreError: 36660:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069504708136 granted:559613176, total:355375116496 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355375116496 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3823701.759941] LustreError: 36660:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 733 previous similar messages [3823757.075173] Lustre: oak-MDT0003: haven't heard from client 483e78b0-28de-426c-85ab-066cae7b9b98 (at 10.51.3.32@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900496063c00, cur 1714088010 expire 1714087860 last 1714087783 [3823757.097048] Lustre: Skipped 1 previous similar message [3823762.078590] Lustre: oak-MDT0000: haven't heard from client 483e78b0-28de-426c-85ab-066cae7b9b98 (at 10.51.3.32@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90110c0fcc00, cur 1714088015 expire 1714087865 last 1714087788 [3824302.064952] LustreError: 36658:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069505554288 granted:559613176, total:355388797292 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355388797292 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3824302.095494] LustreError: 36658:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 437 previous similar messages [3824903.063496] LustreError: 35145:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069505552420 granted:559613176, total:355449618892 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355449618892 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3824903.094039] LustreError: 35145:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 428 previous similar messages [3825503.473389] LustreError: 36680:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0209_UUID release: 16517233108325478064 granted:230227968, total:30071094428 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:30071094428 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3825503.504083] LustreError: 36680:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 525 previous similar messages [3826105.731855] LustreError: 36661:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107829248268 granted:310379184, total:355491627964 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355491627964 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3826105.762396] LustreError: 36661:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 736 previous similar messages [3826709.206887] LustreError: 35153:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107829308448 granted:310379184, total:355475433920 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355475433920 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3826709.237429] LustreError: 35153:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 306 previous similar messages [3827312.438341] LustreError: 36669:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107935190892 granted:486160868, total:355474385344 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355474385344 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3827312.468884] LustreError: 36669:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 626 previous similar messages [3827912.832836] LustreError: 35207:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107935223096 granted:486160868, total:355474385344 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355474385344 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3827912.863508] LustreError: 35207:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 2222 previous similar messages [3828264.864394] Lustre: oak-MDT0003: haven't heard from client ee73734e-01a0-4e27-a350-02bfd79a1e2f (at 10.51.9.69@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903ab6166400, cur 1714092518 expire 1714092368 last 1714092291 [3828271.864988] Lustre: oak-MDT0000: haven't heard from client 1f1b4278-5659-4f19-9679-32a4d96c5818 (at 10.51.9.68@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901bf3fd4800, cur 1714092525 expire 1714092375 last 1714092298 [3828271.886859] Lustre: Skipped 1 previous similar message [3828513.134481] LustreError: 36676:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107918950796 granted:483067692, total:355474385344 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355474385344 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3828513.165150] LustreError: 36676:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 367 previous similar messages [3829113.846209] LustreError: 35205:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3829113.876070] LustreError: 35205:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 680 previous similar messages [3829715.260431] LustreError: 36693:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107918885164 granted:483067692, total:355490117308 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355490117308 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3829715.290970] LustreError: 36693:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 840 previous similar messages [3830317.011305] LustreError: 36668:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107935230044 granted:486160868, total:355490117308 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355490117308 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3830317.041843] LustreError: 36668:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 435 previous similar messages [3830920.071866] LustreError: 35158:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107822001248 granted:310379184, total:355490117308 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355490117308 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3830920.102409] LustreError: 35158:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 587 previous similar messages [3831053.736469] Lustre: oak-MDT0000: haven't heard from client 04bf6ced-3c7a-34e7-b5b8-694aae231544 (at 10.210.12.7@tcp1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900e2495d400, cur 1714095307 expire 1714095157 last 1714095080 [3831053.758347] Lustre: Skipped 1 previous similar message [3831070.736455] Lustre: oak-MDT0003: haven't heard from client 04bf6ced-3c7a-34e7-b5b8-694aae231544 (at 10.210.12.7@tcp1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900e2495b800, cur 1714095324 expire 1714095174 last 1714095097 [3831521.751823] LustreError: 31507:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0209_UUID release: 16517233108325478064 granted:230227968, total:30085779576 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:30085779576 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3831521.782196] LustreError: 31507:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 421 previous similar messages [3832122.941670] LustreError: 35185:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107913530248 granted:424654352, total:355490117308 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355490117308 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3832122.972209] LustreError: 35185:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 647 previous similar messages [3832723.916307] LustreError: 35195:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069503713420 granted:559613176, total:355490117308 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355490117308 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3832723.946853] LustreError: 35195:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 482 previous similar messages [3833329.243064] LustreError: 36683:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107913523632 granted:424654352, total:355490117308 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355490117308 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3833329.273602] LustreError: 36683:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 386 previous similar messages [3833930.979334] LustreError: 35204:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0220_UUID release: 18446621107811400488 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3833931.009004] LustreError: 35204:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 447 previous similar messages [3834532.837457] LustreError: 35197:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107821947352 granted:310379184, total:355490117308 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355490117308 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3834532.867996] LustreError: 35197:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 424 previous similar messages [3835134.597209] LustreError: 36679:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0209_UUID release: 16517233108325478064 granted:230227968, total:30085779576 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:30085779576 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3835134.627574] LustreError: 36679:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 324 previous similar messages [3835736.025399] LustreError: 35221:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107918921428 granted:483067692, total:355490117308 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355490117308 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3835736.055936] LustreError: 35221:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 445 previous similar messages [3835994.824309] LustreError: 44902:0:(mdt_open.c:1217:mdt_cross_open()) oak-MDT0003: [0x32c0020c91:0x1eb60:0x0] doesn't exist!: rc = -14 [3836337.969374] LustreError: 36685:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107918914576 granted:483067692, total:355520529580 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355520529580 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3836337.999922] LustreError: 36685:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 685 previous similar messages [3836938.070499] LustreError: 35158:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069503663064 granted:559613176, total:355520529580 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355520529580 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3836938.101200] LustreError: 35158:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 415 previous similar messages [3837540.717177] LustreError: 35223:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107913469392 granted:424654352, total:355520529580 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355520529580 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3837540.747711] LustreError: 35223:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 393 previous similar messages [3838141.470640] LustreError: 36697:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107918900208 granted:483067692, total:355520529580 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355520529580 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3838141.501172] LustreError: 36697:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 324 previous similar messages [3838741.445525] LustreError: 35154:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069503669460 granted:559613176, total:355520529580 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355520529580 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3838741.476068] LustreError: 35154:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 637 previous similar messages [3839342.353249] LustreError: 35195:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069503662956 granted:559613176, total:355520529580 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355520529580 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3839342.383789] LustreError: 35195:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 670 previous similar messages [3839945.341533] LustreError: 35226:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107913475784 granted:424654352, total:355520529580 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355520529580 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3839945.372073] LustreError: 35226:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 512 previous similar messages [3840067.319257] Lustre: oak-MDT0003: haven't heard from client f623b1a5-488c-43c1-a25e-061c7ec8804f (at 10.51.12.8@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9033870e5800, cur 1714104321 expire 1714104171 last 1714104094 [3840262.307566] Lustre: oak-MDT0000: haven't heard from client 8fa1eb23-136a-430a-b73f-8e9c6f425cdb (at 10.51.7.68@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900463940c00, cur 1714104516 expire 1714104366 last 1714104289 [3840262.329444] Lustre: Skipped 1 previous similar message [3840546.300444] LustreError: 36653:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107935134008 granted:486160868, total:355520529580 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355520529580 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3840546.330988] LustreError: 36653:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 586 previous similar messages [3841148.086546] LustreError: 35163:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0209_UUID release: 16517233108325478064 granted:230227968, total:30085779576 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:30085779576 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3841148.116919] LustreError: 35163:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 471 previous similar messages [3841305.265956] Lustre: oak-MDT0000: haven't heard from client 7c0e0c7c-be10-4ed3-ae38-ac03e63efde0 (at 10.51.2.68@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9033870e5000, cur 1714105559 expire 1714105409 last 1714105332 [3841305.287838] Lustre: Skipped 1 previous similar message [3841753.734148] LustreError: 35177:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107828355024 granted:310379184, total:355520529580 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355520529580 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3841753.764695] LustreError: 35177:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 501 previous similar messages [3842354.062976] LustreError: 36681:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107918835812 granted:483067692, total:355520529580 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355520529580 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3842354.093515] LustreError: 36681:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 610 previous similar messages [3842954.200104] LustreError: 35229:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107918860048 granted:483067692, total:355520529580 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355520529580 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3842954.230642] LustreError: 35229:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 620 previous similar messages [3843556.177856] LustreError: 35147:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107828320624 granted:310379184, total:355535263668 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355535263668 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3843556.208399] LustreError: 35147:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 499 previous similar messages [3844157.181394] LustreError: 35203:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069503630284 granted:559613176, total:355535263668 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355535263668 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3844157.211940] LustreError: 35203:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 933 previous similar messages [3844242.126834] Lustre: oak-MDT0000: haven't heard from client 38d4bf10-d9a6-4fc0-8c71-0dd6666d3ae7 (at 10.51.14.21@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900c6a4a7000, cur 1714108496 expire 1714108346 last 1714108269 [3844242.148797] Lustre: Skipped 1 previous similar message [3844757.596341] LustreError: 35206:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107935149464 granted:486160868, total:355535263668 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355535263668 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3844757.626891] LustreError: 35206:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 771 previous similar messages [3845358.943209] LustreError: 36686:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107918839488 granted:483067692, total:355535263668 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355535263668 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3845358.973743] LustreError: 36686:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 637 previous similar messages [3845960.082988] LustreError: 35166:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107918875044 granted:483067692, total:355535263668 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355535263668 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3845960.113525] LustreError: 35166:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 661 previous similar messages [3846561.454150] LustreError: 36653:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069503619616 granted:559613176, total:355535263668 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355535263668 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3846561.484695] LustreError: 36653:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 730 previous similar messages [3847161.715890] LustreError: 35154:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107935149288 granted:486160868, total:355535263668 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355535263668 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3847161.746430] LustreError: 35154:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 1063 previous similar messages [3847762.128568] LustreError: 36666:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107935132720 granted:486160868, total:355535263668 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355535263668 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3847762.159114] LustreError: 36666:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 712 previous similar messages [3848363.660337] LustreError: 36669:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107935114308 granted:486160868, total:355535263668 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355535263668 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3848363.690891] LustreError: 36669:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 702 previous similar messages [3848963.837175] LustreError: 36656:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107828317156 granted:310379184, total:355535263668 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355535263668 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3848963.867717] LustreError: 36656:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 627 previous similar messages [3849117.920296] Lustre: oak-MDT0003: Client d38d6d15-66cf-9803-e9cb-790d8dfe2495 (at 10.210.12.107@tcp1) reconnecting [3849117.930730] Lustre: Skipped 3 previous similar messages [3849563.968419] LustreError: 35195:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107828311736 granted:310379184, total:355535263668 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355535263668 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3849563.998964] LustreError: 35195:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 564 previous similar messages [3849670.871296] Lustre: oak-MDT0003: haven't heard from client 142c7c44-9043-4ff0-8ae9-7d241254b95b (at 10.51.4.48@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902ad0efa800, cur 1714113925 expire 1714113775 last 1714113698 [3849670.893166] Lustre: Skipped 1 previous similar message [3850165.563105] LustreError: 35205:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107828292608 granted:310379184, total:355535263668 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355535263668 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3850165.593650] LustreError: 35205:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 438 previous similar messages [3850768.107911] LustreError: 36677:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107918847156 granted:483067692, total:355535263668 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355535263668 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3850768.138455] LustreError: 36677:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 505 previous similar messages [3851371.443054] LustreError: 36660:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069503646008 granted:559613176, total:355535263668 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355535263668 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3851371.473596] LustreError: 36660:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 562 previous similar messages [3851877.768599] Lustre: oak-MDT0000: haven't heard from client c49b729f-bc51-4f25-a562-de036437ed78 (at 10.51.1.59@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901190e84000, cur 1714116132 expire 1714115982 last 1714115905 [3851877.790481] Lustre: Skipped 1 previous similar message [3851973.080288] LustreError: 35218:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 3475144454509496284 granted:126893260, total:30085779576 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:30085779576 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3851973.110569] LustreError: 35218:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 603 previous similar messages [3852084.759150] Lustre: oak-MDT0003: haven't heard from client b1c9c965-0338-4547-9aec-4787ead609ee (at 10.51.14.20@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902c2be7d400, cur 1714116339 expire 1714116189 last 1714116112 [3852084.781108] Lustre: Skipped 1 previous similar message [3852400.746780] Lustre: oak-MDT0003: haven't heard from client bd10b825-ca57-43ec-8260-9c7f25611813 (at 10.51.12.11@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900c378c8c00, cur 1714116655 expire 1714116505 last 1714116428 [3852400.768738] Lustre: Skipped 1 previous similar message [3852573.224003] LustreError: 35167:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107913362552 granted:424654352, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3852573.254547] LustreError: 35167:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 611 previous similar messages [3852706.728120] Lustre: oak-MDT0003: haven't heard from client 45302401-a238-410c-a4d2-3a1dee457a87 (at 10.51.3.59@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9028aab6c800, cur 1714116961 expire 1714116811 last 1714116734 [3852706.750097] Lustre: Skipped 1 previous similar message [3853173.710356] LustreError: 35151:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107828303808 granted:310379184, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3853173.740908] LustreError: 35151:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 515 previous similar messages [3853775.223608] LustreError: 35180:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107935091596 granted:486160868, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3853775.254288] LustreError: 35180:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 338 previous similar messages [3854379.436650] LustreError: 35181:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107935104772 granted:486160868, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3854379.467312] LustreError: 35181:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 475 previous similar messages [3854980.822938] LustreError: 35169:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107913398996 granted:424654352, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3854980.853476] LustreError: 35169:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 641 previous similar messages [3855084.626330] Lustre: oak-MDT0003: haven't heard from client 4a869b6e-5150-4270-843e-b40e36feaf40 (at 10.51.2.50@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902286217000, cur 1714119339 expire 1714119189 last 1714119112 [3855084.648204] Lustre: Skipped 1 previous similar message [3855581.194727] LustreError: 35177:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107828285884 granted:310379184, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3855581.225273] LustreError: 35177:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 518 previous similar messages [3856182.782336] LustreError: 35175:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107913398132 granted:424654352, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3856182.812873] LustreError: 35175:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 540 previous similar messages [3856783.562127] LustreError: 36677:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107918821648 granted:483067692, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3856783.592658] LustreError: 36677:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 447 previous similar messages [3857385.995030] LustreError: 35234:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107918828896 granted:483067692, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3857386.025662] LustreError: 35234:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 462 previous similar messages [3857989.468986] LustreError: 36696:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107913418840 granted:424654352, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3857989.499530] LustreError: 36696:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 603 previous similar messages [3858589.821752] LustreError: 36688:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107913395892 granted:424654352, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3858589.852288] LustreError: 36688:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 528 previous similar messages [3859190.912606] LustreError: 36683:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107913387948 granted:424654352, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3859190.943143] LustreError: 36683:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 903 previous similar messages [3859791.322980] LustreError: 35164:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107918813008 granted:483067692, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3859791.353513] LustreError: 35164:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 961 previous similar messages [3860391.693197] LustreError: 35155:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020c_UUID release: 18446621080987365160 granted:1028, total:134206344 qmt:oak-QMT0000 pool:dt-0x0 id:9826 enforced:1 hard:134217728 soft:0 granted:134206344 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:3510885 default:no [3860391.722701] LustreError: 35155:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 1061 previous similar messages [3860991.850973] LustreError: 35158:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107935083660 granted:486160868, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3860991.881511] LustreError: 35158:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 646 previous similar messages [3861592.709920] LustreError: 35170:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107913395860 granted:424654352, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3861592.740465] LustreError: 35170:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 822 previous similar messages [3862193.241212] LustreError: 36664:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107828236296 granted:310379184, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3862193.271756] LustreError: 36664:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 698 previous similar messages [3862795.822207] LustreError: 35149:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069503569964 granted:559613176, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3862795.852749] LustreError: 35149:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 623 previous similar messages [3863398.447644] LustreError: 31510:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069503577860 granted:559613176, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3863398.478193] LustreError: 31510:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 629 previous similar messages [3863999.596677] LustreError: 36666:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107828256004 granted:310379184, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3863999.627416] LustreError: 36666:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 755 previous similar messages [3864601.173633] LustreError: 36696:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107913384924 granted:424654352, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3864601.204174] LustreError: 36696:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 621 previous similar messages [3865201.751624] LustreError: 35176:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107935084308 granted:486160868, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3865201.782169] LustreError: 35176:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 649 previous similar messages [3865805.261562] LustreError: 35205:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107828198040 granted:310379184, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3865805.292106] LustreError: 35205:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 913 previous similar messages [3866255.107089] Lustre: oak-MDT0003: haven't heard from client 98b94eab-ebe6-4330-a59c-752aa87aaacd (at 10.51.0.13@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900d6f380000, cur 1714130510 expire 1714130360 last 1714130283 [3866255.128968] Lustre: Skipped 1 previous similar message [3866270.111231] Lustre: oak-MDT0000: haven't heard from client 98b94eab-ebe6-4330-a59c-752aa87aaacd (at 10.51.0.13@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9020f8c0f800, cur 1714130525 expire 1714130375 last 1714130298 [3866405.487389] LustreError: 36686:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107913385168 granted:424654352, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3866405.517924] LustreError: 36686:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 665 previous similar messages [3867006.231035] LustreError: 35177:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069503563216 granted:559613176, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3867006.261577] LustreError: 35177:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 666 previous similar messages [3867608.438064] LustreError: 35206:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107828197748 granted:310379184, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3867608.468605] LustreError: 35206:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 715 previous similar messages [3868208.536993] LustreError: 36661:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107935061152 granted:486160868, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3868208.567682] LustreError: 36661:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 872 previous similar messages [3868809.310717] LustreError: 35158:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107828235524 granted:310379184, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3868809.341253] LustreError: 35158:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 710 previous similar messages [3869411.033113] LustreError: 35196:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107935042224 granted:486160868, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3869411.063651] LustreError: 35196:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 633 previous similar messages [3870011.198305] LustreError: 36671:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107828193184 granted:310379184, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3870011.228853] LustreError: 36671:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 807 previous similar messages [3870246.916629] Lustre: oak-MDT0000: haven't heard from client 92cc86fa-bf2a-4fc7-8881-6ed97ca40101 (at 10.51.2.66@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90322f0c3000, cur 1714134502 expire 1714134352 last 1714134275 [3870611.398047] LustreError: 35207:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069503558304 granted:559613176, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3870611.428586] LustreError: 35207:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 733 previous similar messages [3871212.724833] LustreError: 36665:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107935038280 granted:486160868, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3871212.755381] LustreError: 36665:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 893 previous similar messages [3871823.554260] LustreError: 36655:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069503564568 granted:559613176, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3871823.584801] LustreError: 36655:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 703 previous similar messages [3872425.285379] LustreError: 35175:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107918727756 granted:483067692, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3872425.316038] LustreError: 35175:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 596 previous similar messages [3873025.488312] LustreError: 36678:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107918738492 granted:483067692, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3873025.518857] LustreError: 36678:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 917 previous similar messages [3873625.644791] LustreError: 35169:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0209_UUID release: 16517233108324758056 granted:230227968, total:30101624804 qmt:oak-QMT0000 pool:dt-0x0 id:13812 enforced:1 hard:60000000000 soft:0 granted:30101624804 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3873625.675151] LustreError: 35169:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 949 previous similar messages [3874225.763618] LustreError: 35204:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069503514948 granted:559613176, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3874225.794242] LustreError: 35204:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 1052 previous similar messages [3874827.249942] LustreError: 35207:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107828193828 granted:310379184, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3874827.280484] LustreError: 35207:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 773 previous similar messages [3875427.448249] LustreError: 35195:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107828158212 granted:310379184, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3875427.478789] LustreError: 35195:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 731 previous similar messages [3876028.430409] LustreError: 35171:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107918715744 granted:483067692, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3876028.460947] LustreError: 35171:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 787 previous similar messages [3876629.513506] LustreError: 36695:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107913277388 granted:424654352, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3876629.544040] LustreError: 36695:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 1119 previous similar messages [3877232.576220] LustreError: 36653:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107828182384 granted:310379184, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3877232.606766] LustreError: 36653:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 825 previous similar messages [3877658.582490] Lustre: oak-MDT0003: haven't heard from client e88c0e55-5851-4b16-9449-d3332c21014b (at 10.50.13.2@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901d79323c00, cur 1714141914 expire 1714141764 last 1714141687 [3877658.604371] Lustre: Skipped 1 previous similar message [3877669.587768] Lustre: oak-MDT0000: haven't heard from client e88c0e55-5851-4b16-9449-d3332c21014b (at 10.50.13.2@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9011322c2400, cur 1714141925 expire 1714141775 last 1714141698 [3877834.954873] LustreError: 35225:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107913331252 granted:424654352, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3877834.985415] LustreError: 35225:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 876 previous similar messages [3878436.185580] LustreError: 35147:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107935005568 granted:486160868, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3878436.216117] LustreError: 35147:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 949 previous similar messages [3879037.044490] LustreError: 35224:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107913279352 granted:424654352, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3879037.075027] LustreError: 35224:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 1052 previous similar messages [3879638.014158] LustreError: 35200:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3879638.044006] LustreError: 35200:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 776 previous similar messages [3879842.474917] Lustre: oak-MDT0003: haven't heard from client 559c5a0f-07c0-47c0-a7fd-5550f3fb033e (at 10.51.9.42@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901288028c00, cur 1714144098 expire 1714143948 last 1714143871 [3880238.030347] LustreError: 36670:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069503531028 granted:559613176, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3880238.060886] LustreError: 36670:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 602 previous similar messages [3880838.042849] LustreError: 35183:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069503528780 granted:559613176, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3880838.073389] LustreError: 35183:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 684 previous similar messages [3881290.407041] Lustre: oak-MDT0000: haven't heard from client ef14f016-c87f-457b-a3fc-bad60acb9b24 (at 10.50.13.14@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9010e8539800, cur 1714145546 expire 1714145396 last 1714145319 [3881290.429008] Lustre: Skipped 7 previous similar messages [3881439.049671] LustreError: 35196:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107828187160 granted:310379184, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3881439.080220] LustreError: 35196:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 443 previous similar messages [3882039.629554] LustreError: 36655:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020c_UUID release: 18446621080987365160 granted:1028, total:134206344 qmt:oak-QMT0000 pool:dt-0x0 id:9826 enforced:1 hard:134217728 soft:0 granted:134206344 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:3510885 default:no [3882039.659191] LustreError: 36655:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 625 previous similar messages [3882625.347305] Lustre: oak-MDT0000: haven't heard from client a803f203-8808-43b5-b3fc-a51c7faadbd8 (at 10.51.9.7@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9031925dbc00, cur 1714146881 expire 1714146731 last 1714146654 [3882625.369094] Lustre: Skipped 1 previous similar message [3882640.836456] LustreError: 36685:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107913266740 granted:424654352, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3882640.866997] LustreError: 36685:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 446 previous similar messages [3883221.318455] Lustre: oak-MDT0003: haven't heard from client 5f997c60-29df-4256-82cd-d4bde73c938d (at 10.50.8.3@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9006ec37f400, cur 1714147477 expire 1714147327 last 1714147250 [3883221.340244] Lustre: Skipped 1 previous similar message [3883241.681287] LustreError: 36657:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069503445460 granted:559613176, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3883241.711831] LustreError: 36657:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 553 previous similar messages [3883842.353261] LustreError: 36661:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107935009012 granted:486160868, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3883842.383811] LustreError: 36661:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 604 previous similar messages [3884443.870901] LustreError: 35184:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107918684336 granted:483067692, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3884443.901442] LustreError: 35184:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 784 previous similar messages [3884653.258990] Lustre: oak-MDT0003: haven't heard from client f0e41ec1-bbe3-445e-8932-97079e29ea17 (at 10.51.18.5@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9002fce09400, cur 1714148909 expire 1714148759 last 1714148682 [3884653.280861] Lustre: Skipped 5 previous similar messages [3884661.252880] Lustre: oak-MDT0000: haven't heard from client f0e41ec1-bbe3-445e-8932-97079e29ea17 (at 10.51.18.5@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900fcba8a000, cur 1714148917 expire 1714148767 last 1714148690 [3885043.933420] LustreError: 35148:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107935006464 granted:486160868, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3885043.963963] LustreError: 35148:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 490 previous similar messages [3885176.228305] Lustre: oak-MDT0000: haven't heard from client 77b17f96-5d0d-40e4-b99c-02d103a1f96f (at 10.51.3.1@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9031b2ece800, cur 1714149432 expire 1714149282 last 1714149205 [3885644.899888] LustreError: 35180:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069503449280 granted:559613176, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3885644.930429] LustreError: 35180:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 617 previous similar messages [3885887.194135] Lustre: oak-MDT0003: haven't heard from client 5a688b3d-d88c-4eb9-9d25-08806075423b (at 10.51.12.9@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9015ee4e3000, cur 1714150143 expire 1714149993 last 1714149916 [3885887.216009] Lustre: Skipped 1 previous similar message [3885901.193556] Lustre: oak-MDT0000: haven't heard from client 5a688b3d-d88c-4eb9-9d25-08806075423b (at 10.51.12.9@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9008447ec400, cur 1714150157 expire 1714150007 last 1714149930 [3886247.295072] LustreError: 36661:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107934987452 granted:486160868, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3886247.326104] LustreError: 36661:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 703 previous similar messages [3886848.308546] LustreError: 36691:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107918679104 granted:483067692, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3886848.339084] LustreError: 36691:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 513 previous similar messages [3887448.920778] LustreError: 36681:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107913257592 granted:424654352, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3887448.951321] LustreError: 36681:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 697 previous similar messages [3887465.121047] Lustre: oak-MDT0000: haven't heard from client 49217cef-12ea-4cea-801c-5182234028dc (at 10.51.15.4@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9012416c1c00, cur 1714151721 expire 1714151571 last 1714151494 [3888053.907738] LustreError: 36669:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107828130404 granted:310379184, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3888053.938287] LustreError: 36669:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 594 previous similar messages [3888339.079804] Lustre: oak-MDT0000: haven't heard from client ab5ef027-cdc0-4fef-a101-21611e1a8c07 (at 10.50.13.14@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900f9a5e4800, cur 1714152595 expire 1714152445 last 1714152368 [3888339.101769] Lustre: Skipped 1 previous similar message [3888654.394167] LustreError: 35203:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020c_UUID release: 18446621080987365160 granted:1028, total:134206344 qmt:oak-QMT0000 pool:dt-0x0 id:9826 enforced:1 hard:134217728 soft:0 granted:134206344 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:3510885 default:no [3888654.423664] LustreError: 35203:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 526 previous similar messages [3889254.839821] LustreError: 36667:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107934903304 granted:486160868, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3889254.870531] LustreError: 36667:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 578 previous similar messages [3889858.063931] LustreError: 35147:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107828149476 granted:310379184, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3889858.094470] LustreError: 35147:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 564 previous similar messages [3890461.878117] LustreError: 35176:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107828124548 granted:310379184, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3890461.908666] LustreError: 35176:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 407 previous similar messages [3891065.387286] LustreError: 35198:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107828116224 granted:310379184, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3891065.417830] LustreError: 35198:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 535 previous similar messages [3891668.138660] LustreError: 36653:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107934976008 granted:486160868, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3891668.169205] LustreError: 36653:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 610 previous similar messages [3892268.449386] LustreError: 35221:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107913245944 granted:424654352, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3892268.479926] LustreError: 35221:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 674 previous similar messages [3892823.871823] Lustre: oak-MDT0000: haven't heard from client 9efc3830-8bd9-4e69-aa71-aa080040e20b (at 10.51.3.1@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9031925dac00, cur 1714157080 expire 1714156930 last 1714156853 [3892823.893614] Lustre: Skipped 1 previous similar message [3892869.533110] LustreError: 35197:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107828132364 granted:310379184, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3892869.563774] LustreError: 35197:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 629 previous similar messages [3893472.258458] LustreError: 36657:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020c_UUID release: 18446621080987365160 granted:1028, total:134206344 qmt:oak-QMT0000 pool:dt-0x0 id:9826 enforced:1 hard:134217728 soft:0 granted:134206344 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:3510885 default:no [3893472.287959] LustreError: 36657:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 640 previous similar messages [3893882.821467] Lustre: oak-MDT0000: haven't heard from client 5d8ea015-7fed-4e9f-b627-a0cf7770c335 (at 10.51.1.39@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903859105000, cur 1714158139 expire 1714157989 last 1714157912 [3893882.843472] Lustre: Skipped 1 previous similar message [3894073.469860] LustreError: 36664:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107814850800 granted:16384, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3894073.469861] LustreError: 35204:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107934964796 granted:486160868, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3894073.469864] LustreError: 35204:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 570 previous similar messages [3894675.315274] LustreError: 31509:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107934961972 granted:486160868, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3894675.345960] LustreError: 31509:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 425 previous similar messages [3895275.288144] LustreError: 31509:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107934897304 granted:486160868, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3895275.318688] LustreError: 31509:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 554 previous similar messages [3895700.734511] Lustre: oak-MDT0000: haven't heard from client 05db008e-7380-49cb-90e3-7ad8320a5a32 (at 10.51.6.58@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9007197a8c00, cur 1714159957 expire 1714159807 last 1714159730 [3895700.756384] Lustre: Skipped 3 previous similar messages [3895876.223728] LustreError: 36698:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107918605536 granted:483067692, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3895876.254263] LustreError: 36698:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 692 previous similar messages [3896479.154187] LustreError: 36659:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107934958084 granted:486160868, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3896479.184732] LustreError: 36659:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 689 previous similar messages [3896499.558284] LustreError: 44965:0:(mdt_open.c:1217:mdt_cross_open()) oak-MDT0003: [0x32c0020e56:0xd848:0x0] doesn't exist!: rc = -14 [3896879.679808] Lustre: oak-MDT0003: haven't heard from client c9b925aa-5891-436a-a310-71cc7b4092b4 (at 10.51.7.35@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90143ea25400, cur 1714161136 expire 1714160986 last 1714160909 [3896879.701690] Lustre: Skipped 1 previous similar message [3896882.678998] Lustre: oak-MDT0000: haven't heard from client 25c28af0-92a3-4ebf-9304-706235cc50a6 (at 10.51.7.26@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9017b7727400, cur 1714161139 expire 1714160989 last 1714160912 [3896882.701062] Lustre: Skipped 8 previous similar messages [3897079.183798] LustreError: 35165:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107918644232 granted:483067692, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3897079.214333] LustreError: 35165:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 424 previous similar messages [3897679.348237] LustreError: 36696:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107918612512 granted:483067692, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3897679.378777] LustreError: 36696:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 441 previous similar messages [3898281.796354] LustreError: 36655:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107934919572 granted:486160868, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3898281.827070] LustreError: 36655:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 480 previous similar messages [3898882.459406] LustreError: 35163:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107918617168 granted:483067692, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3898882.489943] LustreError: 35163:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 571 previous similar messages [3899483.501365] LustreError: 35168:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107918555700 granted:483067692, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3899483.531903] LustreError: 35168:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 851 previous similar messages [3900085.577442] LustreError: 35207:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069503434660 granted:559613176, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3900085.607985] LustreError: 35207:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 908 previous similar messages [3900685.782456] LustreError: 36654:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107828104136 granted:310379184, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3900685.812996] LustreError: 36654:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 985 previous similar messages [3901250.475585] Lustre: oak-MDT0000: haven't heard from client d7ed904c-d538-4ba0-8d41-3d4f4783921b (at 10.50.6.34@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9032b3c25400, cur 1714165507 expire 1714165357 last 1714165280 [3901250.497458] Lustre: Skipped 8 previous similar messages [3901288.852193] LustreError: 36659:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107828101848 granted:310379184, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3901288.882738] LustreError: 36659:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 672 previous similar messages [3901407.818743] ------------[ cut here ]------------ [3901407.823554] WARNING: CPU: 21 PID: 35146 at lib/list_debug.c:62 __list_del_entry+0x8e/0xe0 [3901407.831911] list_del corruption. next->prev should be ffff903864e680c8, but was ffff902375b46048 [3901407.840872] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) lov(OE) osc(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) mpt2sas mptctl mptbase bonding rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dm_service_time dm_multipath dm_mod dell_smbios dell_wmi_descriptor dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr cdc_ether ses usbnet enclosure mii sg i2c_piix4 wmi ipmi_si ipmi_devintf ipmi_msghandler tpm_crb [3901407.913514] acpi_power_meter ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) fb_sys_fops ttm mlxfw(OE) ahci mpt3sas(OE) bnxt_en tg3 libahci mlx_compat(OE) crct10dif_pclmul crct10dif_common raid_class drm crc32c_intel ptp libata scsi_transport_sas devlink drm_panel_orientation_quirks megaraid_sas pps_core [3901407.948078] CPU: 21 PID: 35146 Comm: mdt_rdpg01_005 Kdump: loaded Tainted: G OE ------------ 3.10.0-1160.108.1.el7_lustre.pl1.x86_64 #1 [3901407.961541] Hardware name: Dell Inc. PowerEdge R6525/0N7YGH, BIOS 2.14.1 12/17/2023 [3901407.969374] Call Trace: [3901407.972021] [] dump_stack+0x19/0x1f [3901407.977351] [] __warn+0xd8/0x100 [3901407.982530] [] warn_slowpath_fmt+0x5f/0x80 [3901407.988450] [] __list_del_entry+0x8e/0xe0 [3901407.994360] [] mdt_close_internal+0x12c/0x250 [mdt] [3901408.001077] [] mdt_close+0x291/0x900 [mdt] [3901408.007075] [] tgt_request_handle+0x93f/0x19d0 [ptlrpc] [3901408.014152] [] ? ptlrpc_nrs_req_get_nolock0+0xd5/0x170 [ptlrpc] [3901408.021900] [] ? ktime_get_real_seconds+0xe/0x20 [libcfs] [3901408.029146] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [3901408.036992] [] ? task_rq_unlock+0x20/0x20 [3901408.042831] [] ? __wake_up+0x13/0x20 [3901408.048252] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [3901408.054716] [] ? ptlrpc_wait_event+0x5d0/0x5d0 [ptlrpc] [3901408.061773] [] kthread+0xd1/0xe0 [3901408.066831] [] ? insert_kthread_work+0x40/0x40 [3901408.073106] [] ret_from_fork_nospec_begin+0x7/0x21 [3901408.079726] [] ? insert_kthread_work+0x40/0x40 [3901408.085998] ---[ end trace d06f3d22018e6a05 ]--- [3901408.094749] ------------[ cut here ]------------ [3901408.099691] WARNING: CPU: 53 PID: 35155 at lib/list_debug.c:59 __list_del_entry+0xb1/0xe0 [3901408.108040] list_del corruption. prev->next should be ffff902cef12e448, but was ffff903864e680c8 [3901408.116998] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) lov(OE) osc(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) mpt2sas mptctl mptbase bonding rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dm_service_time dm_multipath dm_mod dell_smbios dell_wmi_descriptor dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr cdc_ether ses usbnet enclosure mii sg i2c_piix4 wmi ipmi_si ipmi_devintf ipmi_msghandler tpm_crb [3901408.189401] acpi_power_meter ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) fb_sys_fops ttm mlxfw(OE) ahci mpt3sas(OE) bnxt_en tg3 libahci mlx_compat(OE) crct10dif_pclmul crct10dif_common raid_class drm crc32c_intel ptp libata scsi_transport_sas devlink drm_panel_orientation_quirks megaraid_sas pps_core [3901408.224096] CPU: 53 PID: 35155 Comm: mdt_rdpg01_013 Kdump: loaded Tainted: G W OE ------------ 3.10.0-1160.108.1.el7_lustre.pl1.x86_64 #1 [3901408.237558] Hardware name: Dell Inc. PowerEdge R6525/0N7YGH, BIOS 2.14.1 12/17/2023 [3901408.245385] Call Trace: [3901408.248016] [] dump_stack+0x19/0x1f [3901408.253335] [] __warn+0xd8/0x100 [3901408.258394] [] warn_slowpath_fmt+0x5f/0x80 [3901408.264317] [] __list_del_entry+0xb1/0xe0 [3901408.270165] [] mdt_close_internal+0x12c/0x250 [mdt] [3901408.276880] [] mdt_close+0x291/0x900 [mdt] [3901408.282834] [] tgt_request_handle+0x93f/0x19d0 [ptlrpc] [3901408.289908] [] ? ptlrpc_nrs_req_get_nolock0+0xd5/0x170 [ptlrpc] [3901408.297657] [] ? ktime_get_real_seconds+0xe/0x20 [libcfs] [3901408.304895] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [3901408.312724] [] ? task_rq_unlock+0x20/0x20 [3901408.318563] [] ? __wake_up+0x13/0x20 [3901408.323985] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [3901408.330446] [] ? ptlrpc_wait_event+0x5d0/0x5d0 [ptlrpc] [3901408.337496] [] kthread+0xd1/0xe0 [3901408.342553] [] ? insert_kthread_work+0x40/0x40 [3901408.348969] [] ret_from_fork_nospec_begin+0x7/0x21 [3901408.355589] [] ? insert_kthread_work+0x40/0x40 [3901408.361860] ---[ end trace d06f3d22018e6a06 ]--- [3901412.928297] ------------[ cut here ]------------ [3901412.933111] WARNING: CPU: 21 PID: 35146 at lib/list_debug.c:59 __list_del_entry+0xb1/0xe0 [3901412.941461] list_del corruption. prev->next should be ffff902375b46048, but was ffff902cef12e448 [3901412.950415] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) lov(OE) osc(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) mpt2sas mptctl mptbase bonding rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dm_service_time dm_multipath dm_mod dell_smbios dell_wmi_descriptor dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr cdc_ether ses usbnet enclosure mii sg i2c_piix4 wmi ipmi_si ipmi_devintf ipmi_msghandler tpm_crb [3901413.022942] acpi_power_meter ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) fb_sys_fops ttm mlxfw(OE) ahci mpt3sas(OE) bnxt_en tg3 libahci mlx_compat(OE) crct10dif_pclmul crct10dif_common raid_class drm crc32c_intel ptp libata scsi_transport_sas devlink drm_panel_orientation_quirks megaraid_sas pps_core [3901413.057482] CPU: 21 PID: 35146 Comm: mdt_rdpg01_005 Kdump: loaded Tainted: G W OE ------------ 3.10.0-1160.108.1.el7_lustre.pl1.x86_64 #1 [3901413.070941] Hardware name: Dell Inc. PowerEdge R6525/0N7YGH, BIOS 2.14.1 12/17/2023 [3901413.078769] Call Trace: [3901413.081408] [] dump_stack+0x19/0x1f [3901413.086867] [] __warn+0xd8/0x100 [3901413.091921] [] warn_slowpath_fmt+0x5f/0x80 [3901413.097842] [] __list_del_entry+0xb1/0xe0 [3901413.103688] [] mdt_close_internal+0x12c/0x250 [mdt] [3901413.110397] [] mdt_close+0x291/0x900 [mdt] [3901413.116346] [] tgt_request_handle+0x93f/0x19d0 [ptlrpc] [3901413.123408] [] ? ptlrpc_nrs_req_get_nolock0+0xd5/0x170 [ptlrpc] [3901413.131155] [] ? ktime_get_real_seconds+0xe/0x20 [libcfs] [3901413.138390] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [3901413.146349] [] ? task_rq_unlock+0x20/0x20 [3901413.152179] [] ? __wake_up+0x13/0x20 [3901413.157591] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [3901413.164046] [] ? ptlrpc_wait_event+0x5d0/0x5d0 [ptlrpc] [3901413.171091] [] kthread+0xd1/0xe0 [3901413.176142] [] ? insert_kthread_work+0x40/0x40 [3901413.182411] [] ret_from_fork_nospec_begin+0x7/0x21 [3901413.189027] [] ? insert_kthread_work+0x40/0x40 [3901413.195295] ---[ end trace d06f3d22018e6a07 ]--- [3901413.201842] ------------[ cut here ]------------ [3901413.206650] WARNING: CPU: 19 PID: 35183 at lib/list_debug.c:59 __list_del_entry+0xb1/0xe0 [3901413.214995] list_del corruption. prev->next should be ffff90327a2907c8, but was ffff902375b46048 [3901413.223956] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) lov(OE) osc(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) mpt2sas mptctl mptbase bonding rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dm_service_time dm_multipath dm_mod dell_smbios dell_wmi_descriptor dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr cdc_ether ses usbnet enclosure mii sg i2c_piix4 wmi ipmi_si ipmi_devintf ipmi_msghandler tpm_crb [3901413.296499] acpi_power_meter ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) fb_sys_fops ttm mlxfw(OE) ahci mpt3sas(OE) bnxt_en tg3 libahci mlx_compat(OE) crct10dif_pclmul crct10dif_common raid_class drm crc32c_intel ptp libata scsi_transport_sas devlink drm_panel_orientation_quirks megaraid_sas pps_core [3901413.331054] CPU: 19 PID: 35183 Comm: mdt_rdpg01_024 Kdump: loaded Tainted: G W OE ------------ 3.10.0-1160.108.1.el7_lustre.pl1.x86_64 #1 [3901413.344514] Hardware name: Dell Inc. PowerEdge R6525/0N7YGH, BIOS 2.14.1 12/17/2023 [3901413.352342] Call Trace: [3901413.354978] [] dump_stack+0x19/0x1f [3901413.360304] [] __warn+0xd8/0x100 [3901413.365361] [] warn_slowpath_fmt+0x5f/0x80 [3901413.371289] [] __list_del_entry+0xb1/0xe0 [3901413.377145] [] mdt_close_internal+0x12c/0x250 [mdt] [3901413.383859] [] mdt_close+0x291/0x900 [mdt] [3901413.389824] [] tgt_request_handle+0x93f/0x19d0 [ptlrpc] [3901413.396903] [] ? ptlrpc_nrs_req_get_nolock0+0xd5/0x170 [ptlrpc] [3901413.404659] [] ? ktime_get_real_seconds+0xe/0x20 [libcfs] [3901413.411900] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [3901413.419737] [] ? task_rq_unlock+0x20/0x20 [3901413.425573] [] ? __wake_up+0x13/0x20 [3901413.430991] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [3901413.437456] [] ? ptlrpc_wait_event+0x5d0/0x5d0 [ptlrpc] [3901413.444512] [] kthread+0xd1/0xe0 [3901413.449572] [] ? insert_kthread_work+0x40/0x40 [3901413.455849] [] ret_from_fork_nospec_begin+0x7/0x21 [3901413.462470] [] ? insert_kthread_work+0x40/0x40 [3901413.468741] ---[ end trace d06f3d22018e6a08 ]--- [3901413.793888] ------------[ cut here ]------------ [3901413.798684] WARNING: CPU: 53 PID: 35146 at lib/list_debug.c:62 __list_del_entry+0x8e/0xe0 [3901413.807037] list_del corruption. next->prev should be ffff9035044c01c8, but was ffff903864e680c8 [3901413.815996] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) lov(OE) osc(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) mpt2sas mptctl mptbase bonding rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dm_service_time dm_multipath dm_mod dell_smbios dell_wmi_descriptor dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr cdc_ether ses usbnet enclosure mii sg i2c_piix4 wmi ipmi_si ipmi_devintf ipmi_msghandler tpm_crb [3901413.888393] acpi_power_meter ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) fb_sys_fops ttm mlxfw(OE) ahci mpt3sas(OE) bnxt_en tg3 libahci mlx_compat(OE) crct10dif_pclmul crct10dif_common raid_class drm crc32c_intel ptp libata scsi_transport_sas devlink drm_panel_orientation_quirks megaraid_sas pps_core [3901413.922929] CPU: 53 PID: 35146 Comm: mdt_rdpg01_005 Kdump: loaded Tainted: G W OE ------------ 3.10.0-1160.108.1.el7_lustre.pl1.x86_64 #1 [3901413.936391] Hardware name: Dell Inc. PowerEdge R6525/0N7YGH, BIOS 2.14.1 12/17/2023 [3901413.944216] Call Trace: [3901413.946844] [] dump_stack+0x19/0x1f [3901413.952166] [] __warn+0xd8/0x100 [3901413.957225] [] warn_slowpath_fmt+0x5f/0x80 [3901413.963144] [] __list_del_entry+0x8e/0xe0 [3901413.968994] [] mdt_close_internal+0x12c/0x250 [mdt] [3901413.975707] [] mdt_close+0x291/0x900 [mdt] [3901413.981653] [] tgt_request_handle+0x93f/0x19d0 [ptlrpc] [3901413.988717] [] ? ptlrpc_nrs_req_get_nolock0+0xd5/0x170 [ptlrpc] [3901413.996633] [] ? ktime_get_real_seconds+0xe/0x20 [libcfs] [3901414.003873] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [3901414.011707] [] ? task_rq_unlock+0x20/0x20 [3901414.017546] [] ? __wake_up+0x13/0x20 [3901414.022972] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [3901414.029429] [] ? ptlrpc_wait_event+0x5d0/0x5d0 [ptlrpc] [3901414.036479] [] kthread+0xd1/0xe0 [3901414.041534] [] ? insert_kthread_work+0x40/0x40 [3901414.047809] [] ret_from_fork_nospec_begin+0x7/0x21 [3901414.054431] [] ? insert_kthread_work+0x40/0x40 [3901414.060701] ---[ end trace d06f3d22018e6a09 ]--- [3901414.107174] ------------[ cut here ]------------ [3901414.111989] WARNING: CPU: 49 PID: 36667 at lib/list_debug.c:62 __list_del_entry+0x8e/0xe0 [3901414.120345] list_del corruption. next->prev should be ffff902edefc9b48, but was ffff9035044c01c8 [3901414.129301] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) lov(OE) osc(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) mpt2sas mptctl mptbase bonding rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dm_service_time dm_multipath dm_mod dell_smbios dell_wmi_descriptor dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr cdc_ether ses usbnet enclosure mii sg i2c_piix4 wmi ipmi_si ipmi_devintf ipmi_msghandler tpm_crb [3901414.201886] acpi_power_meter ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) fb_sys_fops ttm mlxfw(OE) ahci mpt3sas(OE) bnxt_en tg3 libahci mlx_compat(OE) crct10dif_pclmul crct10dif_common raid_class drm crc32c_intel ptp libata scsi_transport_sas devlink drm_panel_orientation_quirks megaraid_sas pps_core [3901414.236613] CPU: 49 PID: 36667 Comm: mdt_rdpg01_054 Kdump: loaded Tainted: G W OE ------------ 3.10.0-1160.108.1.el7_lustre.pl1.x86_64 #1 [3901414.250074] Hardware name: Dell Inc. PowerEdge R6525/0N7YGH, BIOS 2.14.1 12/17/2023 [3901414.257903] Call Trace: [3901414.260546] [] dump_stack+0x19/0x1f [3901414.265872] [] __warn+0xd8/0x100 [3901414.270937] [] warn_slowpath_fmt+0x5f/0x80 [3901414.276861] [] __list_del_entry+0x8e/0xe0 [3901414.282714] [] mdt_close_internal+0x12c/0x250 [mdt] [3901414.289426] [] mdt_close+0x291/0x900 [mdt] [3901414.295394] [] tgt_request_handle+0x93f/0x19d0 [ptlrpc] [3901414.302467] [] ? ptlrpc_nrs_req_get_nolock0+0xd5/0x170 [ptlrpc] [3901414.310215] [] ? ktime_get_real_seconds+0xe/0x20 [libcfs] [3901414.317455] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [3901414.325287] [] ? task_rq_unlock+0x20/0x20 [3901414.331125] [] ? __wake_up+0x13/0x20 [3901414.336544] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [3901414.342999] [] ? ptlrpc_wait_event+0x5d0/0x5d0 [ptlrpc] [3901414.350055] [] kthread+0xd1/0xe0 [3901414.355114] [] ? insert_kthread_work+0x40/0x40 [3901414.361396] [] ret_from_fork_nospec_begin+0x7/0x21 [3901414.368015] [] ? insert_kthread_work+0x40/0x40 [3901414.374283] ---[ end trace d06f3d22018e6a0a ]--- [3901414.973532] ------------[ cut here ]------------ [3901414.978327] WARNING: CPU: 21 PID: 35144 at lib/list_debug.c:59 __list_del_entry+0xb1/0xe0 [3901414.986683] list_del corruption. prev->next should be ffff902ff3f8b2c8, but was ffff90327a2907c8 [3901414.995642] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) lov(OE) osc(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) mpt2sas mptctl mptbase bonding rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dm_service_time dm_multipath dm_mod dell_smbios dell_wmi_descriptor dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr cdc_ether ses usbnet enclosure mii sg i2c_piix4 wmi ipmi_si ipmi_devintf ipmi_msghandler tpm_crb [3901415.068035] acpi_power_meter ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) fb_sys_fops ttm mlxfw(OE) ahci mpt3sas(OE) bnxt_en tg3 libahci mlx_compat(OE) crct10dif_pclmul crct10dif_common raid_class drm crc32c_intel ptp libata scsi_transport_sas devlink drm_panel_orientation_quirks megaraid_sas pps_core [3901415.102597] CPU: 21 PID: 35144 Comm: mdt_rdpg01_003 Kdump: loaded Tainted: G W OE ------------ 3.10.0-1160.108.1.el7_lustre.pl1.x86_64 #1 [3901415.116062] Hardware name: Dell Inc. PowerEdge R6525/0N7YGH, BIOS 2.14.1 12/17/2023 [3901415.123889] Call Trace: [3901415.126532] [] dump_stack+0x19/0x1f [3901415.131993] [] __warn+0xd8/0x100 [3901415.137054] [] warn_slowpath_fmt+0x5f/0x80 [3901415.142979] [] __list_del_entry+0xb1/0xe0 [3901415.148838] [] mdt_close_internal+0x12c/0x250 [mdt] [3901415.155557] [] mdt_close+0x291/0x900 [mdt] [3901415.161508] [] tgt_request_handle+0x93f/0x19d0 [ptlrpc] [3901415.168585] [] ? ptlrpc_nrs_req_get_nolock0+0xd5/0x170 [ptlrpc] [3901415.176333] [] ? ktime_get_real_seconds+0xe/0x20 [libcfs] [3901415.183745] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [3901415.191579] [] ? task_rq_unlock+0x20/0x20 [3901415.197417] [] ? __wake_up+0x13/0x20 [3901415.202843] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [3901415.209311] [] ? ptlrpc_wait_event+0x5d0/0x5d0 [ptlrpc] [3901415.216360] [] kthread+0xd1/0xe0 [3901415.221426] [] ? insert_kthread_work+0x40/0x40 [3901415.227699] [] ret_from_fork_nospec_begin+0x7/0x21 [3901415.234318] [] ? insert_kthread_work+0x40/0x40 [3901415.240592] ---[ end trace d06f3d22018e6a0b ]--- [3901416.678224] ------------[ cut here ]------------ [3901416.683024] WARNING: CPU: 16 PID: 36667 at lib/list_debug.c:59 __list_del_entry+0xb1/0xe0 [3901416.691375] list_del corruption. prev->next should be ffff90376696a748, but was ffff902ff3f8b2c8 [3901416.700337] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) lov(OE) osc(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) mpt2sas mptctl mptbase bonding rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dm_service_time dm_multipath dm_mod dell_smbios dell_wmi_descriptor dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr cdc_ether ses usbnet enclosure mii sg i2c_piix4 wmi ipmi_si ipmi_devintf ipmi_msghandler tpm_crb [3901416.772758] acpi_power_meter ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) fb_sys_fops ttm mlxfw(OE) ahci mpt3sas(OE) bnxt_en tg3 libahci mlx_compat(OE) crct10dif_pclmul crct10dif_common raid_class drm crc32c_intel ptp libata scsi_transport_sas devlink drm_panel_orientation_quirks megaraid_sas pps_core [3901416.807304] CPU: 16 PID: 36667 Comm: mdt_rdpg01_054 Kdump: loaded Tainted: G W OE ------------ 3.10.0-1160.108.1.el7_lustre.pl1.x86_64 #1 [3901416.820764] Hardware name: Dell Inc. PowerEdge R6525/0N7YGH, BIOS 2.14.1 12/17/2023 [3901416.828592] Call Trace: [3901416.831232] [] dump_stack+0x19/0x1f [3901416.836548] [] __warn+0xd8/0x100 [3901416.841609] [] warn_slowpath_fmt+0x5f/0x80 [3901416.847536] [] __list_del_entry+0xb1/0xe0 [3901416.853377] [] mdt_close_internal+0x12c/0x250 [mdt] [3901416.860088] [] mdt_close+0x291/0x900 [mdt] [3901416.866031] [] tgt_request_handle+0x93f/0x19d0 [ptlrpc] [3901416.873100] [] ? ptlrpc_nrs_req_get_nolock0+0xd5/0x170 [ptlrpc] [3901416.880842] [] ? ktime_get_real_seconds+0xe/0x20 [libcfs] [3901416.888082] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [3901416.896055] [] ? task_rq_unlock+0x20/0x20 [3901416.902048] [] ? __wake_up+0x13/0x20 [3901416.907481] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [3901416.913955] [] ? ptlrpc_wait_event+0x5d0/0x5d0 [ptlrpc] [3901416.921011] [] kthread+0xd1/0xe0 [3901416.926070] [] ? insert_kthread_work+0x40/0x40 [3901416.932342] [] ret_from_fork_nospec_begin+0x7/0x21 [3901416.938959] [] ? insert_kthread_work+0x40/0x40 [3901416.945225] ---[ end trace d06f3d22018e6a0c ]--- [3901417.509049] ------------[ cut here ]------------ [3901417.513861] WARNING: CPU: 25 PID: 35196 at lib/list_debug.c:59 __list_del_entry+0xb1/0xe0 [3901417.522213] list_del corruption. prev->next should be ffff903f128ebdc8, but was ffff90376696a748 [3901417.531176] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) lov(OE) osc(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) mpt2sas mptctl mptbase bonding rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dm_service_time dm_multipath dm_mod dell_smbios dell_wmi_descriptor dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr cdc_ether ses usbnet enclosure mii sg i2c_piix4 wmi ipmi_si ipmi_devintf ipmi_msghandler tpm_crb [3901417.603749] acpi_power_meter ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) fb_sys_fops ttm mlxfw(OE) ahci mpt3sas(OE) bnxt_en tg3 libahci mlx_compat(OE) crct10dif_pclmul crct10dif_common raid_class drm crc32c_intel ptp libata scsi_transport_sas devlink drm_panel_orientation_quirks megaraid_sas pps_core [3901417.638306] CPU: 25 PID: 35196 Comm: mdt_rdpg01_027 Kdump: loaded Tainted: G W OE ------------ 3.10.0-1160.108.1.el7_lustre.pl1.x86_64 #1 [3901417.651764] Hardware name: Dell Inc. PowerEdge R6525/0N7YGH, BIOS 2.14.1 12/17/2023 [3901417.659591] Call Trace: [3901417.662236] [] dump_stack+0x19/0x1f [3901417.667563] [] __warn+0xd8/0x100 [3901417.672617] [] warn_slowpath_fmt+0x5f/0x80 [3901417.678544] [] __list_del_entry+0xb1/0xe0 [3901417.684401] [] mdt_close_internal+0x12c/0x250 [mdt] [3901417.691106] [] mdt_close+0x291/0x900 [mdt] [3901417.697067] [] tgt_request_handle+0x93f/0x19d0 [ptlrpc] [3901417.704136] [] ? ptlrpc_nrs_req_get_nolock0+0xd5/0x170 [ptlrpc] [3901417.711878] [] ? ktime_get_real_seconds+0xe/0x20 [libcfs] [3901417.719120] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [3901417.726951] [] ? task_rq_unlock+0x20/0x20 [3901417.732788] [] ? __wake_up+0x13/0x20 [3901417.738208] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [3901417.744671] [] ? ptlrpc_wait_event+0x5d0/0x5d0 [ptlrpc] [3901417.751728] [] kthread+0xd1/0xe0 [3901417.756808] [] ? insert_kthread_work+0x40/0x40 [3901417.763081] [] ret_from_fork_nospec_begin+0x7/0x21 [3901417.769701] [] ? insert_kthread_work+0x40/0x40 [3901417.775973] ---[ end trace d06f3d22018e6a0d ]--- [3901505.518397] LustreError: 44846:0:(mdt_open.c:1217:mdt_cross_open()) oak-MDT0003: [0x32c0020e56:0x11e0e:0x0] doesn't exist!: rc = -14 [3901505.530483] LustreError: 44846:0:(mdt_open.c:1217:mdt_cross_open()) Skipped 2 previous similar messages [3901538.482753] ------------[ cut here ]------------ [3901538.487564] WARNING: CPU: 20 PID: 35202 at lib/list_debug.c:59 __list_del_entry+0xb1/0xe0 [3901538.495920] list_del corruption. prev->next should be ffff902607036f48, but was ffff903f128ebdc0 [3901538.504877] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) lov(OE) osc(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) mpt2sas mptctl mptbase bonding rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dm_service_time dm_multipath dm_mod dell_smbios dell_wmi_descriptor dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr cdc_ether ses usbnet enclosure mii sg i2c_piix4 wmi ipmi_si ipmi_devintf ipmi_msghandler tpm_crb [3901538.577331] acpi_power_meter ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) fb_sys_fops ttm mlxfw(OE) ahci mpt3sas(OE) bnxt_en tg3 libahci mlx_compat(OE) crct10dif_pclmul crct10dif_common raid_class drm crc32c_intel ptp libata scsi_transport_sas devlink drm_panel_orientation_quirks megaraid_sas pps_core [3901538.611902] CPU: 20 PID: 35202 Comm: mdt_rdpg01_033 Kdump: loaded Tainted: G W OE ------------ 3.10.0-1160.108.1.el7_lustre.pl1.x86_64 #1 [3901538.625368] Hardware name: Dell Inc. PowerEdge R6525/0N7YGH, BIOS 2.14.1 12/17/2023 [3901538.633201] Call Trace: [3901538.635842] [] dump_stack+0x19/0x1f [3901538.641159] [] __warn+0xd8/0x100 [3901538.646218] [] warn_slowpath_fmt+0x5f/0x80 [3901538.652148] [] __list_del_entry+0xb1/0xe0 [3901538.658005] [] mdt_close_internal+0x12c/0x250 [mdt] [3901538.665158] [] mdt_close+0x291/0x900 [mdt] [3901538.671112] [] tgt_request_handle+0x93f/0x19d0 [ptlrpc] [3901538.678176] [] ? ptlrpc_nrs_req_get_nolock0+0xd5/0x170 [ptlrpc] [3901538.685922] [] ? ktime_get_real_seconds+0xe/0x20 [libcfs] [3901538.693159] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [3901538.700993] [] ? task_rq_unlock+0x20/0x20 [3901538.706829] [] ? __wake_up+0x13/0x20 [3901538.712240] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [3901538.718697] [] ? ptlrpc_wait_event+0x5d0/0x5d0 [ptlrpc] [3901538.725751] [] kthread+0xd1/0xe0 [3901538.730810] [] ? insert_kthread_work+0x40/0x40 [3901538.737080] [] ret_from_fork_nospec_begin+0x7/0x21 [3901538.743699] [] ? insert_kthread_work+0x40/0x40 [3901538.749971] ---[ end trace d06f3d22018e6a0e ]--- [3901538.763543] ------------[ cut here ]------------ [3901538.768515] WARNING: CPU: 62 PID: 35152 at lib/list_debug.c:62 __list_del_entry+0x8e/0xe0 [3901538.776975] list_del corruption. next->prev should be ffff903640715d48, but was ffff902edefc9b48 [3901538.785927] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) lov(OE) osc(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) mpt2sas mptctl mptbase bonding rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dm_service_time dm_multipath dm_mod dell_smbios dell_wmi_descriptor dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr cdc_ether ses usbnet enclosure mii sg i2c_piix4 wmi ipmi_si ipmi_devintf ipmi_msghandler tpm_crb [3901538.858357] acpi_power_meter ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) fb_sys_fops ttm mlxfw(OE) ahci mpt3sas(OE) bnxt_en tg3 libahci mlx_compat(OE) crct10dif_pclmul crct10dif_common raid_class drm crc32c_intel ptp libata scsi_transport_sas devlink drm_panel_orientation_quirks megaraid_sas pps_core [3901538.893952] CPU: 62 PID: 35152 Comm: mdt_rdpg01_010 Kdump: loaded Tainted: G W OE ------------ 3.10.0-1160.108.1.el7_lustre.pl1.x86_64 #1 [3901538.907415] Hardware name: Dell Inc. PowerEdge R6525/0N7YGH, BIOS 2.14.1 12/17/2023 [3901538.915517] Call Trace: [3901538.918151] [] dump_stack+0x19/0x1f [3901538.923475] [] __warn+0xd8/0x100 [3901538.928536] [] warn_slowpath_fmt+0x5f/0x80 [3901538.934463] [] __list_del_entry+0x8e/0xe0 [3901538.940339] [] mdt_close_internal+0x12c/0x250 [mdt] [3901538.947050] [] mdt_close+0x291/0x900 [mdt] [3901538.953015] [] tgt_request_handle+0x93f/0x19d0 [ptlrpc] [3901538.960079] [] ? ptlrpc_nrs_req_get_nolock0+0xd5/0x170 [ptlrpc] [3901538.967823] [] ? ktime_get_real_seconds+0xe/0x20 [libcfs] [3901538.975059] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [3901538.982892] [] ? task_rq_unlock+0x20/0x20 [3901538.988729] [] ? __wake_up+0x13/0x20 [3901538.994144] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [3901539.000600] [] ? ptlrpc_wait_event+0x5d0/0x5d0 [ptlrpc] [3901539.007653] [] kthread+0xd1/0xe0 [3901539.012712] [] ? insert_kthread_work+0x40/0x40 [3901539.019102] [] ret_from_fork_nospec_begin+0x7/0x21 [3901539.026051] [] ? insert_kthread_work+0x40/0x40 [3901539.032324] ---[ end trace d06f3d22018e6a0f ]--- [3901539.037585] ------------[ cut here ]------------ [3901539.042408] WARNING: CPU: 25 PID: 36672 at lib/list_debug.c:62 __list_del_entry+0x8e/0xe0 [3901539.050761] list_del corruption. next->prev should be ffff903ca48351c8, but was ffff903640715d48 [3901539.059721] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) lov(OE) osc(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) mpt2sas mptctl mptbase bonding rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dm_service_time dm_multipath dm_mod dell_smbios dell_wmi_descriptor dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr cdc_ether ses usbnet enclosure mii sg i2c_piix4 wmi ipmi_si ipmi_devintf ipmi_msghandler tpm_crb [3901539.132285] acpi_power_meter ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) fb_sys_fops ttm mlxfw(OE) ahci mpt3sas(OE) bnxt_en tg3 libahci mlx_compat(OE) crct10dif_pclmul crct10dif_common raid_class drm crc32c_intel ptp libata scsi_transport_sas devlink drm_panel_orientation_quirks megaraid_sas pps_core [3901539.166824] CPU: 25 PID: 36672 Comm: mdt_rdpg01_059 Kdump: loaded Tainted: G W OE ------------ 3.10.0-1160.108.1.el7_lustre.pl1.x86_64 #1 [3901539.180286] Hardware name: Dell Inc. PowerEdge R6525/0N7YGH, BIOS 2.14.1 12/17/2023 [3901539.188227] Call Trace: [3901539.190869] [] dump_stack+0x19/0x1f [3901539.196187] [] __warn+0xd8/0x100 [3901539.201243] [] warn_slowpath_fmt+0x5f/0x80 [3901539.207172] [] __list_del_entry+0x8e/0xe0 [3901539.213039] [] mdt_close_internal+0x12c/0x250 [mdt] [3901539.219753] [] mdt_close+0x291/0x900 [mdt] [3901539.225729] [] tgt_request_handle+0x93f/0x19d0 [ptlrpc] [3901539.232803] [] ? ptlrpc_nrs_req_get_nolock0+0xd5/0x170 [ptlrpc] [3901539.240549] [] ? ktime_get_real_seconds+0xe/0x20 [libcfs] [3901539.247794] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [3901539.255629] [] ? task_rq_unlock+0x20/0x20 [3901539.261470] [] ? queued_spin_lock_slowpath+0xb/0x13 [3901539.268178] [] ? _raw_spin_lock+0x24/0x30 [3901539.274036] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [3901539.280499] [] ? ptlrpc_wait_event+0x5d0/0x5d0 [ptlrpc] [3901539.287723] [] kthread+0xd1/0xe0 [3901539.292780] [] ? insert_kthread_work+0x40/0x40 [3901539.299055] [] ret_from_fork_nospec_begin+0x7/0x21 [3901539.305674] [] ? insert_kthread_work+0x40/0x40 [3901539.311949] ---[ end trace d06f3d22018e6a10 ]--- [3901539.321658] ------------[ cut here ]------------ [3901539.326622] WARNING: CPU: 29 PID: 35204 at lib/list_debug.c:59 __list_del_entry+0xb1/0xe0 [3901539.334979] list_del corruption. prev->next should be ffff903d2fc22a48, but was ffff902607036f48 [3901539.343940] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) lov(OE) osc(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) mpt2sas mptctl mptbase bonding rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dm_service_time dm_multipath dm_mod dell_smbios dell_wmi_descriptor dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr cdc_ether ses usbnet enclosure mii sg i2c_piix4 wmi ipmi_si ipmi_devintf ipmi_msghandler tpm_crb [3901539.416380] acpi_power_meter ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) fb_sys_fops ttm mlxfw(OE) ahci mpt3sas(OE) bnxt_en tg3 libahci mlx_compat(OE) crct10dif_pclmul crct10dif_common raid_class drm crc32c_intel ptp libata scsi_transport_sas devlink drm_panel_orientation_quirks megaraid_sas pps_core [3901539.451034] CPU: 29 PID: 35204 Comm: mdt_rdpg01_035 Kdump: loaded Tainted: G W OE ------------ 3.10.0-1160.108.1.el7_lustre.pl1.x86_64 #1 [3901539.464499] Hardware name: Dell Inc. PowerEdge R6525/0N7YGH, BIOS 2.14.1 12/17/2023 [3901539.472333] Call Trace: [3901539.474975] [] dump_stack+0x19/0x1f [3901539.480312] [] __warn+0xd8/0x100 [3901539.485374] [] warn_slowpath_fmt+0x5f/0x80 [3901539.491303] [] __list_del_entry+0xb1/0xe0 [3901539.497204] [] mdt_close_internal+0x12c/0x250 [mdt] [3901539.503918] [] mdt_close+0x291/0x900 [mdt] [3901539.509887] [] tgt_request_handle+0x93f/0x19d0 [ptlrpc] [3901539.516961] [] ? ptlrpc_nrs_req_get_nolock0+0xd5/0x170 [ptlrpc] [3901539.524707] [] ? ktime_get_real_seconds+0xe/0x20 [libcfs] [3901539.531951] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [3901539.539887] [] ? task_rq_unlock+0x20/0x20 [3901539.545727] [] ? __wake_up+0x13/0x20 [3901539.551146] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [3901539.557613] [] ? ptlrpc_wait_event+0x5d0/0x5d0 [ptlrpc] [3901539.564666] [] kthread+0xd1/0xe0 [3901539.569724] [] ? insert_kthread_work+0x40/0x40 [3901539.576001] [] ret_from_fork_nospec_begin+0x7/0x21 [3901539.582618] [] ? insert_kthread_work+0x40/0x40 [3901539.588883] ---[ end trace d06f3d22018e6a11 ]--- [3901539.596754] ------------[ cut here ]------------ [3901539.601832] WARNING: CPU: 20 PID: 35192 at lib/list_debug.c:62 __list_del_entry+0x8e/0xe0 [3901539.610178] list_del corruption. next->prev should be ffff904027395f48, but was ffff903ca48351c8 [3901539.619131] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) lov(OE) osc(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) mpt2sas mptctl mptbase bonding rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dm_service_time dm_multipath dm_mod dell_smbios dell_wmi_descriptor dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr cdc_ether ses usbnet enclosure mii sg i2c_piix4 wmi ipmi_si ipmi_devintf ipmi_msghandler tpm_crb [3901539.691509] acpi_power_meter ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) fb_sys_fops ttm mlxfw(OE) ahci mpt3sas(OE) bnxt_en tg3 libahci mlx_compat(OE) crct10dif_pclmul crct10dif_common raid_class drm crc32c_intel ptp libata scsi_transport_sas devlink drm_panel_orientation_quirks megaraid_sas pps_core [3901539.726182] CPU: 20 PID: 35192 Comm: mdt_rdpg01_025 Kdump: loaded Tainted: G W OE ------------ 3.10.0-1160.108.1.el7_lustre.pl1.x86_64 #1 [3901539.739645] Hardware name: Dell Inc. PowerEdge R6525/0N7YGH, BIOS 2.14.1 12/17/2023 [3901539.747471] Call Trace: [3901539.750099] [] dump_stack+0x19/0x1f [3901539.755420] [] __warn+0xd8/0x100 [3901539.760483] [] warn_slowpath_fmt+0x5f/0x80 [3901539.766408] [] __list_del_entry+0x8e/0xe0 [3901539.772249] [] mdt_close_internal+0x12c/0x250 [mdt] [3901539.778955] [] mdt_close+0x291/0x900 [mdt] [3901539.784899] [] tgt_request_handle+0x93f/0x19d0 [ptlrpc] [3901539.791962] [] ? ptlrpc_nrs_req_get_nolock0+0xd5/0x170 [ptlrpc] [3901539.799707] [] ? ktime_get_real_seconds+0xe/0x20 [libcfs] [3901539.806946] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [3901539.814778] [] ? task_rq_unlock+0x20/0x20 [3901539.820615] [] ? __wake_up+0x13/0x20 [3901539.826176] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [3901539.832633] [] ? ptlrpc_wait_event+0x5d0/0x5d0 [ptlrpc] [3901539.839684] [] kthread+0xd1/0xe0 [3901539.844734] [] ? insert_kthread_work+0x40/0x40 [3901539.851003] [] ret_from_fork_nospec_begin+0x7/0x21 [3901539.857624] [] ? insert_kthread_work+0x40/0x40 [3901539.863887] ---[ end trace d06f3d22018e6a12 ]--- [3901889.081832] LustreError: 36699:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107918600132 granted:483067692, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3901889.112482] LustreError: 36699:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 456 previous similar messages [3902081.435940] Lustre: oak-MDT0000: haven't heard from client de9a6cb4-43b4-46db-8c0a-39d23959eab3 (at 10.51.3.1@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903360af1c00, cur 1714166338 expire 1714166188 last 1714166111 [3902081.457732] Lustre: Skipped 1 previous similar message [3902491.952426] LustreError: 35236:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107918605712 granted:483067692, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3902491.982966] LustreError: 35236:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 690 previous similar messages [3903093.831868] LustreError: 36686:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107913211572 granted:424654352, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3903093.862414] LustreError: 36686:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 788 previous similar messages [3903696.029464] LustreError: 35219:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107913198204 granted:424654352, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3903696.059996] LustreError: 35219:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 484 previous similar messages [3904299.431987] LustreError: 36653:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107934922720 granted:486160868, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3904299.462530] LustreError: 36653:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 476 previous similar messages [3904901.061861] LustreError: 35167:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107913198592 granted:424654352, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3904901.092410] LustreError: 35167:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 578 previous similar messages [3905290.833884] ------------[ cut here ]------------ [3905290.838702] WARNING: CPU: 50 PID: 36656 at lib/list_debug.c:59 __list_del_entry+0xb1/0xe0 [3905290.847052] list_del corruption. prev->next should be ffff903229f1be48, but was ffff903d2fc22a40 [3905290.856015] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) lov(OE) osc(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) mpt2sas mptctl mptbase bonding rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dm_service_time dm_multipath dm_mod dell_smbios dell_wmi_descriptor dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr cdc_ether ses usbnet enclosure mii sg i2c_piix4 wmi ipmi_si ipmi_devintf ipmi_msghandler tpm_crb [3905290.928475] acpi_power_meter ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) fb_sys_fops ttm mlxfw(OE) ahci mpt3sas(OE) bnxt_en tg3 libahci mlx_compat(OE) crct10dif_pclmul crct10dif_common raid_class drm crc32c_intel ptp libata scsi_transport_sas devlink drm_panel_orientation_quirks megaraid_sas pps_core [3905290.963030] CPU: 50 PID: 36656 Comm: mdt_rdpg01_044 Kdump: loaded Tainted: G W OE ------------ 3.10.0-1160.108.1.el7_lustre.pl1.x86_64 #1 [3905290.976492] Hardware name: Dell Inc. PowerEdge R6525/0N7YGH, BIOS 2.14.1 12/17/2023 [3905290.984318] Call Trace: [3905290.986954] [] dump_stack+0x19/0x1f [3905290.992278] [] __warn+0xd8/0x100 [3905290.997335] [] warn_slowpath_fmt+0x5f/0x80 [3905291.003256] [] __list_del_entry+0xb1/0xe0 [3905291.009128] [] mdt_close_internal+0x12c/0x250 [mdt] [3905291.015831] [] mdt_close+0x291/0x900 [mdt] [3905291.021787] [] tgt_request_handle+0x93f/0x19d0 [ptlrpc] [3905291.028853] [] ? ptlrpc_nrs_req_get_nolock0+0xd5/0x170 [ptlrpc] [3905291.036597] [] ? ktime_get_real_seconds+0xe/0x20 [libcfs] [3905291.043835] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [3905291.051668] [] ? task_rq_unlock+0x20/0x20 [3905291.057506] [] ? __wake_up+0x13/0x20 [3905291.062918] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [3905291.069374] [] ? ptlrpc_wait_event+0x5d0/0x5d0 [ptlrpc] [3905291.076431] [] kthread+0xd1/0xe0 [3905291.081487] [] ? insert_kthread_work+0x40/0x40 [3905291.087757] [] ret_from_fork_nospec_begin+0x7/0x21 [3905291.094375] [] ? insert_kthread_work+0x40/0x40 [3905291.100649] ---[ end trace d06f3d22018e6a13 ]--- [3905310.872973] ------------[ cut here ]------------ [3905310.877777] WARNING: CPU: 26 PID: 36655 at lib/list_debug.c:62 __list_del_entry+0x8e/0xe0 [3905310.886126] list_del corruption. next->prev should be ffff903ed6eb0ac8, but was ffff904027395f50 [3905310.895086] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) lov(OE) osc(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) mpt2sas mptctl mptbase bonding rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dm_service_time dm_multipath dm_mod dell_smbios dell_wmi_descriptor dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr cdc_ether ses usbnet enclosure mii sg i2c_piix4 wmi ipmi_si ipmi_devintf ipmi_msghandler tpm_crb [3905310.967526] acpi_power_meter ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) fb_sys_fops ttm mlxfw(OE) ahci mpt3sas(OE) bnxt_en tg3 libahci mlx_compat(OE) crct10dif_pclmul crct10dif_common raid_class drm crc32c_intel ptp libata scsi_transport_sas devlink drm_panel_orientation_quirks megaraid_sas pps_core [3905311.002070] CPU: 26 PID: 36655 Comm: mdt_rdpg01_043 Kdump: loaded Tainted: G W OE ------------ 3.10.0-1160.108.1.el7_lustre.pl1.x86_64 #1 [3905311.015533] Hardware name: Dell Inc. PowerEdge R6525/0N7YGH, BIOS 2.14.1 12/17/2023 [3905311.023359] Call Trace: [3905311.025996] [] dump_stack+0x19/0x1f [3905311.031320] [] __warn+0xd8/0x100 [3905311.036374] [] warn_slowpath_fmt+0x5f/0x80 [3905311.042297] [] __list_del_entry+0x8e/0xe0 [3905311.048152] [] mdt_close_internal+0x12c/0x250 [mdt] [3905311.054858] [] mdt_close+0x291/0x900 [mdt] [3905311.060814] [] tgt_request_handle+0x93f/0x19d0 [ptlrpc] [3905311.067876] [] ? ptlrpc_nrs_req_get_nolock0+0xd5/0x170 [ptlrpc] [3905311.075619] [] ? ktime_get_real_seconds+0xe/0x20 [libcfs] [3905311.082859] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [3905311.090692] [] ? task_rq_unlock+0x20/0x20 [3905311.096531] [] ? __wake_up+0x13/0x20 [3905311.101940] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [3905311.108397] [] ? ptlrpc_wait_event+0x5d0/0x5d0 [ptlrpc] [3905311.115452] [] kthread+0xd1/0xe0 [3905311.120510] [] ? insert_kthread_work+0x40/0x40 [3905311.126788] [] ret_from_fork_nospec_begin+0x7/0x21 [3905311.133404] [] ? insert_kthread_work+0x40/0x40 [3905311.139671] ---[ end trace d06f3d22018e6a14 ]--- [3905311.149234] ------------[ cut here ]------------ [3905311.154049] WARNING: CPU: 23 PID: 35158 at lib/list_debug.c:62 __list_del_entry+0x8e/0xe0 [3905311.162402] list_del corruption. next->prev should be ffff902cd7b061c8, but was ffff903ed6eb0ac8 [3905311.171362] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) lov(OE) osc(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) mpt2sas mptctl mptbase bonding rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dm_service_time dm_multipath dm_mod dell_smbios dell_wmi_descriptor dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr cdc_ether ses usbnet enclosure mii sg i2c_piix4 wmi ipmi_si ipmi_devintf ipmi_msghandler tpm_crb [3905311.243803] acpi_power_meter ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) fb_sys_fops ttm mlxfw(OE) ahci mpt3sas(OE) bnxt_en tg3 libahci mlx_compat(OE) crct10dif_pclmul crct10dif_common raid_class drm crc32c_intel ptp libata scsi_transport_sas devlink drm_panel_orientation_quirks megaraid_sas pps_core [3905311.278529] CPU: 23 PID: 35158 Comm: mdt_rdpg01_016 Kdump: loaded Tainted: G W OE ------------ 3.10.0-1160.108.1.el7_lustre.pl1.x86_64 #1 [3905311.291991] Hardware name: Dell Inc. PowerEdge R6525/0N7YGH, BIOS 2.14.1 12/17/2023 [3905311.299824] Call Trace: [3905311.302463] [] dump_stack+0x19/0x1f [3905311.307789] [] __warn+0xd8/0x100 [3905311.312845] [] warn_slowpath_fmt+0x5f/0x80 [3905311.318775] [] __list_del_entry+0x8e/0xe0 [3905311.324657] [] mdt_close_internal+0x12c/0x250 [mdt] [3905311.331375] [] mdt_close+0x291/0x900 [mdt] [3905311.337338] [] tgt_request_handle+0x93f/0x19d0 [ptlrpc] [3905311.344410] [] ? ptlrpc_nrs_req_get_nolock0+0xd5/0x170 [ptlrpc] [3905311.352157] [] ? ktime_get_real_seconds+0xe/0x20 [libcfs] [3905311.359400] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [3905311.367379] [] ? task_rq_unlock+0x20/0x20 [3905311.373219] [] ? __wake_up+0x13/0x20 [3905311.378635] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [3905311.385095] [] ? ptlrpc_wait_event+0x5d0/0x5d0 [ptlrpc] [3905311.392150] [] kthread+0xd1/0xe0 [3905311.397209] [] ? insert_kthread_work+0x40/0x40 [3905311.403482] [] ret_from_fork_nospec_begin+0x7/0x21 [3905311.410408] [] ? insert_kthread_work+0x40/0x40 [3905311.416682] ---[ end trace d06f3d22018e6a15 ]--- [3905441.131323] ------------[ cut here ]------------ [3905441.136133] WARNING: CPU: 29 PID: 36662 at lib/list_debug.c:62 __list_del_entry+0x8e/0xe0 [3905441.144488] list_del corruption. next->prev should be ffff9032e7e296c8, but was 5a5a5a5a5a5a5a5a [3905441.153446] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) lov(OE) osc(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) mpt2sas mptctl mptbase bonding rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dm_service_time dm_multipath dm_mod dell_smbios dell_wmi_descriptor dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr cdc_ether ses usbnet enclosure mii sg i2c_piix4 wmi ipmi_si ipmi_devintf ipmi_msghandler tpm_crb [3905441.225876] acpi_power_meter ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) fb_sys_fops ttm mlxfw(OE) ahci mpt3sas(OE) bnxt_en tg3 libahci mlx_compat(OE) crct10dif_pclmul crct10dif_common raid_class drm crc32c_intel ptp libata scsi_transport_sas devlink drm_panel_orientation_quirks megaraid_sas pps_core [3905441.260431] CPU: 29 PID: 36662 Comm: mdt_rdpg01_050 Kdump: loaded Tainted: G W OE ------------ 3.10.0-1160.108.1.el7_lustre.pl1.x86_64 #1 [3905441.273894] Hardware name: Dell Inc. PowerEdge R6525/0N7YGH, BIOS 2.14.1 12/17/2023 [3905441.281729] Call Trace: [3905441.284371] [] dump_stack+0x19/0x1f [3905441.289696] [] __warn+0xd8/0x100 [3905441.294753] [] warn_slowpath_fmt+0x5f/0x80 [3905441.300681] [] __list_del_entry+0x8e/0xe0 [3905441.306558] [] mdt_close_internal+0x12c/0x250 [mdt] [3905441.313270] [] mdt_close+0x291/0x900 [mdt] [3905441.319224] [] tgt_request_handle+0x93f/0x19d0 [ptlrpc] [3905441.326289] [] ? ptlrpc_nrs_req_get_nolock0+0xd5/0x170 [ptlrpc] [3905441.334039] [] ? ktime_get_real_seconds+0xe/0x20 [libcfs] [3905441.341281] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [3905441.349112] [] ? task_rq_unlock+0x20/0x20 [3905441.354949] [] ? __wake_up+0x13/0x20 [3905441.360360] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [3905441.366818] [] ? ptlrpc_wait_event+0x5d0/0x5d0 [ptlrpc] [3905441.373872] [] kthread+0xd1/0xe0 [3905441.378931] [] ? insert_kthread_work+0x40/0x40 [3905441.385200] [] ret_from_fork_nospec_begin+0x7/0x21 [3905441.391818] [] ? insert_kthread_work+0x40/0x40 [3905441.398090] ---[ end trace d06f3d22018e6a16 ]--- [3905441.406815] ------------[ cut here ]------------ [3905441.411612] WARNING: CPU: 29 PID: 36662 at lib/list_debug.c:59 __list_del_entry+0xb1/0xe0 [3905441.419957] list_del corruption. prev->next should be ffff9028bf10b9c8, but was 5a5a5a5a5a5a5a5a [3905441.428908] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) lov(OE) osc(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) mpt2sas mptctl mptbase bonding rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dm_service_time dm_multipath dm_mod dell_smbios dell_wmi_descriptor dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr cdc_ether ses usbnet enclosure mii sg i2c_piix4 wmi ipmi_si ipmi_devintf ipmi_msghandler tpm_crb [3905441.501505] acpi_power_meter ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) fb_sys_fops ttm mlxfw(OE) ahci mpt3sas(OE) bnxt_en tg3 libahci mlx_compat(OE) crct10dif_pclmul crct10dif_common raid_class drm crc32c_intel ptp libata scsi_transport_sas devlink drm_panel_orientation_quirks megaraid_sas pps_core [3905441.536221] CPU: 29 PID: 36662 Comm: mdt_rdpg01_050 Kdump: loaded Tainted: G W OE ------------ 3.10.0-1160.108.1.el7_lustre.pl1.x86_64 #1 [3905441.549686] Hardware name: Dell Inc. PowerEdge R6525/0N7YGH, BIOS 2.14.1 12/17/2023 [3905441.557509] Call Trace: [3905441.560137] [] dump_stack+0x19/0x1f [3905441.565450] [] __warn+0xd8/0x100 [3905441.570502] [] warn_slowpath_fmt+0x5f/0x80 [3905441.576422] [] __list_del_entry+0xb1/0xe0 [3905441.582269] [] mdt_close_internal+0x12c/0x250 [mdt] [3905441.588973] [] mdt_close+0x291/0x900 [mdt] [3905441.594918] [] tgt_request_handle+0x93f/0x19d0 [ptlrpc] [3905441.601986] [] ? ptlrpc_nrs_req_get_nolock0+0xd5/0x170 [ptlrpc] [3905441.609730] [] ? ktime_get_real_seconds+0xe/0x20 [libcfs] [3905441.616968] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [3905441.624796] [] ? task_rq_unlock+0x20/0x20 [3905441.630628] [] ? __wake_up+0x13/0x20 [3905441.636179] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [3905441.642636] [] ? ptlrpc_wait_event+0x5d0/0x5d0 [ptlrpc] [3905441.649686] [] kthread+0xd1/0xe0 [3905441.654739] [] ? insert_kthread_work+0x40/0x40 [3905441.661008] [] ret_from_fork_nospec_begin+0x7/0x21 [3905441.667626] [] ? insert_kthread_work+0x40/0x40 [3905441.674004] ---[ end trace d06f3d22018e6a17 ]--- [3905441.679372] ------------[ cut here ]------------ [3905441.684165] WARNING: CPU: 29 PID: 36662 at lib/list_debug.c:59 __list_del_entry+0xb1/0xe0 [3905441.692516] list_del corruption. prev->next should be ffff9029b04dc248, but was ffff9028bf10b9c8 [3905441.701466] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) lov(OE) osc(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) mpt2sas mptctl mptbase bonding rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dm_service_time dm_multipath dm_mod dell_smbios dell_wmi_descriptor dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr cdc_ether ses usbnet enclosure mii sg i2c_piix4 wmi ipmi_si ipmi_devintf ipmi_msghandler tpm_crb [3905441.773994] acpi_power_meter ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) fb_sys_fops ttm mlxfw(OE) ahci mpt3sas(OE) bnxt_en tg3 libahci mlx_compat(OE) crct10dif_pclmul crct10dif_common raid_class drm crc32c_intel ptp libata scsi_transport_sas devlink drm_panel_orientation_quirks megaraid_sas pps_core [3905441.808556] CPU: 29 PID: 36662 Comm: mdt_rdpg01_050 Kdump: loaded Tainted: G W OE ------------ 3.10.0-1160.108.1.el7_lustre.pl1.x86_64 #1 [3905441.822186] Hardware name: Dell Inc. PowerEdge R6525/0N7YGH, BIOS 2.14.1 12/17/2023 [3905441.830011] Call Trace: [3905441.832635] [] dump_stack+0x19/0x1f [3905441.837948] [] __warn+0xd8/0x100 [3905441.843000] [] warn_slowpath_fmt+0x5f/0x80 [3905441.848920] [] __list_del_entry+0xb1/0xe0 [3905441.854759] [] mdt_close_internal+0x12c/0x250 [mdt] [3905441.861465] [] mdt_close+0x291/0x900 [mdt] [3905441.867404] [] tgt_request_handle+0x93f/0x19d0 [ptlrpc] [3905441.874466] [] ? ptlrpc_nrs_req_get_nolock0+0xd5/0x170 [ptlrpc] [3905441.882210] [] ? ktime_get_real_seconds+0xe/0x20 [libcfs] [3905441.889710] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [3905441.897538] [] ? task_rq_unlock+0x20/0x20 [3905441.903371] [] ? __wake_up+0x13/0x20 [3905441.908783] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [3905441.915237] [] ? ptlrpc_wait_event+0x5d0/0x5d0 [ptlrpc] [3905441.922280] [] kthread+0xd1/0xe0 [3905441.927336] [] ? insert_kthread_work+0x40/0x40 [3905441.933602] [] ret_from_fork_nospec_begin+0x7/0x21 [3905441.940212] [] ? insert_kthread_work+0x40/0x40 [3905441.946480] ---[ end trace d06f3d22018e6a18 ]--- [3905441.952279] ------------[ cut here ]------------ [3905441.957069] WARNING: CPU: 29 PID: 36662 at lib/list_debug.c:59 __list_del_entry+0xb1/0xe0 [3905441.965414] list_del corruption. prev->next should be ffff902c36bb6348, but was ffff9029b04dc248 [3905441.974366] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) lov(OE) osc(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) mpt2sas mptctl mptbase bonding rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dm_service_time dm_multipath dm_mod dell_smbios dell_wmi_descriptor dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr cdc_ether ses usbnet enclosure mii sg i2c_piix4 wmi ipmi_si ipmi_devintf ipmi_msghandler tpm_crb [3905442.046876] acpi_power_meter ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) fb_sys_fops ttm mlxfw(OE) ahci mpt3sas(OE) bnxt_en tg3 libahci mlx_compat(OE) crct10dif_pclmul crct10dif_common raid_class drm crc32c_intel ptp libata scsi_transport_sas devlink drm_panel_orientation_quirks megaraid_sas pps_core [3905442.081940] CPU: 29 PID: 36662 Comm: mdt_rdpg01_050 Kdump: loaded Tainted: G W OE ------------ 3.10.0-1160.108.1.el7_lustre.pl1.x86_64 #1 [3905442.095559] Hardware name: Dell Inc. PowerEdge R6525/0N7YGH, BIOS 2.14.1 12/17/2023 [3905442.103385] Call Trace: [3905442.106010] [] dump_stack+0x19/0x1f [3905442.111324] [] __warn+0xd8/0x100 [3905442.116375] [] warn_slowpath_fmt+0x5f/0x80 [3905442.122296] [] __list_del_entry+0xb1/0xe0 [3905442.128132] [] mdt_close_internal+0x12c/0x250 [mdt] [3905442.134839] [] mdt_close+0x291/0x900 [mdt] [3905442.140779] [] tgt_request_handle+0x93f/0x19d0 [ptlrpc] [3905442.147839] [] ? ptlrpc_nrs_req_get_nolock0+0xd5/0x170 [ptlrpc] [3905442.155582] [] ? ktime_get_real_seconds+0xe/0x20 [libcfs] [3905442.162813] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [3905442.170639] [] ? task_rq_unlock+0x20/0x20 [3905442.176467] [] ? __wake_up+0x13/0x20 [3905442.181880] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [3905442.188337] [] ? ptlrpc_wait_event+0x5d0/0x5d0 [ptlrpc] [3905442.195544] [] kthread+0xd1/0xe0 [3905442.200595] [] ? insert_kthread_work+0x40/0x40 [3905442.206862] [] ret_from_fork_nospec_begin+0x7/0x21 [3905442.213474] [] ? insert_kthread_work+0x40/0x40 [3905442.219740] ---[ end trace d06f3d22018e6a19 ]--- [3905442.224880] ------------[ cut here ]------------ [3905442.229675] WARNING: CPU: 29 PID: 36662 at lib/list_debug.c:59 __list_del_entry+0xb1/0xe0 [3905442.238020] list_del corruption. prev->next should be ffff902fedb43f48, but was ffff902c36bb6348 [3905442.246971] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) lov(OE) osc(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) mpt2sas mptctl mptbase bonding rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dm_service_time dm_multipath dm_mod dell_smbios dell_wmi_descriptor dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr cdc_ether ses usbnet enclosure mii sg i2c_piix4 wmi ipmi_si ipmi_devintf ipmi_msghandler tpm_crb [3905442.319369] acpi_power_meter ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) fb_sys_fops ttm mlxfw(OE) ahci mpt3sas(OE) bnxt_en tg3 libahci mlx_compat(OE) crct10dif_pclmul crct10dif_common raid_class drm crc32c_intel ptp libata scsi_transport_sas devlink drm_panel_orientation_quirks megaraid_sas pps_core [3905442.353920] CPU: 29 PID: 36662 Comm: mdt_rdpg01_050 Kdump: loaded Tainted: G W OE ------------ 3.10.0-1160.108.1.el7_lustre.pl1.x86_64 #1 [3905442.367383] Hardware name: Dell Inc. PowerEdge R6525/0N7YGH, BIOS 2.14.1 12/17/2023 [3905442.375209] Call Trace: [3905442.377836] [] dump_stack+0x19/0x1f [3905442.383148] [] __warn+0xd8/0x100 [3905442.388201] [] warn_slowpath_fmt+0x5f/0x80 [3905442.394118] [] __list_del_entry+0xb1/0xe0 [3905442.399956] [] mdt_close_internal+0x12c/0x250 [mdt] [3905442.406664] [] mdt_close+0x291/0x900 [mdt] [3905442.412601] [] tgt_request_handle+0x93f/0x19d0 [ptlrpc] [3905442.419665] [] ? ptlrpc_nrs_req_get_nolock0+0xd5/0x170 [ptlrpc] [3905442.427407] [] ? ktime_get_real_seconds+0xe/0x20 [libcfs] [3905442.434638] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [3905442.442469] [] ? task_rq_unlock+0x20/0x20 [3905442.448302] [] ? __wake_up+0x13/0x20 [3905442.453712] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [3905442.460162] [] ? ptlrpc_wait_event+0x5d0/0x5d0 [ptlrpc] [3905442.467211] [] kthread+0xd1/0xe0 [3905442.472266] [] ? insert_kthread_work+0x40/0x40 [3905442.478531] [] ret_from_fork_nospec_begin+0x7/0x21 [3905442.485144] [] ? insert_kthread_work+0x40/0x40 [3905442.491409] ---[ end trace d06f3d22018e6a1a ]--- [3905442.502505] ------------[ cut here ]------------ [3905442.507307] WARNING: CPU: 61 PID: 36662 at lib/list_debug.c:59 __list_del_entry+0xb1/0xe0 [3905442.515659] list_del corruption. prev->next should be ffff903caf5b51c8, but was ffff902fedb43f48 [3905442.524611] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) lov(OE) osc(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) mpt2sas mptctl mptbase bonding rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dm_service_time dm_multipath dm_mod dell_smbios dell_wmi_descriptor dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr cdc_ether ses usbnet enclosure mii sg i2c_piix4 wmi ipmi_si ipmi_devintf ipmi_msghandler tpm_crb [3905442.597207] acpi_power_meter ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) fb_sys_fops ttm mlxfw(OE) ahci mpt3sas(OE) bnxt_en tg3 libahci mlx_compat(OE) crct10dif_pclmul crct10dif_common raid_class drm crc32c_intel ptp libata scsi_transport_sas devlink drm_panel_orientation_quirks megaraid_sas pps_core [3905442.631733] CPU: 61 PID: 36662 Comm: mdt_rdpg01_050 Kdump: loaded Tainted: G W OE ------------ 3.10.0-1160.108.1.el7_lustre.pl1.x86_64 #1 [3905442.645195] Hardware name: Dell Inc. PowerEdge R6525/0N7YGH, BIOS 2.14.1 12/17/2023 [3905442.653021] Call Trace: [3905442.655648] [] dump_stack+0x19/0x1f [3905442.660959] [] __warn+0xd8/0x100 [3905442.666013] [] warn_slowpath_fmt+0x5f/0x80 [3905442.671935] [] __list_del_entry+0xb1/0xe0 [3905442.677782] [] mdt_close_internal+0x12c/0x250 [mdt] [3905442.684486] [] mdt_close+0x291/0x900 [mdt] [3905442.690438] [] tgt_request_handle+0x93f/0x19d0 [ptlrpc] [3905442.697683] [] ? ptlrpc_nrs_req_get_nolock0+0xd5/0x170 [ptlrpc] [3905442.705433] [] ? ktime_get_real_seconds+0xe/0x20 [libcfs] [3905442.712671] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [3905442.720501] [] ? task_rq_unlock+0x20/0x20 [3905442.726342] [] ? __wake_up+0x13/0x20 [3905442.731771] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [3905442.738233] [] ? ptlrpc_wait_event+0x5d0/0x5d0 [ptlrpc] [3905442.745287] [] kthread+0xd1/0xe0 [3905442.750349] [] ? insert_kthread_work+0x40/0x40 [3905442.756752] [] ret_from_fork_nospec_begin+0x7/0x21 [3905442.763366] [] ? insert_kthread_work+0x40/0x40 [3905442.769639] ---[ end trace d06f3d22018e6a1b ]--- [3905442.778663] ------------[ cut here ]------------ [3905442.783468] WARNING: CPU: 61 PID: 36662 at lib/list_debug.c:59 __list_del_entry+0xb1/0xe0 [3905442.791814] list_del corruption. prev->next should be ffff903047454ec8, but was ffff903caf5b51c8 [3905442.800766] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) lov(OE) osc(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) mpt2sas mptctl mptbase bonding rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dm_service_time dm_multipath dm_mod dell_smbios dell_wmi_descriptor dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr cdc_ether ses usbnet enclosure mii sg i2c_piix4 wmi ipmi_si ipmi_devintf ipmi_msghandler tpm_crb [3905442.873337] acpi_power_meter ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) fb_sys_fops ttm mlxfw(OE) ahci mpt3sas(OE) bnxt_en tg3 libahci mlx_compat(OE) crct10dif_pclmul crct10dif_common raid_class drm crc32c_intel ptp libata scsi_transport_sas devlink drm_panel_orientation_quirks megaraid_sas pps_core [3905442.908047] CPU: 61 PID: 36662 Comm: mdt_rdpg01_050 Kdump: loaded Tainted: G W OE ------------ 3.10.0-1160.108.1.el7_lustre.pl1.x86_64 #1 [3905442.921506] Hardware name: Dell Inc. PowerEdge R6525/0N7YGH, BIOS 2.14.1 12/17/2023 [3905442.929338] Call Trace: [3905442.931969] [] dump_stack+0x19/0x1f [3905442.937280] [] __warn+0xd8/0x100 [3905442.942333] [] warn_slowpath_fmt+0x5f/0x80 [3905442.948252] [] __list_del_entry+0xb1/0xe0 [3905442.954094] [] mdt_close_internal+0x12c/0x250 [mdt] [3905442.960797] [] mdt_close+0x291/0x900 [mdt] [3905442.966736] [] tgt_request_handle+0x93f/0x19d0 [ptlrpc] [3905442.973798] [] ? ptlrpc_nrs_req_get_nolock0+0xd5/0x170 [ptlrpc] [3905442.981541] [] ? ktime_get_real_seconds+0xe/0x20 [libcfs] [3905442.988772] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [3905442.996725] [] ? task_rq_unlock+0x20/0x20 [3905443.002557] [] ? __wake_up+0x13/0x20 [3905443.007968] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [3905443.014422] [] ? ptlrpc_wait_event+0x5d0/0x5d0 [ptlrpc] [3905443.021469] [] kthread+0xd1/0xe0 [3905443.026529] [] ? insert_kthread_work+0x40/0x40 [3905443.032794] [] ret_from_fork_nospec_begin+0x7/0x21 [3905443.039405] [] ? insert_kthread_work+0x40/0x40 [3905443.045671] ---[ end trace d06f3d22018e6a1c ]--- [3905443.051926] ------------[ cut here ]------------ [3905443.056722] WARNING: CPU: 61 PID: 36662 at lib/list_debug.c:59 __list_del_entry+0xb1/0xe0 [3905443.065067] list_del corruption. prev->next should be ffff9031bca85dc8, but was ffff903047454ec8 [3905443.074019] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) lov(OE) osc(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) mpt2sas mptctl mptbase bonding rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dm_service_time dm_multipath dm_mod dell_smbios dell_wmi_descriptor dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr cdc_ether ses usbnet enclosure mii sg i2c_piix4 wmi ipmi_si ipmi_devintf ipmi_msghandler tpm_crb [3905443.146436] acpi_power_meter ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) fb_sys_fops ttm mlxfw(OE) ahci mpt3sas(OE) bnxt_en tg3 libahci mlx_compat(OE) crct10dif_pclmul crct10dif_common raid_class drm crc32c_intel ptp libata scsi_transport_sas devlink drm_panel_orientation_quirks megaraid_sas pps_core [3905443.180964] CPU: 61 PID: 36662 Comm: mdt_rdpg01_050 Kdump: loaded Tainted: G W OE ------------ 3.10.0-1160.108.1.el7_lustre.pl1.x86_64 #1 [3905443.194422] Hardware name: Dell Inc. PowerEdge R6525/0N7YGH, BIOS 2.14.1 12/17/2023 [3905443.202248] Call Trace: [3905443.204873] [] dump_stack+0x19/0x1f [3905443.210189] [] __warn+0xd8/0x100 [3905443.215241] [] warn_slowpath_fmt+0x5f/0x80 [3905443.221161] [] __list_del_entry+0xb1/0xe0 [3905443.226995] [] mdt_close_internal+0x12c/0x250 [mdt] [3905443.233706] [] mdt_close+0x291/0x900 [mdt] [3905443.239643] [] tgt_request_handle+0x93f/0x19d0 [ptlrpc] [3905443.246705] [] ? ptlrpc_nrs_req_get_nolock0+0xd5/0x170 [ptlrpc] [3905443.254448] [] ? ktime_get_real_seconds+0xe/0x20 [libcfs] [3905443.261681] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [3905443.269507] [] ? task_rq_unlock+0x20/0x20 [3905443.275344] [] ? __wake_up+0x13/0x20 [3905443.280753] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [3905443.287210] [] ? ptlrpc_wait_event+0x5d0/0x5d0 [ptlrpc] [3905443.294262] [] kthread+0xd1/0xe0 [3905443.299312] [] ? insert_kthread_work+0x40/0x40 [3905443.305580] [] ret_from_fork_nospec_begin+0x7/0x21 [3905443.312191] [] ? insert_kthread_work+0x40/0x40 [3905443.318455] ---[ end trace d06f3d22018e6a1d ]--- [3905443.324426] ------------[ cut here ]------------ [3905443.329223] WARNING: CPU: 61 PID: 36662 at lib/list_debug.c:59 __list_del_entry+0xb1/0xe0 [3905443.337568] list_del corruption. prev->next should be ffff9025863f2548, but was ffff9031bca85dc8 [3905443.346518] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) lov(OE) osc(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) mpt2sas mptctl mptbase bonding rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dm_service_time dm_multipath dm_mod dell_smbios dell_wmi_descriptor dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr cdc_ether ses usbnet enclosure mii sg i2c_piix4 wmi ipmi_si ipmi_devintf ipmi_msghandler tpm_crb [3905443.418933] acpi_power_meter ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) fb_sys_fops ttm mlxfw(OE) ahci mpt3sas(OE) bnxt_en tg3 libahci mlx_compat(OE) crct10dif_pclmul crct10dif_common raid_class drm crc32c_intel ptp libata scsi_transport_sas devlink drm_panel_orientation_quirks megaraid_sas pps_core [3905443.453487] CPU: 61 PID: 36662 Comm: mdt_rdpg01_050 Kdump: loaded Tainted: G W OE ------------ 3.10.0-1160.108.1.el7_lustre.pl1.x86_64 #1 [3905443.467123] Hardware name: Dell Inc. PowerEdge R6525/0N7YGH, BIOS 2.14.1 12/17/2023 [3905443.475233] Call Trace: [3905443.477860] [] dump_stack+0x19/0x1f [3905443.483171] [] __warn+0xd8/0x100 [3905443.488224] [] warn_slowpath_fmt+0x5f/0x80 [3905443.494142] [] __list_del_entry+0xb1/0xe0 [3905443.499982] [] mdt_close_internal+0x12c/0x250 [mdt] [3905443.506688] [] mdt_close+0x291/0x900 [mdt] [3905443.512618] [] tgt_request_handle+0x93f/0x19d0 [ptlrpc] [3905443.519680] [] ? ptlrpc_nrs_req_get_nolock0+0xd5/0x170 [ptlrpc] [3905443.527425] [] ? ktime_get_real_seconds+0xe/0x20 [libcfs] [3905443.534660] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [3905443.542489] [] ? task_rq_unlock+0x20/0x20 [3905443.548319] [] ? __wake_up+0x13/0x20 [3905443.553729] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [3905443.560176] [] ? ptlrpc_wait_event+0x5d0/0x5d0 [ptlrpc] [3905443.567218] [] kthread+0xd1/0xe0 [3905443.572514] [] ? insert_kthread_work+0x40/0x40 [3905443.578781] [] ret_from_fork_nospec_begin+0x7/0x21 [3905443.585392] [] ? insert_kthread_work+0x40/0x40 [3905443.591662] ---[ end trace d06f3d22018e6a1e ]--- [3905443.597712] ------------[ cut here ]------------ [3905443.602510] WARNING: CPU: 29 PID: 36662 at lib/list_debug.c:59 __list_del_entry+0xb1/0xe0 [3905443.610858] list_del corruption. prev->next should be ffff9023a4ec0dc8, but was ffff9025863f2548 [3905443.619817] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) lov(OE) osc(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) mpt2sas mptctl mptbase bonding rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dm_service_time dm_multipath dm_mod dell_smbios dell_wmi_descriptor dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr cdc_ether ses usbnet enclosure mii sg i2c_piix4 wmi ipmi_si ipmi_devintf ipmi_msghandler tpm_crb [3905443.692232] acpi_power_meter ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) fb_sys_fops ttm mlxfw(OE) ahci mpt3sas(OE) bnxt_en tg3 libahci mlx_compat(OE) crct10dif_pclmul crct10dif_common raid_class drm crc32c_intel ptp libata scsi_transport_sas devlink drm_panel_orientation_quirks megaraid_sas pps_core [3905443.726968] CPU: 29 PID: 36662 Comm: mdt_rdpg01_050 Kdump: loaded Tainted: G W OE ------------ 3.10.0-1160.108.1.el7_lustre.pl1.x86_64 #1 [3905443.740430] Hardware name: Dell Inc. PowerEdge R6525/0N7YGH, BIOS 2.14.1 12/17/2023 [3905443.748254] Call Trace: [3905443.750883] [] dump_stack+0x19/0x1f [3905443.756204] [] __warn+0xd8/0x100 [3905443.761264] [] warn_slowpath_fmt+0x5f/0x80 [3905443.767184] [] __list_del_entry+0xb1/0xe0 [3905443.773030] [] mdt_close_internal+0x12c/0x250 [mdt] [3905443.779737] [] mdt_close+0x291/0x900 [mdt] [3905443.785687] [] tgt_request_handle+0x93f/0x19d0 [ptlrpc] [3905443.792755] [] ? ptlrpc_nrs_req_get_nolock0+0xd5/0x170 [ptlrpc] [3905443.800499] [] ? ktime_get_real_seconds+0xe/0x20 [libcfs] [3905443.807742] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [3905443.815567] [] ? task_rq_unlock+0x20/0x20 [3905443.821400] [] ? __wake_up+0x13/0x20 [3905443.826822] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [3905443.833276] [] ? ptlrpc_wait_event+0x5d0/0x5d0 [ptlrpc] [3905443.840327] [] kthread+0xd1/0xe0 [3905443.845379] [] ? insert_kthread_work+0x40/0x40 [3905443.851646] [] ret_from_fork_nospec_begin+0x7/0x21 [3905443.858260] [] ? insert_kthread_work+0x40/0x40 [3905443.864533] ---[ end trace d06f3d22018e6a1f ]--- [3905480.206631] ------------[ cut here ]------------ [3905480.211443] WARNING: CPU: 29 PID: 35158 at lib/list_debug.c:59 __list_del_entry+0xb1/0xe0 [3905480.219796] list_del corruption. prev->next should be ffff9023d5a523c8, but was ffff9023a4ec0dc8 [3905480.228755] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) lov(OE) osc(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) mpt2sas mptctl mptbase bonding rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dm_service_time dm_multipath dm_mod dell_smbios dell_wmi_descriptor dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr cdc_ether ses usbnet enclosure mii sg i2c_piix4 wmi ipmi_si ipmi_devintf ipmi_msghandler tpm_crb [3905480.301167] acpi_power_meter ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) fb_sys_fops ttm mlxfw(OE) ahci mpt3sas(OE) bnxt_en tg3 libahci mlx_compat(OE) crct10dif_pclmul crct10dif_common raid_class drm crc32c_intel ptp libata scsi_transport_sas devlink drm_panel_orientation_quirks megaraid_sas pps_core [3905480.335721] CPU: 29 PID: 35158 Comm: mdt_rdpg01_016 Kdump: loaded Tainted: G W OE ------------ 3.10.0-1160.108.1.el7_lustre.pl1.x86_64 #1 [3905480.349185] Hardware name: Dell Inc. PowerEdge R6525/0N7YGH, BIOS 2.14.1 12/17/2023 [3905480.357008] Call Trace: [3905480.359646] [] dump_stack+0x19/0x1f [3905480.364964] [] __warn+0xd8/0x100 [3905480.370018] [] warn_slowpath_fmt+0x5f/0x80 [3905480.375938] [] __list_del_entry+0xb1/0xe0 [3905480.381818] [] mdt_close_internal+0x12c/0x250 [mdt] [3905480.388528] [] mdt_close+0x291/0x900 [mdt] [3905480.394489] [] tgt_request_handle+0x93f/0x19d0 [ptlrpc] [3905480.401554] [] ? ptlrpc_nrs_req_get_nolock0+0xd5/0x170 [ptlrpc] [3905480.409298] [] ? ktime_get_real_seconds+0xe/0x20 [libcfs] [3905480.416536] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [3905480.424372] [] ? task_rq_unlock+0x20/0x20 [3905480.430209] [] ? __wake_up+0x13/0x20 [3905480.435620] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [3905480.442076] [] ? ptlrpc_wait_event+0x5d0/0x5d0 [ptlrpc] [3905480.449131] [] kthread+0xd1/0xe0 [3905480.454192] [] ? insert_kthread_work+0x40/0x40 [3905480.460467] [] ret_from_fork_nospec_begin+0x7/0x21 [3905480.467082] [] ? insert_kthread_work+0x40/0x40 [3905480.473349] ---[ end trace d06f3d22018e6a20 ]--- [3905480.478808] ------------[ cut here ]------------ [3905480.483602] WARNING: CPU: 28 PID: 35158 at lib/list_debug.c:59 __list_del_entry+0xb1/0xe0 [3905480.491945] list_del corruption. prev->next should be ffff9034d93492c8, but was ffff9023d5a523c8 [3905480.500897] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) lov(OE) osc(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) mpt2sas mptctl mptbase bonding rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dm_service_time dm_multipath dm_mod dell_smbios dell_wmi_descriptor dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr cdc_ether ses usbnet enclosure mii sg i2c_piix4 wmi ipmi_si ipmi_devintf ipmi_msghandler tpm_crb [3905480.573305] acpi_power_meter ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) fb_sys_fops ttm mlxfw(OE) ahci mpt3sas(OE) bnxt_en tg3 libahci mlx_compat(OE) crct10dif_pclmul crct10dif_common raid_class drm crc32c_intel ptp libata scsi_transport_sas devlink drm_panel_orientation_quirks megaraid_sas pps_core [3905480.607847] CPU: 28 PID: 35158 Comm: mdt_rdpg01_016 Kdump: loaded Tainted: G W OE ------------ 3.10.0-1160.108.1.el7_lustre.pl1.x86_64 #1 [3905480.621310] Hardware name: Dell Inc. PowerEdge R6525/0N7YGH, BIOS 2.14.1 12/17/2023 [3905480.629134] Call Trace: [3905480.631763] [] dump_stack+0x19/0x1f [3905480.637074] [] __warn+0xd8/0x100 [3905480.642127] [] warn_slowpath_fmt+0x5f/0x80 [3905480.648047] [] __list_del_entry+0xb1/0xe0 [3905480.653885] [] mdt_close_internal+0x12c/0x250 [mdt] [3905480.660591] [] mdt_close+0x291/0x900 [mdt] [3905480.666544] [] tgt_request_handle+0x93f/0x19d0 [ptlrpc] [3905480.673612] [] ? ptlrpc_nrs_req_get_nolock0+0xd5/0x170 [ptlrpc] [3905480.681354] [] ? ktime_get_real_seconds+0xe/0x20 [libcfs] [3905480.688593] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [3905480.696422] [] ? task_rq_unlock+0x20/0x20 [3905480.702256] [] ? __wake_up+0x13/0x20 [3905480.707665] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [3905480.714117] [] ? ptlrpc_wait_event+0x5d0/0x5d0 [ptlrpc] [3905480.721165] [] kthread+0xd1/0xe0 [3905480.726223] [] ? insert_kthread_work+0x40/0x40 [3905480.732495] [] ret_from_fork_nospec_begin+0x7/0x21 [3905480.739113] [] ? insert_kthread_work+0x40/0x40 [3905480.745380] ---[ end trace d06f3d22018e6a21 ]--- [3905480.757018] ------------[ cut here ]------------ [3905480.762007] WARNING: CPU: 28 PID: 35158 at lib/list_debug.c:59 __list_del_entry+0xb1/0xe0 [3905480.770468] list_del corruption. prev->next should be ffff9023dc3cc348, but was ffff9034d93492c8 [3905480.779419] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) lov(OE) osc(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) mpt2sas mptctl mptbase bonding rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dm_service_time dm_multipath dm_mod dell_smbios dell_wmi_descriptor dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr cdc_ether ses usbnet enclosure mii sg i2c_piix4 wmi ipmi_si ipmi_devintf ipmi_msghandler tpm_crb [3905480.851792] acpi_power_meter ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) fb_sys_fops ttm mlxfw(OE) ahci mpt3sas(OE) bnxt_en tg3 libahci mlx_compat(OE) crct10dif_pclmul crct10dif_common raid_class drm crc32c_intel ptp libata scsi_transport_sas devlink drm_panel_orientation_quirks megaraid_sas pps_core [3905480.886483] CPU: 28 PID: 35158 Comm: mdt_rdpg01_016 Kdump: loaded Tainted: G W OE ------------ 3.10.0-1160.108.1.el7_lustre.pl1.x86_64 #1 [3905480.899945] Hardware name: Dell Inc. PowerEdge R6525/0N7YGH, BIOS 2.14.1 12/17/2023 [3905480.907772] Call Trace: [3905480.910399] [] dump_stack+0x19/0x1f [3905480.915712] [] __warn+0xd8/0x100 [3905480.920765] [] warn_slowpath_fmt+0x5f/0x80 [3905480.926683] [] __list_del_entry+0xb1/0xe0 [3905480.932518] [] mdt_close_internal+0x12c/0x250 [mdt] [3905480.939226] [] mdt_close+0x291/0x900 [mdt] [3905480.945171] [] tgt_request_handle+0x93f/0x19d0 [ptlrpc] [3905480.952235] [] ? ptlrpc_nrs_req_get_nolock0+0xd5/0x170 [ptlrpc] [3905480.959981] [] ? ktime_get_real_seconds+0xe/0x20 [libcfs] [3905480.967222] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [3905480.975051] [] ? task_rq_unlock+0x20/0x20 [3905480.980881] [] ? __wake_up+0x13/0x20 [3905480.986292] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [3905480.992742] [] ? ptlrpc_wait_event+0x5d0/0x5d0 [ptlrpc] [3905480.999791] [] kthread+0xd1/0xe0 [3905481.004999] [] ? insert_kthread_work+0x40/0x40 [3905481.011267] [] ret_from_fork_nospec_begin+0x7/0x21 [3905481.017879] [] ? insert_kthread_work+0x40/0x40 [3905481.024145] ---[ end trace d06f3d22018e6a22 ]--- [3905481.032658] ------------[ cut here ]------------ [3905481.037460] WARNING: CPU: 28 PID: 35158 at lib/list_debug.c:59 __list_del_entry+0xb1/0xe0 [3905481.045811] list_del corruption. prev->next should be ffff902a8ec1a948, but was ffff9023dc3cc348 [3905481.054761] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) lov(OE) osc(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) mpt2sas mptctl mptbase bonding rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dm_service_time dm_multipath dm_mod dell_smbios dell_wmi_descriptor dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr cdc_ether ses usbnet enclosure mii sg i2c_piix4 wmi ipmi_si ipmi_devintf ipmi_msghandler tpm_crb [3905481.127308] acpi_power_meter ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) fb_sys_fops ttm mlxfw(OE) ahci mpt3sas(OE) bnxt_en tg3 libahci mlx_compat(OE) crct10dif_pclmul crct10dif_common raid_class drm crc32c_intel ptp libata scsi_transport_sas devlink drm_panel_orientation_quirks megaraid_sas pps_core [3905481.161841] CPU: 28 PID: 35158 Comm: mdt_rdpg01_016 Kdump: loaded Tainted: G W OE ------------ 3.10.0-1160.108.1.el7_lustre.pl1.x86_64 #1 [3905481.175477] Hardware name: Dell Inc. PowerEdge R6525/0N7YGH, BIOS 2.14.1 12/17/2023 [3905481.183305] Call Trace: [3905481.185929] [] dump_stack+0x19/0x1f [3905481.191242] [] __warn+0xd8/0x100 [3905481.196295] [] warn_slowpath_fmt+0x5f/0x80 [3905481.202213] [] __list_del_entry+0xb1/0xe0 [3905481.208052] [] mdt_close_internal+0x12c/0x250 [mdt] [3905481.214758] [] mdt_close+0x291/0x900 [mdt] [3905481.220700] [] tgt_request_handle+0x93f/0x19d0 [ptlrpc] [3905481.227764] [] ? ptlrpc_nrs_req_get_nolock0+0xd5/0x170 [ptlrpc] [3905481.235504] [] ? ktime_get_real_seconds+0xe/0x20 [libcfs] [3905481.242734] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [3905481.250566] [] ? task_rq_unlock+0x20/0x20 [3905481.256397] [] ? __wake_up+0x13/0x20 [3905481.261809] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [3905481.268399] [] ? ptlrpc_wait_event+0x5d0/0x5d0 [ptlrpc] [3905481.275448] [] kthread+0xd1/0xe0 [3905481.280612] [] ? insert_kthread_work+0x40/0x40 [3905481.287472] [] ret_from_fork_nospec_begin+0x7/0x21 [3905481.294091] [] ? insert_kthread_work+0x40/0x40 [3905481.300365] ---[ end trace d06f3d22018e6a23 ]--- [3905481.308037] ------------[ cut here ]------------ [3905481.312835] WARNING: CPU: 60 PID: 35176 at lib/list_debug.c:59 __list_del_entry+0xb1/0xe0 [3905481.321181] list_del corruption. prev->next should be ffff90104417c9c8, but was ffff902a8ec1a948 [3905481.330139] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) lov(OE) osc(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) mpt2sas mptctl mptbase bonding rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dm_service_time dm_multipath dm_mod dell_smbios dell_wmi_descriptor dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr cdc_ether ses usbnet enclosure mii sg i2c_piix4 wmi ipmi_si ipmi_devintf ipmi_msghandler tpm_crb [3905481.402686] acpi_power_meter ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) fb_sys_fops ttm mlxfw(OE) ahci mpt3sas(OE) bnxt_en tg3 libahci mlx_compat(OE) crct10dif_pclmul crct10dif_common raid_class drm crc32c_intel ptp libata scsi_transport_sas devlink drm_panel_orientation_quirks megaraid_sas pps_core [3905481.437213] CPU: 60 PID: 35176 Comm: mdt_rdpg01_017 Kdump: loaded Tainted: G W OE ------------ 3.10.0-1160.108.1.el7_lustre.pl1.x86_64 #1 [3905481.450672] Hardware name: Dell Inc. PowerEdge R6525/0N7YGH, BIOS 2.14.1 12/17/2023 [3905481.458497] Call Trace: [3905481.461131] [] dump_stack+0x19/0x1f [3905481.466448] [] __warn+0xd8/0x100 [3905481.471500] [] warn_slowpath_fmt+0x5f/0x80 [3905481.477419] [] __list_del_entry+0xb1/0xe0 [3905481.483259] [] mdt_close_internal+0x12c/0x250 [mdt] [3905481.489963] [] mdt_close+0x291/0x900 [mdt] [3905481.495902] [] tgt_request_handle+0x93f/0x19d0 [ptlrpc] [3905481.502964] [] ? ptlrpc_nrs_req_get_nolock0+0xd5/0x170 [ptlrpc] [3905481.510706] [] ? ktime_get_real_seconds+0xe/0x20 [libcfs] [3905481.517939] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [3905481.525908] [] ? task_rq_unlock+0x20/0x20 [3905481.531738] [] ? __wake_up+0x13/0x20 [3905481.537153] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [3905481.543609] [] ? ptlrpc_wait_event+0x5d0/0x5d0 [ptlrpc] [3905481.550660] [] kthread+0xd1/0xe0 [3905481.555709] [] ? insert_kthread_work+0x40/0x40 [3905481.561979] [] ret_from_fork_nospec_begin+0x7/0x21 [3905481.568590] [] ? insert_kthread_work+0x40/0x40 [3905481.574854] ---[ end trace d06f3d22018e6a24 ]--- [3905481.580328] ------------[ cut here ]------------ [3905481.585127] WARNING: CPU: 60 PID: 35176 at lib/list_debug.c:59 __list_del_entry+0xb1/0xe0 [3905481.593478] list_del corruption. prev->next should be ffff903cf14cb6c8, but was ffff90104417c9c8 [3905481.602432] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) lov(OE) osc(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) mpt2sas mptctl mptbase bonding rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dm_service_time dm_multipath dm_mod dell_smbios dell_wmi_descriptor dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr cdc_ether ses usbnet enclosure mii sg i2c_piix4 wmi ipmi_si ipmi_devintf ipmi_msghandler tpm_crb [3905481.674983] acpi_power_meter ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) fb_sys_fops ttm mlxfw(OE) ahci mpt3sas(OE) bnxt_en tg3 libahci mlx_compat(OE) crct10dif_pclmul crct10dif_common raid_class drm crc32c_intel ptp libata scsi_transport_sas devlink drm_panel_orientation_quirks megaraid_sas pps_core [3905481.709511] CPU: 60 PID: 35176 Comm: mdt_rdpg01_017 Kdump: loaded Tainted: G W OE ------------ 3.10.0-1160.108.1.el7_lustre.pl1.x86_64 #1 [3905481.722973] Hardware name: Dell Inc. PowerEdge R6525/0N7YGH, BIOS 2.14.1 12/17/2023 [3905481.730798] Call Trace: [3905481.733427] [] dump_stack+0x19/0x1f [3905481.738737] [] __warn+0xd8/0x100 [3905481.743791] [] warn_slowpath_fmt+0x5f/0x80 [3905481.749710] [] __list_del_entry+0xb1/0xe0 [3905481.755554] [] mdt_close_internal+0x12c/0x250 [mdt] [3905481.762256] [] mdt_close+0x291/0x900 [mdt] [3905481.768194] [] tgt_request_handle+0x93f/0x19d0 [ptlrpc] [3905481.775255] [] ? ptlrpc_nrs_req_get_nolock0+0xd5/0x170 [ptlrpc] [3905481.783002] [] ? ktime_get_real_seconds+0xe/0x20 [libcfs] [3905481.790238] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [3905481.798069] [] ? task_rq_unlock+0x20/0x20 [3905481.803900] [] ? __wake_up+0x13/0x20 [3905481.809318] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [3905481.815887] [] ? ptlrpc_wait_event+0x5d0/0x5d0 [ptlrpc] [3905481.823114] [] kthread+0xd1/0xe0 [3905481.828167] [] ? insert_kthread_work+0x40/0x40 [3905481.834434] [] ret_from_fork_nospec_begin+0x7/0x21 [3905481.841046] [] ? insert_kthread_work+0x40/0x40 [3905481.847313] ---[ end trace d06f3d22018e6a25 ]--- [3905481.854225] ------------[ cut here ]------------ [3905481.859020] WARNING: CPU: 60 PID: 35176 at lib/list_debug.c:59 __list_del_entry+0xb1/0xe0 [3905481.867365] list_del corruption. prev->next should be ffff903f0685f948, but was ffff903cf14cb6c8 [3905481.876318] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) lov(OE) osc(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) mpt2sas mptctl mptbase bonding rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dm_service_time dm_multipath dm_mod dell_smbios dell_wmi_descriptor dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr cdc_ether ses usbnet enclosure mii sg i2c_piix4 wmi ipmi_si ipmi_devintf ipmi_msghandler tpm_crb [3905481.948834] acpi_power_meter ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) fb_sys_fops ttm mlxfw(OE) ahci mpt3sas(OE) bnxt_en tg3 libahci mlx_compat(OE) crct10dif_pclmul crct10dif_common raid_class drm crc32c_intel ptp libata scsi_transport_sas devlink drm_panel_orientation_quirks megaraid_sas pps_core [3905481.983389] CPU: 60 PID: 35176 Comm: mdt_rdpg01_017 Kdump: loaded Tainted: G W OE ------------ 3.10.0-1160.108.1.el7_lustre.pl1.x86_64 #1 [3905481.996855] Hardware name: Dell Inc. PowerEdge R6525/0N7YGH, BIOS 2.14.1 12/17/2023 [3905482.004686] Call Trace: [3905482.007314] [] dump_stack+0x19/0x1f [3905482.012626] [] __warn+0xd8/0x100 [3905482.017679] [] warn_slowpath_fmt+0x5f/0x80 [3905482.023595] [] __list_del_entry+0xb1/0xe0 [3905482.029437] [] mdt_close_internal+0x12c/0x250 [mdt] [3905482.036142] [] mdt_close+0x291/0x900 [mdt] [3905482.042081] [] tgt_request_handle+0x93f/0x19d0 [ptlrpc] [3905482.049143] [] ? ptlrpc_nrs_req_get_nolock0+0xd5/0x170 [ptlrpc] [3905482.056887] [] ? ktime_get_real_seconds+0xe/0x20 [libcfs] [3905482.064132] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [3905482.071963] [] ? task_rq_unlock+0x20/0x20 [3905482.077796] [] ? __wake_up+0x13/0x20 [3905482.083207] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [3905482.089664] [] ? ptlrpc_wait_event+0x5d0/0x5d0 [ptlrpc] [3905482.096982] [] kthread+0xd1/0xe0 [3905482.102038] [] ? insert_kthread_work+0x40/0x40 [3905482.108304] [] ret_from_fork_nospec_begin+0x7/0x21 [3905482.114917] [] ? insert_kthread_work+0x40/0x40 [3905482.121179] ---[ end trace d06f3d22018e6a26 ]--- [3905482.127538] ------------[ cut here ]------------ [3905482.132359] WARNING: CPU: 24 PID: 35176 at lib/list_debug.c:59 __list_del_entry+0xb1/0xe0 [3905482.140707] list_del corruption. prev->next should be ffff903b362d2648, but was ffff903f0685f948 [3905482.149659] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) lov(OE) osc(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) mpt2sas mptctl mptbase bonding rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dm_service_time dm_multipath dm_mod dell_smbios dell_wmi_descriptor dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr cdc_ether ses usbnet enclosure mii sg i2c_piix4 wmi ipmi_si ipmi_devintf ipmi_msghandler tpm_crb [3905482.222193] acpi_power_meter ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) fb_sys_fops ttm mlxfw(OE) ahci mpt3sas(OE) bnxt_en tg3 libahci mlx_compat(OE) crct10dif_pclmul crct10dif_common raid_class drm crc32c_intel ptp libata scsi_transport_sas devlink drm_panel_orientation_quirks megaraid_sas pps_core [3905482.256738] CPU: 24 PID: 35176 Comm: mdt_rdpg01_017 Kdump: loaded Tainted: G W OE ------------ 3.10.0-1160.108.1.el7_lustre.pl1.x86_64 #1 [3905482.270201] Hardware name: Dell Inc. PowerEdge R6525/0N7YGH, BIOS 2.14.1 12/17/2023 [3905482.278034] Call Trace: [3905482.280671] [] dump_stack+0x19/0x1f [3905482.285989] [] __warn+0xd8/0x100 [3905482.291044] [] warn_slowpath_fmt+0x5f/0x80 [3905482.296963] [] __list_del_entry+0xb1/0xe0 [3905482.302821] [] mdt_close_internal+0x12c/0x250 [mdt] [3905482.309527] [] mdt_close+0x291/0x900 [mdt] [3905482.315492] [] tgt_request_handle+0x93f/0x19d0 [ptlrpc] [3905482.322556] [] ? ptlrpc_nrs_req_get_nolock0+0xd5/0x170 [ptlrpc] [3905482.330308] [] ? ktime_get_real_seconds+0xe/0x20 [libcfs] [3905482.337548] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [3905482.345379] [] ? task_rq_unlock+0x20/0x20 [3905482.351216] [] ? __wake_up+0x13/0x20 [3905482.356637] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [3905482.363091] [] ? ptlrpc_wait_event+0x5d0/0x5d0 [ptlrpc] [3905482.370145] [] kthread+0xd1/0xe0 [3905482.375195] [] ? insert_kthread_work+0x40/0x40 [3905482.381466] [] ret_from_fork_nospec_begin+0x7/0x21 [3905482.388086] [] ? insert_kthread_work+0x40/0x40 [3905482.394357] ---[ end trace d06f3d22018e6a27 ]--- [3905482.399940] ------------[ cut here ]------------ [3905482.404740] WARNING: CPU: 56 PID: 35176 at lib/list_debug.c:59 __list_del_entry+0xb1/0xe0 [3905482.413084] list_del corruption. prev->next should be ffff902198374548, but was ffff903b362d2648 [3905482.422039] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) lov(OE) osc(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) mpt2sas mptctl mptbase bonding rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dm_service_time dm_multipath dm_mod dell_smbios dell_wmi_descriptor dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr cdc_ether ses usbnet enclosure mii sg i2c_piix4 wmi ipmi_si ipmi_devintf ipmi_msghandler tpm_crb [3905482.494452] acpi_power_meter ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) fb_sys_fops ttm mlxfw(OE) ahci mpt3sas(OE) bnxt_en tg3 libahci mlx_compat(OE) crct10dif_pclmul crct10dif_common raid_class drm crc32c_intel ptp libata scsi_transport_sas devlink drm_panel_orientation_quirks megaraid_sas pps_core [3905482.529002] CPU: 56 PID: 35176 Comm: mdt_rdpg01_017 Kdump: loaded Tainted: G W OE ------------ 3.10.0-1160.108.1.el7_lustre.pl1.x86_64 #1 [3905482.542921] Hardware name: Dell Inc. PowerEdge R6525/0N7YGH, BIOS 2.14.1 12/17/2023 [3905482.550752] Call Trace: [3905482.553377] [] dump_stack+0x19/0x1f [3905482.558690] [] __warn+0xd8/0x100 [3905482.563744] [] warn_slowpath_fmt+0x5f/0x80 [3905482.569663] [] __list_del_entry+0xb1/0xe0 [3905482.575511] [] mdt_close_internal+0x12c/0x250 [mdt] [3905482.582218] [] mdt_close+0x291/0x900 [mdt] [3905482.588162] [] tgt_request_handle+0x93f/0x19d0 [ptlrpc] [3905482.595225] [] ? ptlrpc_nrs_req_get_nolock0+0xd5/0x170 [ptlrpc] [3905482.602972] [] ? ktime_get_real_seconds+0xe/0x20 [libcfs] [3905482.610209] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [3905482.618038] [] ? task_rq_unlock+0x20/0x20 [3905482.623869] [] ? __wake_up+0x13/0x20 [3905482.629291] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [3905482.635748] [] ? ptlrpc_wait_event+0x5d0/0x5d0 [ptlrpc] [3905482.643000] [] kthread+0xd1/0xe0 [3905482.648061] [] ? insert_kthread_work+0x40/0x40 [3905482.654329] [] ret_from_fork_nospec_begin+0x7/0x21 [3905482.660948] [] ? insert_kthread_work+0x40/0x40 [3905482.667212] ---[ end trace d06f3d22018e6a28 ]--- [3905482.672642] ------------[ cut here ]------------ [3905482.677437] WARNING: CPU: 56 PID: 35176 at lib/list_debug.c:59 __list_del_entry+0xb1/0xe0 [3905482.685785] list_del corruption. prev->next should be ffff9022d6fc68c8, but was ffff902198374548 [3905482.694736] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) lov(OE) osc(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) mpt2sas mptctl mptbase bonding rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dm_service_time dm_multipath dm_mod dell_smbios dell_wmi_descriptor dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr cdc_ether ses usbnet enclosure mii sg i2c_piix4 wmi ipmi_si ipmi_devintf ipmi_msghandler tpm_crb [3905482.767167] acpi_power_meter ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) fb_sys_fops ttm mlxfw(OE) ahci mpt3sas(OE) bnxt_en tg3 libahci mlx_compat(OE) crct10dif_pclmul crct10dif_common raid_class drm crc32c_intel ptp libata scsi_transport_sas devlink drm_panel_orientation_quirks megaraid_sas pps_core [3905482.801710] CPU: 56 PID: 35176 Comm: mdt_rdpg01_017 Kdump: loaded Tainted: G W OE ------------ 3.10.0-1160.108.1.el7_lustre.pl1.x86_64 #1 [3905482.815176] Hardware name: Dell Inc. PowerEdge R6525/0N7YGH, BIOS 2.14.1 12/17/2023 [3905482.823000] Call Trace: [3905482.825628] [] dump_stack+0x19/0x1f [3905482.830938] [] __warn+0xd8/0x100 [3905482.835992] [] warn_slowpath_fmt+0x5f/0x80 [3905482.841912] [] __list_del_entry+0xb1/0xe0 [3905482.847752] [] mdt_close_internal+0x12c/0x250 [mdt] [3905482.854456] [] mdt_close+0x291/0x900 [mdt] [3905482.860402] [] tgt_request_handle+0x93f/0x19d0 [ptlrpc] [3905482.867469] [] ? ptlrpc_nrs_req_get_nolock0+0xd5/0x170 [ptlrpc] [3905482.875392] [] ? ktime_get_real_seconds+0xe/0x20 [libcfs] [3905482.882629] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [3905482.890460] [] ? task_rq_unlock+0x20/0x20 [3905482.896293] [] ? __wake_up+0x13/0x20 [3905482.901711] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [3905482.908160] [] ? ptlrpc_wait_event+0x5d0/0x5d0 [ptlrpc] [3905482.915213] [] kthread+0xd1/0xe0 [3905482.920265] [] ? insert_kthread_work+0x40/0x40 [3905482.926532] [] ret_from_fork_nospec_begin+0x7/0x21 [3905482.933151] [] ? insert_kthread_work+0x40/0x40 [3905482.939418] ---[ end trace d06f3d22018e6a29 ]--- [3905482.945128] ------------[ cut here ]------------ [3905482.949920] WARNING: CPU: 56 PID: 35176 at lib/list_debug.c:59 __list_del_entry+0xb1/0xe0 [3905482.958265] list_del corruption. prev->next should be ffff902d75d49d48, but was ffff9022d6fc68c8 [3905482.967217] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) lov(OE) osc(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) mpt2sas mptctl mptbase bonding rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dm_service_time dm_multipath dm_mod dell_smbios dell_wmi_descriptor dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr cdc_ether ses usbnet enclosure mii sg i2c_piix4 wmi ipmi_si ipmi_devintf ipmi_msghandler tpm_crb [3905483.039764] acpi_power_meter ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) fb_sys_fops ttm mlxfw(OE) ahci mpt3sas(OE) bnxt_en tg3 libahci mlx_compat(OE) crct10dif_pclmul crct10dif_common raid_class drm crc32c_intel ptp libata scsi_transport_sas devlink drm_panel_orientation_quirks megaraid_sas pps_core [3905483.074297] CPU: 56 PID: 35176 Comm: mdt_rdpg01_017 Kdump: loaded Tainted: G W OE ------------ 3.10.0-1160.108.1.el7_lustre.pl1.x86_64 #1 [3905483.087759] Hardware name: Dell Inc. PowerEdge R6525/0N7YGH, BIOS 2.14.1 12/17/2023 [3905483.095585] Call Trace: [3905483.098212] [] dump_stack+0x19/0x1f [3905483.103524] [] __warn+0xd8/0x100 [3905483.108575] [] warn_slowpath_fmt+0x5f/0x80 [3905483.114496] [] __list_del_entry+0xb1/0xe0 [3905483.120333] [] mdt_close_internal+0x12c/0x250 [mdt] [3905483.127039] [] mdt_close+0x291/0x900 [mdt] [3905483.132983] [] tgt_request_handle+0x93f/0x19d0 [ptlrpc] [3905483.140040] [] ? ptlrpc_nrs_req_get_nolock0+0xd5/0x170 [ptlrpc] [3905483.147786] [] ? ktime_get_real_seconds+0xe/0x20 [libcfs] [3905483.155016] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [3905483.162845] [] ? task_rq_unlock+0x20/0x20 [3905483.168677] [] ? __wake_up+0x13/0x20 [3905483.174091] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [3905483.180544] [] ? ptlrpc_wait_event+0x5d0/0x5d0 [ptlrpc] [3905483.187587] [] kthread+0xd1/0xe0 [3905483.192643] [] ? insert_kthread_work+0x40/0x40 [3905483.198908] [] ret_from_fork_nospec_begin+0x7/0x21 [3905483.205519] [] ? insert_kthread_work+0x40/0x40 [3905483.211786] ---[ end trace d06f3d22018e6a2a ]--- [3905483.217506] ------------[ cut here ]------------ [3905483.222299] WARNING: CPU: 56 PID: 35176 at lib/list_debug.c:59 __list_del_entry+0xb1/0xe0 [3905483.230643] list_del corruption. prev->next should be ffff9032a62c7548, but was ffff902d75d49d48 [3905483.239595] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) lov(OE) osc(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) mpt2sas mptctl mptbase bonding rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dm_service_time dm_multipath dm_mod dell_smbios dell_wmi_descriptor dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr cdc_ether ses usbnet enclosure mii sg i2c_piix4 wmi ipmi_si ipmi_devintf ipmi_msghandler tpm_crb [3905483.312124] acpi_power_meter ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) fb_sys_fops ttm mlxfw(OE) ahci mpt3sas(OE) bnxt_en tg3 libahci mlx_compat(OE) crct10dif_pclmul crct10dif_common raid_class drm crc32c_intel ptp libata scsi_transport_sas devlink drm_panel_orientation_quirks megaraid_sas pps_core [3905483.346658] CPU: 56 PID: 35176 Comm: mdt_rdpg01_017 Kdump: loaded Tainted: G W OE ------------ 3.10.0-1160.108.1.el7_lustre.pl1.x86_64 #1 [3905483.360118] Hardware name: Dell Inc. PowerEdge R6525/0N7YGH, BIOS 2.14.1 12/17/2023 [3905483.368076] Call Trace: [3905483.370704] [] dump_stack+0x19/0x1f [3905483.376014] [] __warn+0xd8/0x100 [3905483.381068] [] warn_slowpath_fmt+0x5f/0x80 [3905483.386986] [] __list_del_entry+0xb1/0xe0 [3905483.392824] [] mdt_close_internal+0x12c/0x250 [mdt] [3905483.399531] [] mdt_close+0x291/0x900 [mdt] [3905483.405469] [] tgt_request_handle+0x93f/0x19d0 [ptlrpc] [3905483.412532] [] ? ptlrpc_nrs_req_get_nolock0+0xd5/0x170 [ptlrpc] [3905483.420277] [] ? ktime_get_real_seconds+0xe/0x20 [libcfs] [3905483.427506] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [3905483.435339] [] ? task_rq_unlock+0x20/0x20 [3905483.441168] [] ? __wake_up+0x13/0x20 [3905483.446582] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [3905483.453036] [] ? ptlrpc_wait_event+0x5d0/0x5d0 [ptlrpc] [3905483.460087] [] kthread+0xd1/0xe0 [3905483.465141] [] ? insert_kthread_work+0x40/0x40 [3905483.471406] [] ret_from_fork_nospec_begin+0x7/0x21 [3905483.478018] [] ? insert_kthread_work+0x40/0x40 [3905483.484285] ---[ end trace d06f3d22018e6a2b ]--- [3905501.289644] LustreError: 36699:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107913168796 granted:424654352, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3905501.320179] LustreError: 36699:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 911 previous similar messages [3905520.285947] ------------[ cut here ]------------ [3905520.290751] WARNING: CPU: 30 PID: 36668 at lib/list_debug.c:62 __list_del_entry+0x8e/0xe0 [3905520.299100] list_del corruption. next->prev should be ffff90330708d0c8, but was ffff9032e7e296c8 [3905520.308057] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) lov(OE) osc(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) mpt2sas mptctl mptbase bonding rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dm_service_time dm_multipath dm_mod dell_smbios dell_wmi_descriptor dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr cdc_ether ses usbnet enclosure mii sg i2c_piix4 wmi ipmi_si ipmi_devintf ipmi_msghandler tpm_crb [3905520.380698] acpi_power_meter ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) fb_sys_fops ttm mlxfw(OE) ahci mpt3sas(OE) bnxt_en tg3 libahci mlx_compat(OE) crct10dif_pclmul crct10dif_common raid_class drm crc32c_intel ptp libata scsi_transport_sas devlink drm_panel_orientation_quirks megaraid_sas pps_core [3905520.415250] CPU: 30 PID: 36668 Comm: mdt_rdpg01_055 Kdump: loaded Tainted: G W OE ------------ 3.10.0-1160.108.1.el7_lustre.pl1.x86_64 #1 [3905520.428714] Hardware name: Dell Inc. PowerEdge R6525/0N7YGH, BIOS 2.14.1 12/17/2023 [3905520.436541] Call Trace: [3905520.439185] [] dump_stack+0x19/0x1f [3905520.444508] [] __warn+0xd8/0x100 [3905520.449566] [] warn_slowpath_fmt+0x5f/0x80 [3905520.455494] [] __list_del_entry+0x8e/0xe0 [3905520.461348] [] mdt_close_internal+0x12c/0x250 [mdt] [3905520.468056] [] mdt_close+0x291/0x900 [mdt] [3905520.474019] [] tgt_request_handle+0x93f/0x19d0 [ptlrpc] [3905520.481198] [] ? ptlrpc_nrs_req_get_nolock0+0xd5/0x170 [ptlrpc] [3905520.488948] [] ? ktime_get_real_seconds+0xe/0x20 [libcfs] [3905520.496188] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [3905520.504021] [] ? task_rq_unlock+0x20/0x20 [3905520.509856] [] ? __wake_up+0x13/0x20 [3905520.515270] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [3905520.521727] [] ? ptlrpc_wait_event+0x5d0/0x5d0 [ptlrpc] [3905520.528780] [] kthread+0xd1/0xe0 [3905520.533837] [] ? insert_kthread_work+0x40/0x40 [3905520.540116] [] ret_from_fork_nospec_begin+0x7/0x21 [3905520.546735] [] ? insert_kthread_work+0x40/0x40 [3905520.553009] ---[ end trace d06f3d22018e6a2c ]--- [3906103.957143] LustreError: 35234:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107918596932 granted:483067692, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3906103.987677] LustreError: 35234:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 627 previous similar messages [3906705.635822] LustreError: 35149:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107934894188 granted:486160868, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3906705.666497] LustreError: 35149:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 492 previous similar messages [3907233.740493] ------------[ cut here ]------------ [3907233.745306] WARNING: CPU: 60 PID: 35177 at lib/list_debug.c:59 __list_del_entry+0xb1/0xe0 [3907233.753667] list_del corruption. prev->next should be ffff9024d2b43b48, but was ffff9032a62c7540 [3907233.762618] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) lov(OE) osc(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) mpt2sas mptctl mptbase bonding rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dm_service_time dm_multipath dm_mod dell_smbios dell_wmi_descriptor dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr cdc_ether ses usbnet enclosure mii sg i2c_piix4 wmi ipmi_si ipmi_devintf ipmi_msghandler tpm_crb [3907233.835258] acpi_power_meter ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) fb_sys_fops ttm mlxfw(OE) ahci mpt3sas(OE) bnxt_en tg3 libahci mlx_compat(OE) crct10dif_pclmul crct10dif_common raid_class drm crc32c_intel ptp libata scsi_transport_sas devlink drm_panel_orientation_quirks megaraid_sas pps_core [3907233.869840] CPU: 60 PID: 35177 Comm: mdt_rdpg01_018 Kdump: loaded Tainted: G W OE ------------ 3.10.0-1160.108.1.el7_lustre.pl1.x86_64 #1 [3907233.883303] Hardware name: Dell Inc. PowerEdge R6525/0N7YGH, BIOS 2.14.1 12/17/2023 [3907233.891138] Call Trace: [3907233.893781] [] dump_stack+0x19/0x1f [3907233.899105] [] __warn+0xd8/0x100 [3907233.904161] [] warn_slowpath_fmt+0x5f/0x80 [3907233.910089] [] __list_del_entry+0xb1/0xe0 [3907233.915950] [] mdt_close_internal+0x12c/0x250 [mdt] [3907233.922657] [] mdt_close+0x291/0x900 [mdt] [3907233.928614] [] tgt_request_handle+0x93f/0x19d0 [ptlrpc] [3907233.935679] [] ? ptlrpc_nrs_req_get_nolock0+0xd5/0x170 [ptlrpc] [3907233.943429] [] ? ktime_get_real_seconds+0xe/0x20 [libcfs] [3907233.950671] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [3907233.958504] [] ? task_rq_unlock+0x20/0x20 [3907233.964340] [] ? __wake_up+0x13/0x20 [3907233.969761] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [3907233.976214] [] ? ptlrpc_wait_event+0x5d0/0x5d0 [ptlrpc] [3907233.983271] [] kthread+0xd1/0xe0 [3907233.988328] [] ? insert_kthread_work+0x40/0x40 [3907233.994775] [] ret_from_fork_nospec_begin+0x7/0x21 [3907234.001387] [] ? insert_kthread_work+0x40/0x40 [3907234.007662] ---[ end trace d06f3d22018e6a2d ]--- [3907306.878532] LustreError: 35201:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107934880656 granted:486160868, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3907306.909242] LustreError: 35201:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 629 previous similar messages [3907909.201270] LustreError: 36691:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107918596640 granted:483067692, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3907909.231813] LustreError: 36691:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 481 previous similar messages [3908511.236403] LustreError: 35162:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107913190992 granted:424654352, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3908511.266941] LustreError: 35162:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 459 previous similar messages [3908869.262294] ------------[ cut here ]------------ [3908869.267115] WARNING: CPU: 28 PID: 35200 at lib/list_debug.c:59 __list_del_entry+0xb1/0xe0 [3908869.275611] list_del corruption. prev->next should be ffff903590ea0448, but was ffff9024d2b43b40 [3908869.284569] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) lov(OE) osc(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) mpt2sas mptctl mptbase bonding rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dm_service_time dm_multipath dm_mod dell_smbios dell_wmi_descriptor dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr cdc_ether ses usbnet enclosure mii sg i2c_piix4 wmi ipmi_si ipmi_devintf ipmi_msghandler tpm_crb [3908869.357199] acpi_power_meter ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) fb_sys_fops ttm mlxfw(OE) ahci mpt3sas(OE) bnxt_en tg3 libahci mlx_compat(OE) crct10dif_pclmul crct10dif_common raid_class drm crc32c_intel ptp libata scsi_transport_sas devlink drm_panel_orientation_quirks megaraid_sas pps_core [3908869.391746] CPU: 28 PID: 35200 Comm: mdt_rdpg01_031 Kdump: loaded Tainted: G W OE ------------ 3.10.0-1160.108.1.el7_lustre.pl1.x86_64 #1 [3908869.405207] Hardware name: Dell Inc. PowerEdge R6525/0N7YGH, BIOS 2.14.1 12/17/2023 [3908869.413034] Call Trace: [3908869.415668] [] dump_stack+0x19/0x1f [3908869.420997] [] __warn+0xd8/0x100 [3908869.426225] [] warn_slowpath_fmt+0x5f/0x80 [3908869.432153] [] __list_del_entry+0xb1/0xe0 [3908869.438036] [] mdt_close_internal+0x12c/0x250 [mdt] [3908869.444752] [] mdt_close+0x291/0x900 [mdt] [3908869.450716] [] tgt_request_handle+0x93f/0x19d0 [ptlrpc] [3908869.457786] [] ? ptlrpc_nrs_req_get_nolock0+0xd5/0x170 [ptlrpc] [3908869.465531] [] ? ktime_get_real_seconds+0xe/0x20 [libcfs] [3908869.472770] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [3908869.480602] [] ? task_rq_unlock+0x20/0x20 [3908869.486438] [] ? __wake_up+0x13/0x20 [3908869.491856] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [3908869.498317] [] ? ptlrpc_wait_event+0x5d0/0x5d0 [ptlrpc] [3908869.505368] [] kthread+0xd1/0xe0 [3908869.510431] [] ? insert_kthread_work+0x40/0x40 [3908869.516706] [] ret_from_fork_nospec_begin+0x7/0x21 [3908869.523323] [] ? insert_kthread_work+0x40/0x40 [3908869.529599] ---[ end trace d06f3d22018e6a2e ]--- [3909022.598204] ------------[ cut here ]------------ [3909022.603017] WARNING: CPU: 56 PID: 35155 at lib/list_debug.c:59 __list_del_entry+0xb1/0xe0 [3909022.611371] list_del corruption. prev->next should be ffff9017707afac8, but was ffff903590ea0440 [3909022.620328] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) lov(OE) osc(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) mpt2sas mptctl mptbase bonding rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dm_service_time dm_multipath dm_mod dell_smbios dell_wmi_descriptor dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr cdc_ether ses usbnet enclosure mii sg i2c_piix4 wmi ipmi_si ipmi_devintf ipmi_msghandler tpm_crb [3909022.692871] acpi_power_meter ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) fb_sys_fops ttm mlxfw(OE) ahci mpt3sas(OE) bnxt_en tg3 libahci mlx_compat(OE) crct10dif_pclmul crct10dif_common raid_class drm crc32c_intel ptp libata scsi_transport_sas devlink drm_panel_orientation_quirks megaraid_sas pps_core [3909022.727433] CPU: 56 PID: 35155 Comm: mdt_rdpg01_013 Kdump: loaded Tainted: G W OE ------------ 3.10.0-1160.108.1.el7_lustre.pl1.x86_64 #1 [3909022.740898] Hardware name: Dell Inc. PowerEdge R6525/0N7YGH, BIOS 2.14.1 12/17/2023 [3909022.748727] Call Trace: [3909022.751367] [] dump_stack+0x19/0x1f [3909022.756697] [] __warn+0xd8/0x100 [3909022.761755] [] warn_slowpath_fmt+0x5f/0x80 [3909022.767677] [] __list_del_entry+0xb1/0xe0 [3909022.773553] [] mdt_close_internal+0x12c/0x250 [mdt] [3909022.780437] [] mdt_close+0x291/0x900 [mdt] [3909022.786392] [] tgt_request_handle+0x93f/0x19d0 [ptlrpc] [3909022.793457] [] ? ptlrpc_nrs_req_get_nolock0+0xd5/0x170 [ptlrpc] [3909022.801208] [] ? ktime_get_real_seconds+0xe/0x20 [libcfs] [3909022.808447] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [3909022.816451] [] ? task_rq_unlock+0x20/0x20 [3909022.822290] [] ? __wake_up+0x13/0x20 [3909022.827700] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [3909022.834157] [] ? ptlrpc_wait_event+0x5d0/0x5d0 [ptlrpc] [3909022.841211] [] kthread+0xd1/0xe0 [3909022.846270] [] ? insert_kthread_work+0x40/0x40 [3909022.852540] [] ret_from_fork_nospec_begin+0x7/0x21 [3909022.859158] [] ? insert_kthread_work+0x40/0x40 [3909022.865425] ---[ end trace d06f3d22018e6a2f ]--- [3909117.457711] LustreError: 35208:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069503389524 granted:559613176, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3909117.488412] LustreError: 35208:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 492 previous similar messages [3909718.288803] LustreError: 35235:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107918585128 granted:483067692, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3909718.319342] LustreError: 35235:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 414 previous similar messages [3910319.027887] LustreError: 36653:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107828079008 granted:310379184, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3910319.058438] LustreError: 36653:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 719 previous similar messages [3910921.035974] LustreError: 36692:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107918552564 granted:483067692, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3910921.066508] LustreError: 36692:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 572 previous similar messages [3911524.187793] LustreError: 35154:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069503408148 granted:559613176, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3911524.218334] LustreError: 35154:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 420 previous similar messages [3912127.468701] LustreError: 35158:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069503405156 granted:559613176, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3912127.499239] LustreError: 35158:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 530 previous similar messages [3912253.958700] Lustre: oak-MDT0000: haven't heard from client 840b89bf-5a00-455b-b0ee-e47b15c3cc33 (at 10.50.3.48@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9010608ea800, cur 1714176511 expire 1714176361 last 1714176284 [3912253.980580] Lustre: Skipped 1 previous similar message [3912472.950192] Lustre: oak-MDT0000: haven't heard from client c8725245-292b-4251-beeb-ac3f8ee24275 (at 10.50.6.34@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff902655a7c800, cur 1714176730 expire 1714176580 last 1714176503 [3912472.972064] Lustre: Skipped 1 previous similar message [3912727.582843] LustreError: 36685:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107918556056 granted:483067692, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3912727.613387] LustreError: 36685:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 560 previous similar messages [3912995.723345] ------------[ cut here ]------------ [3912995.728153] WARNING: CPU: 24 PID: 35180 at lib/list_debug.c:59 __list_del_entry+0xb1/0xe0 [3912995.736509] list_del corruption. prev->next should be ffff903a9045c348, but was ffff9017707afac0 [3912995.745466] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) lov(OE) osc(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) mpt2sas mptctl mptbase bonding rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dm_service_time dm_multipath dm_mod dell_smbios dell_wmi_descriptor dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr cdc_ether ses usbnet enclosure mii sg i2c_piix4 wmi ipmi_si ipmi_devintf ipmi_msghandler tpm_crb [3912995.818094] acpi_power_meter ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) fb_sys_fops ttm mlxfw(OE) ahci mpt3sas(OE) bnxt_en tg3 libahci mlx_compat(OE) crct10dif_pclmul crct10dif_common raid_class drm crc32c_intel ptp libata scsi_transport_sas devlink drm_panel_orientation_quirks megaraid_sas pps_core [3912995.852659] CPU: 24 PID: 35180 Comm: mdt_rdpg01_021 Kdump: loaded Tainted: G W OE ------------ 3.10.0-1160.108.1.el7_lustre.pl1.x86_64 #1 [3912995.866120] Hardware name: Dell Inc. PowerEdge R6525/0N7YGH, BIOS 2.14.1 12/17/2023 [3912995.873947] Call Trace: [3912995.876578] [] dump_stack+0x19/0x1f [3912995.881898] [] __warn+0xd8/0x100 [3912995.886954] [] warn_slowpath_fmt+0x5f/0x80 [3912995.892875] [] __list_del_entry+0xb1/0xe0 [3912995.898912] [] mdt_close_internal+0x12c/0x250 [mdt] [3912995.905620] [] mdt_close+0x291/0x900 [mdt] [3912995.911583] [] tgt_request_handle+0x93f/0x19d0 [ptlrpc] [3912995.918661] [] ? ptlrpc_nrs_req_get_nolock0+0xd5/0x170 [ptlrpc] [3912995.926410] [] ? ktime_get_real_seconds+0xe/0x20 [libcfs] [3912995.933919] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [3912995.941933] [] ? task_rq_unlock+0x20/0x20 [3912995.947772] [] ? __wake_up+0x13/0x20 [3912995.953192] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [3912995.959646] [] ? ptlrpc_wait_event+0x5d0/0x5d0 [ptlrpc] [3912995.966708] [] kthread+0xd1/0xe0 [3912995.971765] [] ? insert_kthread_work+0x40/0x40 [3912995.978040] [] ret_from_fork_nospec_begin+0x7/0x21 [3912995.984663] [] ? insert_kthread_work+0x40/0x40 [3912995.990934] ---[ end trace d06f3d22018e6a30 ]--- [3913005.023530] ------------[ cut here ]------------ [3913005.028364] WARNING: CPU: 17 PID: 35208 at lib/list_debug.c:62 __list_del_entry+0x8e/0xe0 [3913005.036722] list_del corruption. next->prev should be ffff902afa8085c8, but was ffff9035cc9903c8 [3913005.045681] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) lov(OE) osc(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) mpt2sas mptctl mptbase bonding rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dm_service_time dm_multipath dm_mod dell_smbios dell_wmi_descriptor dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr cdc_ether ses usbnet enclosure mii sg i2c_piix4 wmi ipmi_si ipmi_devintf ipmi_msghandler tpm_crb [3913005.118089] acpi_power_meter ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) fb_sys_fops ttm mlxfw(OE) ahci mpt3sas(OE) bnxt_en tg3 libahci mlx_compat(OE) crct10dif_pclmul crct10dif_common raid_class drm crc32c_intel ptp libata scsi_transport_sas devlink drm_panel_orientation_quirks megaraid_sas pps_core [3913005.152638] CPU: 17 PID: 35208 Comm: mdt_rdpg01_039 Kdump: loaded Tainted: G W OE ------------ 3.10.0-1160.108.1.el7_lustre.pl1.x86_64 #1 [3913005.166099] Hardware name: Dell Inc. PowerEdge R6525/0N7YGH, BIOS 2.14.1 12/17/2023 [3913005.173925] Call Trace: [3913005.176560] [] dump_stack+0x19/0x1f [3913005.181886] [] __warn+0xd8/0x100 [3913005.186942] [] warn_slowpath_fmt+0x5f/0x80 [3913005.192871] [] __list_del_entry+0x8e/0xe0 [3913005.198751] [] mdt_close_internal+0x12c/0x250 [mdt] [3913005.205460] [] mdt_close+0x291/0x900 [mdt] [3913005.211431] [] tgt_request_handle+0x93f/0x19d0 [ptlrpc] [3913005.218498] [] ? ptlrpc_nrs_req_get_nolock0+0xd5/0x170 [ptlrpc] [3913005.226250] [] ? ktime_get_real_seconds+0xe/0x20 [libcfs] [3913005.233486] [] ptlrpc_server_handle_request+0x253/0xc30 [ptlrpc] [3913005.241327] [] ? task_rq_unlock+0x20/0x20 [3913005.247167] [] ? __wake_up+0x13/0x20 [3913005.252584] [] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] [3913005.259042] [] ? ptlrpc_wait_event+0x5d0/0x5d0 [ptlrpc] [3913005.266096] [] kthread+0xd1/0xe0 [3913005.271155] [] ? insert_kthread_work+0x40/0x40 [3913005.277425] [] ret_from_fork_nospec_begin+0x7/0x21 [3913005.284041] [] ? insert_kthread_work+0x40/0x40 [3913005.290316] ---[ end trace d06f3d22018e6a31 ]--- [3913087.920587] Lustre: oak-MDT0003: haven't heard from client f6320a95-2cad-454a-ace2-59d1c557ef20 (at 10.50.1.6@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901ab89b8000, cur 1714177345 expire 1714177195 last 1714177118 [3913087.942373] Lustre: Skipped 1 previous similar message [3913331.960175] LustreError: 35147:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107828062916 granted:310379184, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3913331.990712] LustreError: 35147:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 519 previous similar messages [3913478.903077] Lustre: oak-MDT0000: haven't heard from client c76cddb8-f80b-4b16-96c3-7588bfaf794f (at 10.51.3.68@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90147cd02800, cur 1714177736 expire 1714177586 last 1714177509 [3913478.924952] Lustre: Skipped 1 previous similar message [3913933.064236] LustreError: 35181:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107828054212 granted:310379184, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3913933.094780] LustreError: 35181:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 871 previous similar messages [3914533.355978] LustreError: 36696:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0203_UUID release: 18446621107913149240 granted:424654352, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3914533.386509] LustreError: 36696:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 694 previous similar messages [3915133.521496] LustreError: 35147:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0210_UUID release: 18446621107814698316 granted:2811932, total:18222449792 qmt:oak-QMT0000 pool:dt-0x0 id:7067 enforced:1 hard:18253611008 soft:0 granted:18222449792 time:0 qunit: 16384 edquot:0 may_rel:0 revoke:0 default:no [3915133.551351] LustreError: 35147:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 544 previous similar messages [3915734.290888] LustreError: 36666:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069503379588 granted:559613176, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3915734.321435] LustreError: 36666:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 497 previous similar messages [3916078.780185] Lustre: oak-MDT0000: haven't heard from client cd389d3b-04f1-45e9-af8d-5cd283592dd9 (at 10.51.5.58@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900343ba7000, cur 1714180336 expire 1714180186 last 1714180109 [3916078.802060] Lustre: Skipped 69 previous similar messages [3916245.777804] Lustre: oak-MDT0000: haven't heard from client 5e3654bb-c569-4eaf-b0ed-092d15c6f0f9 (at 10.51.12.21@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff90143ebb1c00, cur 1714180503 expire 1714180353 last 1714180276 [3916245.799762] Lustre: Skipped 3 previous similar messages [3916254.773253] Lustre: oak-MDT0003: haven't heard from client 5e3654bb-c569-4eaf-b0ed-092d15c6f0f9 (at 10.51.12.21@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901bea91c800, cur 1714180512 expire 1714180362 last 1714180285 [3916334.342285] LustreError: 35154:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST020a_UUID release: 18446744069503386908 granted:559613176, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3916334.373013] LustreError: 35154:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 635 previous similar messages [3916757.749607] Lustre: oak-MDT0000: haven't heard from client 7b77137b-60f5-4e78-ad05-c05835102290 (at 10.51.16.7@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901105704800, cur 1714181015 expire 1714180865 last 1714180788 [3916935.042419] LustreError: 35180:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107828023556 granted:310379184, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3916935.072962] LustreError: 35180:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 739 previous similar messages [3917211.730106] Lustre: oak-MDT0003: haven't heard from client 27373395-2b49-4a36-b07a-475ca915a46a (at 10.50.10.61@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff900569908800, cur 1714181469 expire 1714181319 last 1714181242 [3917211.752069] Lustre: Skipped 1 previous similar message [3917535.036920] LustreError: 36684:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107918547244 granted:483067692, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3917535.067462] LustreError: 36684:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 738 previous similar messages [3917814.701797] Lustre: oak-MDT0000: haven't heard from client 2820635f-51b5-4c20-b890-1fdc6b3c0489 (at 10.50.5.3@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff903e90376400, cur 1714182072 expire 1714181922 last 1714181845 [3917814.723591] Lustre: Skipped 7 previous similar messages [3917890.697407] Lustre: oak-MDT0000: haven't heard from client e65c3e35-f2ab-47d3-a3f6-56e196202ce3 (at 10.50.5.59@o2ib2) in 167 seconds. I think it's dead, and I am evicting it. exp ffff9030560e4400, cur 1714182148 expire 1714181998 last 1714181981 [3917890.719288] Lustre: Skipped 7 previous similar messages [3917944.695876] Lustre: oak-MDT0003: haven't heard from client e65c3e35-f2ab-47d3-a3f6-56e196202ce3 (at 10.50.5.59@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9039f6cda000, cur 1714182202 expire 1714182052 last 1714181975 [3918137.115628] LustreError: 35196:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0218_UUID release: 18446621107828035300 granted:310379184, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3918137.146185] LustreError: 35196:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 747 previous similar messages [3918738.508234] LustreError: 35163:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107918531516 granted:483067692, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3918738.538772] LustreError: 35163:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 672 previous similar messages [3919339.141265] LustreError: 36671:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0200_UUID release: 18446621107934849388 granted:486160868, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3919339.171932] LustreError: 36671:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 1240 previous similar messages [3919939.352041] LustreError: 35237:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107918472084 granted:483067692, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3919939.382579] LustreError: 35237:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 753 previous similar messages [3920539.532513] LustreError: 36675:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107918536116 granted:483067692, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3920539.563054] LustreError: 36675:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 762 previous similar messages [3921120.550290] Lustre: oak-MDT0000: haven't heard from client 9f433918-f5b1-4aa0-a124-5d5f0e544da5 (at 10.51.1.20@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff901f52bbd000, cur 1714185378 expire 1714185228 last 1714185151 [3921140.530543] LustreError: 35174:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107918508936 granted:483067692, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3921140.561085] LustreError: 35174:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 868 previous similar messages [3921659.523536] Lustre: oak-MDT0003: haven't heard from client e3f43d8d-d548-4317-b3c7-958f7c78d0ee (at 10.51.5.38@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9006a0d3cc00, cur 1714185917 expire 1714185767 last 1714185690 [3921659.545408] Lustre: Skipped 1 previous similar message [3921744.144911] LustreError: 35164:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107918529032 granted:483067692, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3921744.175446] LustreError: 35164:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 806 previous similar messages [3921880.513781] Lustre: oak-MDT0000: haven't heard from client 8aa294f8-1e5a-40b0-97de-4a05ef784a4a (at 10.51.8.67@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9004960b9800, cur 1714186138 expire 1714185988 last 1714185911 [3921880.535812] Lustre: Skipped 3 previous similar messages [3922279.498983] Lustre: oak-MDT0000: haven't heard from client 6c4da50e-dad5-4e88-a9ef-837b5550e9a5 (at 10.50.16.5@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9002fceda400, cur 1714186537 expire 1714186387 last 1714186310 [3922279.520856] Lustre: Skipped 3 previous similar messages [3922344.510161] LustreError: 36685:0:(qmt_handler.c:981:qmt_dqacq0()) $$$ Release too much! uuid:oak-MDT0000-lwp-OST0219_UUID release: 18446621107918515968 granted:483067692, total:355558508604 qmt:oak-QMT0000 pool:dt-0x0 id:7028 enforced:1 hard:375809638400 soft:0 granted:355558508604 time:0 qunit: 16777216 edquot:0 may_rel:0 revoke:0 default:no [3922344.540705] LustreError: 36685:0:(qmt_handler.c:981:qmt_dqacq0()) Skipped 1034 previous similar messages [3922355.502458] Lustre: oak-MDT0000: haven't heard from client dbdc5e22-e94e-4e7a-956f-f02560781534 (at 10.50.6.16@o2ib2) in 224 seconds. I think it's dead, and I am evicting it. exp ffff90219dabe000, cur 1714186613 expire 1714186463 last 1714186389 [3922355.524336] Lustre: Skipped 2 previous similar messages [3922902.137858] BUG: unable to handle kernel paging request at ffff9040df5e80c8 [3922902.145044] IP: [] qmt_lvbo_update+0x271/0xf00 [lquota] [3922902.152063] PGD dc58a5067 PUD 3ffb259063 PMD 3fe02f3063 PTE 8000003fdf5e8061 [3922902.159361] Oops: 0003 [#1] SMP [3922902.162803] Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) lmv(OE) mdc(OE) lov(OE) osc(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache libcfs(OE) mpt2sas mptctl mptbase bonding rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) sunrpc vfat fat dm_round_robin dm_service_time dm_multipath dm_mod dell_smbios dell_wmi_descriptor dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr cdc_ether ses usbnet enclosure mii sg i2c_piix4 wmi ipmi_si ipmi_devintf ipmi_msghandler tpm_crb [3922902.234941] acpi_power_meter ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core(OE) fb_sys_fops ttm mlxfw(OE) ahci mpt3sas(OE) bnxt_en tg3 libahci mlx_compat(OE) crct10dif_pclmul crct10dif_common raid_class drm crc32c_intel ptp libata scsi_transport_sas devlink drm_panel_orientation_quirks megaraid_sas pps_core [3922902.269349] CPU: 11 PID: 33277 Comm: qmt_reba_oak-QM Kdump: loaded Tainted: G W OE ------------ 3.10.0-1160.108.1.el7_lustre.pl1.x86_64 #1 [3922902.282893] Hardware name: Dell Inc. PowerEdge R6525/0N7YGH, BIOS 2.14.1 12/17/2023 [3922902.290710] task: ffff903882efc200 ti: ffff903885dbc000 task.ti: ffff903885dbc000 [3922902.298356] RIP: 0010:[] [] qmt_lvbo_update+0x271/0xf00 [lquota] [3922902.307767] RSP: 0018:ffff903885dbfaf0 EFLAGS: 00010216 [3922902.313252] RAX: 00000000000020c0 RBX: ffff9010ea3ed290 RCX: ffff9040df5e6000 [3922902.320549] RDX: ffff903ec1d0e420 RSI: 000000000000020c RDI: 0000000000000400 [3922902.327854] RBP: ffff903885dbfb88 R08: 0000000000000001 R09: 0000000000000004 [3922902.335152] R10: 0000000000000000 R11: f000000000000000 R12: 0000000000000001 [3922902.342449] R13: ffff9040e8e10488 R14: ffff9025ab640c00 R15: ffff901becc9d400 [3922902.349747] FS: 0000000000000000(0000) GS:ffff9020fecc0000(0000) knlGS:0000000000000000 [3922902.358007] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [3922902.363926] CR2: ffff9040df5e80c8 CR3: 0000003ff98fe000 CR4: 0000000000760ee0 [3922902.371230] PKRU: 00000000 [3922902.374108] Call Trace: [3922902.376743] [] ? ktime_get_real_seconds+0xe/0x20 [libcfs] [3922902.384083] [] ? at_measured+0x5e/0x370 [ptlrpc] [3922902.390564] [] mdt_lvbo_update+0xbb/0x140 [mdt] [3922902.396925] [] ? lustre_msg_set_transno+0x27/0xb0 [ptlrpc] [3922902.404236] [] ldlm_cb_interpret+0x122/0x740 [ptlrpc] [3922902.411118] [] ptlrpc_check_set+0x3f8/0x2290 [ptlrpc] [3922902.417998] [] ptlrpc_set_wait+0x21b/0x840 [ptlrpc] [3922902.424698] [] ? wake_up_atomic_t+0x40/0x40 [3922902.430711] [] ldlm_run_ast_work+0xd5/0x3e0 [ptlrpc] [3922902.437508] [] ldlm_glimpse_locks+0x38/0x110 [ptlrpc] [3922902.444378] [] qmt_glimpse_lock.isra.15+0x37f/0xa60 [lquota] [3922902.451857] [] ? qmt_dqacq+0x880/0x880 [lquota] [3922902.458210] [] qmt_reba_thread+0x4a4/0xa80 [lquota] [3922902.464909] [] ? qmt_glimpse_lock.isra.15+0xa60/0xa60 [lquota] [3922902.472559] [] kthread+0xd1/0xe0 [3922902.477603] [] ? insert_kthread_work+0x40/0x40 [3922902.483864] [] ret_from_fork_nospec_begin+0x7/0x21 [3922902.490474] [] ? insert_kthread_work+0x40/0x40 [3922902.496737] Code: 84 a9 0c 00 00 48 8b 40 18 48 8b 93 00 01 00 00 83 78 10 02 48 8b b8 08 01 00 00 0f 84 a9 04 00 00 48 8b 0a 48 63 c6 48 c1 e0 04 <80> 64 01 08 fb 48 8b 8b 00 01 00 00 48 8b 09 80 64 01 08 f7 48 [3922902.516965] RIP [] qmt_lvbo_update+0x271/0xf00 [lquota] [3922902.524029] RSP [3922902.527687] CR2: ffff9040df5e80c8