[LU-8787] zpool containing MDT0000 out of space

• Type: Question/Request
• Resolution: Done
• Priority: Minor
• Labels: llnl
• Environment: Lustre: Build Version: 2.8.0_5.chaos
• Assignee: nasf (Inactive)
• Reporter: Olaf Faaland

Description

On a DNE file system, MDT0000 ran out of space while one or more other MDTs were in recovery.

   2016-10-31 18:26:53 [20537.964631] Lustre: Skipped 1 previous similar message
   2016-10-31 18:26:58 [20542.793836] LustreError: 31561:0:(osd_handler.c:223:osd_trans_start()) lsh-MDT0000: failed to start transaction due to ENOSPC. Metadata overhead is underestimated or grant_ratio is too low.
   2016-10-31 18:26:58 [20542.815473] LustreError: 31561:0:(osd_handler.c:223:osd_trans_start()) Skipped 39 previous similar messages
   2016-10-31 18:26:58 [20542.827434] LustreError: 31561:0:(llog_cat.c:744:llog_cat_cancel_records()) lsh-OST0009-osc-MDT0000: fail to cancel 1 of 1 llog-records: rc = -28
   2016-10-31 18:26:58 [20542.843771] LustreError: 31561:0:(osp_sync.c:1031:osp_sync_process_committed()) lsh-OST0009-osc-MDT0000: can't cancel record: -28

Obviously the first step is to increase the capacity of the pool. However, after that is done, is further action required? Should I run lfsck, or do anything else?

Issue Links

• is related to: LU-8753 Recovery already passed deadline with DNE

Comments

[Olaf Faaland]

Basic advice that we should delete the update logs and then run lfsck is a sufficient answer.

This occurred during DNE2 testing with Lustre 2.8, which we have decided not to work at any further.  Instead we will test DNE2 when we start testing Lustre 2.10.x.  So we will test the advice only if we encounter the problem again, and in that case will file a new ticket.

[nasf (Inactive)]

Olaf,
Have you tried to remove the huge llogs as Wangdi suggested? Any feedback for that?
Thanks

[Di Wang]

Olaf:
it might be affected, but the chance is normally low as nasf said. Anyway I would suggest you to run lfsck to check and fix the consistency after you delete the update_logs.

[nasf (Inactive)]

Do you expect that the file system is still in a consistent state, and after deleting update_log* I don't need to do anything else? That's my main question for this ticket.
...
I unfortunately don't know the workload at the time the MDT filled up, nor can I get debug logs. The problem was discovered after the servers had been rebooted.

Generally, the llog became bigger after the reboot means there are something to be recovered. But as long as your recovery complete successfully after the reboot, even if you removed the llogs, your namespace should be in consistent status unless there were some inconsistency before your reboot (for ZFS backend, it should very rare case).

[Olaf Faaland]

nasf,
I unfortunately don't know the workload at the time the MDT filled up, nor can I get debug logs. The problem was discovered after the servers had been rebooted.

[Olaf Faaland]

Di,
Do you expect that the file system is still in a consistent state, and after deleting update_log* I don't need to do anything else? That's my main question for this ticket.
thanks,
Olaf

[Di Wang]

you can delete update_log* manually as what we did on LU-8753, which I need further log.
And there is also another ticket LU-8714 about deleting update_log efficiently. I will try to work out the patch.

[nasf (Inactive)]

According to current DNE implementation, the cross-MDTs operations (in detail) will be recorded as llog under the update_log_dir for recovery purpose. For most of use cases, the llog is append only, if there are too much cross-MDTs operations, the llog will become huge. So if you can describe your operations before the out of space, that may help us to judge the issue. Anyway, if there are some Lustre kernel debug logs on the MDT0000, that will be better.

[Olaf Faaland]

Created a separate ticket LU-8794 for the large amount of space occupied by update_log_dir.

This ticket is only for the procedure to be followed when an MDT fills up, since it could happen in production and we need to know the procedure for recovering.

thanks,
Olaf

[Olaf Faaland]

There are 158 files in update_log_dir.
68 size>10GB
29 10GB > size >= 1GB
7 1GB > size >= 1M
44 size < 1M