[LU-8787] zpool containing MDT0000 out of space

• Type: Question/Request
• Resolution: Done
• Priority: Minor
• Labels: llnl
• Environment: Lustre: Build Version: 2.8.0_5.chaos
• Assignee: nasf (Inactive)
• Reporter: Olaf Faaland

Description

On a DNE file system, MDT0000 ran out of space while one or more other MDTs were in recovery.

   2016-10-31 18:26:53 [20537.964631] Lustre: Skipped 1 previous similar message
   2016-10-31 18:26:58 [20542.793836] LustreError: 31561:0:(osd_handler.c:223:osd_trans_start()) lsh-MDT0000: failed to start transaction due to ENOSPC. Metadata overhead is underestimated or grant_ratio is too low.
   2016-10-31 18:26:58 [20542.815473] LustreError: 31561:0:(osd_handler.c:223:osd_trans_start()) Skipped 39 previous similar messages
   2016-10-31 18:26:58 [20542.827434] LustreError: 31561:0:(llog_cat.c:744:llog_cat_cancel_records()) lsh-OST0009-osc-MDT0000: fail to cancel 1 of 1 llog-records: rc = -28
   2016-10-31 18:26:58 [20542.843771] LustreError: 31561:0:(osp_sync.c:1031:osp_sync_process_committed()) lsh-OST0009-osc-MDT0000: can't cancel record: -28

Obviously the first step is to increase the capacity of the pool. However, after that is done, is further action required? Should I run lfsck, or do anything else?

Issue Links

• is related to: LU-8753 Recovery already passed deadline with DNE

Comments

[nasf (Inactive)]

Do you expect that the file system is still in a consistent state, and after deleting update_log* I don't need to do anything else? That's my main question for this ticket.
...
I unfortunately don't know the workload at the time the MDT filled up, nor can I get debug logs. The problem was discovered after the servers had been rebooted.

Generally, the llog became bigger after the reboot means there are something to be recovered. But as long as your recovery complete successfully after the reboot, even if you removed the llogs, your namespace should be in consistent status unless there were some inconsistency before your reboot (for ZFS backend, it should very rare case).

[Olaf Faaland]

nasf,
I unfortunately don't know the workload at the time the MDT filled up, nor can I get debug logs. The problem was discovered after the servers had been rebooted.

[Olaf Faaland]

Di,
Do you expect that the file system is still in a consistent state, and after deleting update_log* I don't need to do anything else? That's my main question for this ticket.
thanks,
Olaf

[Di Wang]

you can delete update_log* manually as what we did on LU-8753, which I need further log.
And there is also another ticket LU-8714 about deleting update_log efficiently. I will try to work out the patch.

[nasf (Inactive)]

According to current DNE implementation, the cross-MDTs operations (in detail) will be recorded as llog under the update_log_dir for recovery purpose. For most of use cases, the llog is append only, if there are too much cross-MDTs operations, the llog will become huge. So if you can describe your operations before the out of space, that may help us to judge the issue. Anyway, if there are some Lustre kernel debug logs on the MDT0000, that will be better.

[Olaf Faaland]

Created a separate ticket LU-8794 for the large amount of space occupied by update_log_dir.

This ticket is only for the procedure to be followed when an MDT fills up, since it could happen in production and we need to know the procedure for recovering.

thanks,
Olaf

[Olaf Faaland]

There are 158 files in update_log_dir.
68 size>10GB
29 10GB > size >= 1GB
7 1GB > size >= 1M
44 size < 1M

[Olaf Faaland]

I find that update_log_dir is taking 1.1T out of the 1.4T available. Shall I create a separate ticket for that? It seems far too large to me, but maybe I'm wrong.

[Peter Jones]

Fan Yong

Could you please look into this issue?

Thanks

Peter