[LU-12912] fsck looping

• Status: Resolved
• Project: Lustre
• Type: Bug
• Priority: Minor
• Reporter: Joe Mervini
• Assignee: Peter Jones
• Resolution: Fixed

Environment

DDN sfa12k IB attached to Dell R730 servers. TOSS 3.5.1 (rhel 7.5)/Lustre 2.10.5/e2fsprogs-1.42.13.wc6-7.el7.x86_64

• Severity: 3

Description

An OST on one of our systems went read-only last week. Today I unmounted the particular OST and ran a read-only fsck against the target. During the fsck the check got stuck on one inode and kept repeating the message:

Block <foo> (<bar>) causes symlink to be too big. IGNORE.

(repeats 12 times)

Too many illegal blocks in inode 33023185.

Clear inode? no

Suppress message? no

 

This continually repeats. Since it was a read-only fsck we terminated it and remounted the file system without going any further with the file system repair without consulting with you first. 

This is on a classified system so providing explicit detail will be difficult. Note: This problem is similar to LU-12386 that was posted last year.

Issue Links

• is related to: LU-12913 fsck found > 1M multilply-claimed blocks

Comments

[Andreas Dilger]

The e2fsck run is looping because it is stuck being told not to fix the inode and also not to stop printing the message. If you run "e2fsck -fy" it will fix the problem by zeroing out the broken inode.

Note that it is recommended to run e2fsck from the latest version, which is 1.45.3-wc1 at this time. It is also a good idea to log the e2fsck output so it is available in. the future if needed.

[Peter Jones]

Joe

Is the system in question one of the supported on-prem deployments or our AWS Lustre service without support?

Peter

[Joe Mervini]

Thanks Andreas. That's kind of what we though but wanted to make sure. Thanks for the e2fsprogs update recommendation.

 

Peter - This is through our maintenance support via DDN. Contract was placed on 3/27/19.

[Ruth Klundt (Inactive)]

The latest e2fsprogs I see is 1.45.2 here:

https://downloads.whamcloud.com/public/e2fsprogs/latest/el7/RPMS/x86_64/

 

is there a 1.45.3-wc1 somewhere else? 

[Andreas Dilger]

Hi Ruth, sorry but I got the version number wrong. The latest version tagged in our git repo is 1.45.2-wc1.

[Ruth Klundt (Inactive)]

Thanks for the confirmation  I figured so. 

[Peter Jones]

Am I correct in thinking that we can close this ticket too?

[Joe Mervini]

I ran fsck against the file system with the new e2fsprogs first with the preen option which abort and then again with the the '-y' option. After the fsck completed I repeated the 'fsck -fy' to ensure it came back clean and remounted the file system.

Everything is good. We can close this issue as resolved.

[Joe Mervini]

Must have been posting at the same time. I planned to run the fsck this morning, which I did and all is good.

[Peter Jones]

Great