[LU-3996] LustreError: 8136:0:(llog_osd.c:241:llog_osd_read_header()) MGS-osd: error reading log header from [0xa:0xa:0x0]: rc = -14

• Type: Bug
• Resolution: Duplicate
• Priority: Blocker
• Affects Version/s: Lustre 2.5.0
• Severity: 3
• Assignee: Mikhail Pershin
• Reporter: Cliff White (Inactive)

Description

Running parallel-scale on Hyperion, initialization for iOR test. Formatted with ZFS.
MDS crashes during test startup.

LustreError: 8136:0:(llog_osd.c:241:llog_osd_read_header()) MGS-osd: error reading log header from [0xa:0xa:0x0]: rc = -14
2013-09-23 11:32:12 LustreError: 8136:0:(mgs_llog.c:1386:record_start_log()) MGS: can't start log lustre-params: rc = -14
2013-09-23 11:32:12 BUG: unable to handle kernel NULL pointer dereference at 00000000000000b8

Console log attached.

Issue Links

• is duplicated by: LU-3871 e2fsck reports inode reference count error

Comments

[Jodi Levi (Inactive)]

Removed fixversion as this is a duplicate.

[Cliff White (Inactive)]

Ran IOR without any crashes. Latest build fixes

[Cliff White (Inactive)]

No, I cannot - ran the one-line test and it does not crash. Will run IOR shortly

[Mikhail Pershin]

Cliff, can you reproduce that issue with commit http://git.whamcloud.com/?p=fs/lustre-release.git;a=commit;h=a217228ce3e1c93fdfeb1d1aa6ff48b3f82abf83 ?

[Cliff White (Inactive)]

Found an easy way to reproduce this:
With clients mounted. on MGS:

1. hyperion-agb5 /root > lctl conf_param lustre.sys.jobid_var=procname_uid
   crashes immediately
   And, if I set
   export JOBID_VAR="existing"
   in my config, the test runs.

[Cliff White (Inactive)]

debub_mb=1024, log dumped every 30 seconds for duration of test.

[Cliff White (Inactive)]

all logs from CONFIGS directory on MGS after crash, dumped to text with llog_reader. Same error as before:

2013-09-30 14:34:04 LustreError: 8465:0:(llog_osd.c:241:llog_osd_read_header()) MGS-osd: error reading log header from [0xa:0xa:0x0]: rc = -14
2013-09-30 14:34:04 LustreError: 8465:0:(mgs_llog.c:1386:record_start_log()) MGS: can't start log lustre-params: rc = -14
2013-09-30 14:34:04 BUG: unable to handle kernel NULL pointer dereference at 00000000000000b8
2013-09-30 14:34:04 IP: [<ffffffffa07ffe99>] llog_handle_put+0x9/0x70 [obdclass]

[Oleg Drokin]

There's a fair chance the llog is created in a bad way from format, if this is really the case, there's no log for this process.
So Taking a look at llog (making sure it is short after the failure) and then keeping track on it right after reformat even before we mount lustre to confirm would be useful, and then we also need to see exactly how malformed is it, just short size, what's inside and so on.

[Cliff White (Inactive)]

Each time this has failed, it has been on a freshly formatted filesystem. I can replicate and look at the log if that is necessary

[Oleg Drokin]

Thanks!

Ok, so the issue is we are trying to do a 8k read and can read only smaller amount of bytes (not enough logging to see how many) from the referenced llog file.

Right now it sounds like the underlying mgs filesystem llog is damaged, we should mount it directly and check what's up with the llog file for [0xa:0xa:0x0] llog, it's probably way too short?