[LU-3996] LustreError: 8136:0:(llog_osd.c:241:llog_osd_read_header()) MGS-osd: error reading log header from [0xa:0xa:0x0]: rc = -14

• Status: Closed
• Project: Lustre
• Affects Version/s: Lustre 2.5.0
• Type: Bug
• Priority: Blocker
• Reporter: Cliff White (Inactive)
• Assignee: Mikhail Pershin
• Resolution: Duplicate
• Severity: 3

Description

Running parallel-scale on Hyperion, initialization for iOR test. Formatted with ZFS.
MDS crashes during test startup.

LustreError: 8136:0:(llog_osd.c:241:llog_osd_read_header()) MGS-osd: error reading log header from [0xa:0xa:0x0]: rc = -14
2013-09-23 11:32:12 LustreError: 8136:0:(mgs_llog.c:1386:record_start_log()) MGS: can't start log lustre-params: rc = -14
2013-09-23 11:32:12 BUG: unable to handle kernel NULL pointer dereference at 00000000000000b8

Console log attached.

Issue Links

• is duplicated by: LU-3871 e2fsck reports inode reference count ...

Comments

[Oleg Drokin]

There's not enough information to see why llog init fails with EFAULT, but the crash reason is obvious, we fail to test for the llog opening status and try to close not opened llog as the result.

Patch for the crash is in http://review.whamcloud.com/7742

[Jodi Levi (Inactive)]

Cliff to provide the additional debug logs.
Will reassign this ticket once that info has been provided.

[Cliff White (Inactive)]

Reproduced crash on 2.4.93 with panic_on_oops=0 Console log and lctl dk attached.

[Oleg Drokin]

Hm, the lctldk output is too late afte the crash, it starts at 11:11am and end on 11:14am on 26th, and the oops was at 11:10
lctl dk needs to be run right after the crash (or you need to increase debug log buffer to a bigger value so it does not wrap super fast)

[Cliff White (Inactive)]

dumplog.115529 contains the error. I have included the dumps from 30 seconds before and 30 seconds after.

[Oleg Drokin]

Thanks!

Ok, so the issue is we are trying to do a 8k read and can read only smaller amount of bytes (not enough logging to see how many) from the referenced llog file.

Right now it sounds like the underlying mgs filesystem llog is damaged, we should mount it directly and check what's up with the llog file for [0xa:0xa:0x0] llog, it's probably way too short?

[Cliff White (Inactive)]

Each time this has failed, it has been on a freshly formatted filesystem. I can replicate and look at the log if that is necessary

[Oleg Drokin]

There's a fair chance the llog is created in a bad way from format, if this is really the case, there's no log for this process.
So Taking a look at llog (making sure it is short after the failure) and then keeping track on it right after reformat even before we mount lustre to confirm would be useful, and then we also need to see exactly how malformed is it, just short size, what's inside and so on.

[Cliff White (Inactive)]

all logs from CONFIGS directory on MGS after crash, dumped to text with llog_reader. Same error as before:

2013-09-30 14:34:04 LustreError: 8465:0:(llog_osd.c:241:llog_osd_read_header()) MGS-osd: error reading log header from [0xa:0xa:0x0]: rc = -14
2013-09-30 14:34:04 LustreError: 8465:0:(mgs_llog.c:1386:record_start_log()) MGS: can't start log lustre-params: rc = -14
2013-09-30 14:34:04 BUG: unable to handle kernel NULL pointer dereference at 00000000000000b8
2013-09-30 14:34:04 IP: [<ffffffffa07ffe99>] llog_handle_put+0x9/0x70 [obdclass]

[Cliff White (Inactive)]

debub_mb=1024, log dumped every 30 seconds for duration of test.

[Cliff White (Inactive)]

Found an easy way to reproduce this:
With clients mounted. on MGS:

1. hyperion-agb5 /root > lctl conf_param lustre.sys.jobid_var=procname_uid
   crashes immediately
   And, if I set
   export JOBID_VAR="existing"
   in my config, the test runs.

[Mikhail Pershin]

Cliff, can you reproduce that issue with commit http://git.whamcloud.com/?p=fs/lustre-release.git;a=commit;h=a217228ce3e1c93fdfeb1d1aa6ff48b3f82abf83 ?

[Cliff White (Inactive)]

No, I cannot - ran the one-line test and it does not crash. Will run IOR shortly

[Cliff White (Inactive)]

Ran IOR without any crashes. Latest build fixes

[Jodi Levi (Inactive)]

Removed fixversion as this is a duplicate.