https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3996 Lustre <p>Running parallel-scale on Hyperion, initialization for iOR test. Formatted with ZFS. <br/> MDS crashes during test startup.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">LustreError: 8136:0:(llog_osd.c:241:llog_osd_read_header()) MGS-osd: error reading log header from [0xa:0xa:0x0]: rc = -14 2013-09-23 11:32:12 LustreError: 8136:0:(mgs_llog.c:1386:record_start_log()) MGS: can't start log lustre-params: rc = -14 2013-09-23 11:32:12 BUG: unable to handle kernel NULL pointer dereference at 00000000000000b8 </pre> </div></div> <p>Console log attached.</p> LU-3996

LustreError: 8136:0:(llog_osd.c:241:llog_osd_read_header()) MGS-osd: error reading log header from [0xa:0xa:0x0]: rc = -14

Bug Blocker Closed Duplicate Mikhail Pershin Cliff White Mon, 23 Sep 2013 18:41:05 +0000 Wed, 16 Oct 2013 03:15:42 +0000 Wed, 16 Oct 2013 03:15:42 +0000 Lustre 2.5.0 0 5 <p>There's not enough information to see why llog init fails with EFAULT, but the crash reason is obvious, we fail to test for the llog opening status and try to close not opened llog as the result.</p> <p>Patch for the crash is in <a href="http://review.whamcloud.com/7742" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7742</a></p> <p>Cliff to provide the additional debug logs.<br/> Will reassign this ticket once that info has been provided.</p> <p>Reproduced crash on 2.4.93 with panic_on_oops=0 Console log and lctl dk attached.</p> <p>Hm, the lctldk output is too late afte the crash, it starts at 11:11am and end on 11:14am on 26th, and the oops was at 11:10<br/> lctl dk needs to be run right after the crash (or you need to increase debug log buffer to a bigger value so it does not wrap super fast)</p> <p>dumplog.115529 contains the error. I have included the dumps from 30 seconds before and 30 seconds after.</p> <p>Thanks!</p> <p>Ok, so the issue is we are trying to do a 8k read and can read only smaller amount of bytes (not enough logging to see how many) from the referenced llog file.</p> <p>Right now it sounds like the underlying mgs filesystem llog is damaged, we should mount it directly and check what's up with the llog file for <span class="error">&#91;0xa:0xa:0x0&#93;</span> llog, it's probably way too short?</p> <p>Each time this has failed, it has been on a freshly formatted filesystem. I can replicate and look at the log if that is necessary</p> <p>There's a fair chance the llog is created in a bad way from format, if this is really the case, there's no log for this process.<br/> So Taking a look at llog (making sure it is short after the failure) and then keeping track on it right after reformat even before we mount lustre to confirm would be useful, and then we also need to see exactly how malformed is it, just short size, what's inside and so on.</p> <p>all logs from CONFIGS directory on MGS after crash, dumped to text with llog_reader. Same error as before: </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">2013-09-30 14:34:04 LustreError: 8465:0:(llog_osd.c:241:llog_osd_read_header()) MGS-osd: error reading log header from [0xa:0xa:0x0]: rc = -14 2013-09-30 14:34:04 LustreError: 8465:0:(mgs_llog.c:1386:record_start_log()) MGS: can't start log lustre-params: rc = -14 2013-09-30 14:34:04 BUG: unable to handle kernel NULL pointer dereference at 00000000000000b8 2013-09-30 14:34:04 IP: [&lt;ffffffffa07ffe99&gt;] llog_handle_put+0x9/0x70 [obdclass] </pre> </div></div> <p>debub_mb=1024, log dumped every 30 seconds for duration of test.</p> <p>Found an easy way to reproduce this:<br/> With clients mounted. on MGS:</p> <ol> <li>hyperion-agb5 /root &gt; lctl conf_param lustre.sys.jobid_var=procname_uid<br/> crashes immediately<br/> And, if I set <br/> export JOBID_VAR="existing"<br/> in my config, the test runs. </li> </ol> <p>Cliff, can you reproduce that issue with commit <a href="http://git.whamcloud.com/?p=fs/lustre-release.git;a=commit;h=a217228ce3e1c93fdfeb1d1aa6ff48b3f82abf83" class="external-link" target="_blank" rel="nofollow noopener">http://git.whamcloud.com/?p=fs/lustre-release.git;a=commit;h=a217228ce3e1c93fdfeb1d1aa6ff48b3f82abf83</a> ?</p> <p>No, I cannot - ran the one-line test and it does not crash. Will run IOR shortly</p> <p>Ran IOR without any crashes. Latest build fixes</p> <p>Removed fixversion as this is a duplicate.</p> Duplicate LU-3871 Development Rank 1|hzw3p3: Rank (Obsolete) 10693 Severity