https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5599 Lustre <p>Lustre Version: 2.4.52<br/> Kernel: patchless_client<br/> Build: v2_4_52 0-gfdd4844-CHANGED-3.9.9-302.fc19.x86_64</p> <p>2nd Instance:<br/> -----------------<br/> From: Cledat, Romain E <br/> Sent: Monday, September 08, 2014 4:34 PM<br/> To: Bernel, BrianX D<br/> Subject: Error </p> <p>Message from syslogd@bar1 at Sep 8 16:08:51 ... <br/> kernel:<span class="error">&#91;1235195.162972&#93;</span> LustreError: 85031:0:(osc_lock.c:1129:osc_lock_enqueue()) ASSERTION( ols-&gt;ols_state == OLS_NEW ) failed: Impossible state: 4 r <br/> Message from syslogd@bar1 at Sep 8 16:08:51 ... <br/> kernel:<span class="error">&#91;1235195.193211&#93;</span> LustreError: 85031:0:(osc_lock.c:1129:osc_lock_enqueue()) LBUG</p> <p>1st Instance:<br/> -----------------<br/> From: Nickerson, Brian R <br/> Sent: Thursday, August 14, 2014 3:44 PM<br/> To: Bernel, BrianX D; Cledat, Romain E<br/> Subject: Kernel crash details</p> <p>Message from syslogd@bar4 at Aug 14 15:34:57 ...<br/> kernel:<span class="error">&#91;1216856.270451&#93;</span> LustreError: 42598:0:(osc_lock.c:1129:osc_lock_enqueue()) ASSERTION( ols-&gt;ols_state == OLS_NEW ) failed: Impossible state: 4</p> <p>Message from syslogd@bar4 at Aug 14 15:34:57 ...<br/> kernel:<span class="error">&#91;1216856.271008&#93;</span> LustreError: 42598:0:(osc_lock.c:1129:osc_lock_enqueue()) LBUG</p> <p>Message from syslogd@bar4 at Aug 14 15:34:57 ...<br/> kernel:<span class="error">&#91;1216856.271830&#93;</span> Kernel panic - not syncing: LBUG</p> Fedora 19 x86_64 on Washington Pass nodes, 1GbE &amp; FDR IB LU-5599

Lustre Error: Impossible state: 4

Bug Major Resolved Cannot Reproduce Bob Glossman Brian Bernel Tue, 9 Sep 2014 17:01:12 +0000 Tue, 24 Jul 2018 13:53:28 +0000 Tue, 24 Jul 2018 13:53:28 +0000 Lustre 2.4.3 0 7 <p>There should be a stack dump to go along with the ASSERTION - can you please acquire and attach to the ticket? In addition, it would be useful to have logs for some time before the actual assertion - are there any LustreErrors previous to this? </p> <p>Bob</p> <p>Could you please help with this issue?</p> <p>Thanks</p> <p>Peter</p> <p>the reported lustre version is "Lustre Version: 2.4.52". this suggests it was built from source or derived from a review build in between our standard release versions with names like 2.4.2 or 2.4.3. Could we get detail about how this lustre was generated or obtained? knowing the exact origin is very important to help us understand the problem.</p> <p>this may not be known but I wonder if there was any particular applications or load that was running near the time of the 2 reported panic instances.</p> <p>Hi Bob,</p> <p>Thanks so much for your help on this. The Lustre kernel for X-Stack was put together by Gabrielle Paciucci. …As for what might have been going on at the time of the problem, the only thing I have to go on at this point is that git was involved. No kdump available, I’m afraid. A cursory look at the logs doesn’t show anything glaringly amiss, but it does corroborate that garret/git was in play in the minute leading up to the Lustre error.</p> <p>(Thanks Josh, for directing me to reply by comment vs email)</p> <p>Regards,</p> <p>Brian</p> <p>Brian,<br/> The stack dumps Cliff asked about a few comments ago is still needed. Any chance of getting those?</p> <p>We do not need a kernel dump to get this information. </p> <p>Can you upload /var/log/messages or "dmesg" right after this issue is hit? </p> <p>Recent Kernel Dump information from bar1 on X-Stack:</p> <p>[ 3.911516] hpgmg-fv<span class="error">&#91;96927&#93;</span>: segfault at d ip 00007fd22c315d3e sp 00007fd20796bd80 error 6 in libocr.so<span class="error">&#91;7fd22c300000+2e000&#93;</span><br/> [ 3.252141] LustreError: 11-0: lustrefs-MDT0000-mdc-ffff881fdadfe000: Communicating with 192.168.2.201@o2ib, operation ldlm_enqueue failed with -11.<br/> [ 3.828093] LustreError: 11-0: lustrefs-MDT0000-mdc-ffff881fdadfe000: Communicating with 192.168.2.201@o2ib, operation ldlm_enqueue failed with -11.<br/> [ 4.189668] LustreError: 11-0: lustrefs-MDT0000-mdc-ffff881fdadfe000: Communicating with 192.168.2.201@o2ib, operation ldlm_enqueue failed with -11.<br/> [ 4.197074] LustreError: Skipped 1 previous similar message<br/> [ 0.136898] LustreError: 11-0: lustrefs-MDT0000-mdc-ffff881fdadfe000: Communicating with 192.168.2.201@o2ib, operation ldlm_enqueue failed with -11.<br/> [ 3.395164] hpgmg-fv<span class="error">&#91;5886&#93;</span>: segfault at 7f91ecb51d19 ip 00007f91ecb48d3e sp 00007f91450c0d10 error 7 in libocr.so<span class="error">&#91;7f91ecb33000+2e000&#93;</span><br/> [ 1.541619] LustreError: 11-0: lustrefs-MDT0000-mdc-ffff881fdadfe000: Communicating with 192.168.2.201@o2ib, operation ldlm_enqueue failed with -11.<br/> [ 1.549642] LustreError: Skipped 1 previous similar message<br/> [ 0.870999] LustreError: 11-0: lustrefs-MDT0000-mdc-ffff881fdadfe000: Communicating with 192.168.2.201@o2ib, operation ldlm_enqueue failed with -11.<br/> [ 1.648428] LustreError: 11-0: lustrefs-MDT0000-mdc-ffff881fdadfe000: Communicating with 192.168.2.201@o2ib, operation ldlm_enqueue failed with -11.<br/> [ 0.486506] hpgmg-fv<span class="error">&#91;7547&#93;</span>: segfault at 7f5a27153d19 ip 00007f5a2714ad3e sp 00007f5978fb3d10 error 7 in libocr.so<span class="error">&#91;7f5a27135000+2e000&#93;</span><br/> [ 0.824354] hpgmg-fv<span class="error">&#91;11206&#93;</span>: segfault at d ip 00007f4f43557d3e sp 00007f4e9bacfd10 error 6 in libocr.so<span class="error">&#91;7f4f43542000+2e000&#93;</span><br/> [ 3.258284] hpgmg-fv<span class="error">&#91;11745&#93;</span>: segfault at d ip 00007fb77412ad3e sp 00007fb6cd6a4d10 error 6 in libocr.so<span class="error">&#91;7fb774115000+2e000&#93;</span><br/> [ 2.546239] LustreError: 1599:0:(file.c:163:ll_close_inode_openhandle()) inode 144118057617399716 mdc close failed: rc = -13<br/> [ 2.559492] LustreError: 1599:0:(file.c:163:ll_close_inode_openhandle()) inode 144117957591658911 mdc close failed: rc = -13<br/> [ 1.467034] LustreError: 1599:0:(file.c:163:ll_close_inode_openhandle()) inode 144118025052861374 mdc close failed: rc = -13<br/> [ 1.476398] LustreError: 1599:0:(file.c:163:ll_close_inode_openhandle()) Skipped 40 previous similar messages<br/> [ 2.468531] LustreError: 1599:0:(file.c:163:ll_close_inode_openhandle()) inode 144118025052858204 mdc close failed: rc = -13<br/> [ 2.478511] LustreError: 1599:0:(file.c:163:ll_close_inode_openhandle()) Skipped 246 previous similar messages<br/> [ 0.176572] LustreError: 1599:0:(file.c:163:ll_close_inode_openhandle()) inode 144118025052850460 mdc close failed: rc = -13<br/> [ 0.186522] LustreError: 1599:0:(file.c:163:ll_close_inode_openhandle()) Skipped 473 previous similar messages<br/> [ 0.030234] LustreError: 1599:0:(file.c:163:ll_close_inode_openhandle()) inode 144118024767594400 mdc close failed: rc = -13<br/> [ 0.039727] LustreError: 1599:0:(file.c:163:ll_close_inode_openhandle()) Skipped 39 previous similar messages<br/> [ 2.904305] hpgmg-fv invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0<br/> [ 2.904309] hpgmg-fv cpuset=/ mems_allowed=0-1<br/> [ 2.904311] Pid: 11573, comm: hpgmg-fv Tainted: GF O 3.9.9-302.fc19.x86_64 #1</p> <p>Related correspondence and screen errors:</p> <p>Yep, Vincent confirms that he was doing a checkout of the repo at the time… Definitely GIT + Lustre. And now they have a full kernel dump <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/>.</p> <p>Romain</p> <p>From: Romain Cledat &lt;romain.e.cledat@intel.com&gt;<br/> Date: Saturday, November 8, 2014 at 12:19 PM<br/> To: "Bernel, BrianX D" &lt;brianx.d.bernel@intel.com&gt;<br/> Subject: Your first kdump <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Hello,</p> <p>It seems bar1 rebooted by itself some 7h ago. After some investigation, I think it crashed and was rebooted due to kdump (yeah)</p> <p>Nov 8 04:51:12 bar1 systemd-logind<span class="error">&#91;748&#93;</span>: New session 17884 of user vincentc.<br/> Nov 7 22:01:56 bar1 rsyslogd: <span class="error">&#91;origin software=&quot;rsyslogd&quot; swVersion=&quot;7.2.6&quot; x-pid=&quot;737&quot; x-info=&quot;http://www.rsyslog.com&quot;&#93;</span> start</p> <p>I think the Nov 7 date is because the clock get reset to a bad value when rebooting. At the end of the reboot, you have:</p> <p>Nov 7 22:03:11 bar1 systemd<span class="error">&#91;1&#93;</span>: Startup finished in 2.269s (kernel) + 3.678s (initrd) + 1min 19.515s (userspace) = 1min 25.463s.<br/> Nov 7 22:03:11 bar1 abrt-server<span class="error">&#91;2784&#93;</span>: No actions are found for event 'notify'<br/> Nov 8 05:03:23 bar1 chronyd<span class="error">&#91;717&#93;</span>: Selected source 149.20.68.17<br/> Nov 8 05:03:23 bar1 chronyd<span class="error">&#91;717&#93;</span>: System clock wrong by 25198.383400 seconds, adjustment started<br/> Nov 8 05:03:23 bar1 chronyd<span class="error">&#91;717&#93;</span>: System clock was stepped by 25198.383 seconds<br/> Nov 8 05:03:23 bar1 systemd<span class="error">&#91;1&#93;</span>: Time has been changed</p> <p>So the machine seems to have been down between 4h51 and 5h03. And sure enough there is something in /var/crash. Relevant lines:<br/> [ 2.431054] LustreError: 29800:0:(osc_lock.c:1129:osc_lock_enqueue()) ASSERTION( ols-&gt;ols_state == OLS_NEW ) failed: Impossible state: 4<br/> [ 2.441581] LustreError: 29800:0:(osc_lock.c:1129:osc_lock_enqueue()) LBUG<br/> [ 2.452016] Pid: 29800, comm: git<br/> [ 2.452017] \x0aCall Trace:</p> <p>What do you know, it happens to be Lustre again <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/>.</p> <p>Romain</p> <p>PS: I am asking Vincent to confirm it was him and it was git. 5 in the morning though… </p> <p>NASA Ames hit this bug this morning on its 2.4.3 -7nasC client running sles11sp3 kernel.</p> <p>It looks like <a href="https://jira.whamcloud.com/browse/LU-3433" title="Encountered a assertion for the ols_state being set to a impossible state" class="issue-link" data-issue-key="LU-3433"><del>LU-3433</del></a>, and patch for 2.4 is here <a href="http://review.whamcloud.com/#/c/9194/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/9194/</a>, we also better has <a href="https://jira.whamcloud.com/browse/LU-4558" title="Crash in cl_lock_put on racer" class="issue-link" data-issue-key="LU-4558"><del>LU-4558</del></a> patch land for 2.4 <a href="http://review.whamcloud.com/#/c/9876/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/9876/</a></p> <p>2.5 already has them.</p> <p>We had both of them in our 2.4.3-7nasC.</p> <p>Can you try apply these patches?</p> <p><a href="http://review.whamcloud.com/#/c/8530/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/8530/</a><br/> <a href="http://review.whamcloud.com/#/c/9783/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/9783/</a><br/> <a href="http://review.whamcloud.com/#/c/9194/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/9194/</a><br/> <a href="http://review.whamcloud.com/#/c/9876/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/9876/</a></p> <p>Jay</p> <p>I think that it makes sense to open a separate ticket for the NASA issue because the environments are really quite different and the root cause could be different</p> <p>Peter</p> <p>Peter,</p> <p>Will do.</p> <p>Jay</p> <p>Is there set of instructions somewhere on how to apply these patches?</p> <p>Out of date ticket</p> Related LU-5492 LU-5910 Development End date Wed, 18 May 2016 17:01:12 +0000 Epic/Theme Kernel Lustre-2.4 Panic Project Rank 1|hzwvtb: Rank (Obsolete) 15656 Severity Start date Tue, 9 Sep 2014 17:01:12 +0000