https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-8320 Lustre <p>MDS crash with LBUG.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">0&gt;LustreError: 39313:0:(llog_osd.c:338:llog_osd_write_rec()) ASSERTION( llh ) failed: ^M &lt;0&gt;LustreError: 39313:0:(llog_osd.c:338:llog_osd_write_rec()) LBUG^M &lt;4&gt;Pid: 39313, comm: mdt02_049^M &lt;4&gt;^M &lt;4&gt;Call Trace:^M &lt;4&gt; [&lt;ffffffffa048b895&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs]^M &lt;4&gt; [&lt;ffffffffa048be97&gt;] lbug_with_loc+0x47/0xb0 [libcfs]^M &lt;4&gt; [&lt;ffffffffa05bed55&gt;] llog_osd_write_rec+0xfb5/0x1370 [obdclass]^M &lt;4&gt; [&lt;ffffffffa0d46ecb&gt;] ? dynlock_unlock+0x16b/0x1d0 [osd_ldiskfs]^M &lt;4&gt; [&lt;ffffffffa0d2e5d2&gt;] ? iam_path_release+0x42/0x70 [osd_ldiskfs]^M &lt;4&gt; [&lt;ffffffffa0590438&gt;] llog_write_rec+0xc8/0x290 [obdclass]^M &lt;4&gt; [&lt;ffffffffa059910d&gt;] llog_cat_add_rec+0xad/0x480 [obdclass]^M &lt;4&gt; [&lt;ffffffffa0590231&gt;] llog_add+0x91/0x1d0 [obdclass]^M &lt;4&gt; [&lt;ffffffffa0fd04f7&gt;] osp_sync_add_rec+0x247/0xad0 [osp]^M &lt;4&gt; [&lt;ffffffffa0fd0e2b&gt;] osp_sync_add+0x7b/0x80 [osp]^M &lt;4&gt; [&lt;ffffffffa0fc27d6&gt;] osp_object_destroy+0x106/0x150 [osp]^M &lt;4&gt; [&lt;ffffffffa0f068e7&gt;] lod_object_destroy+0x1a7/0x350 [lod]^M &lt;4&gt; [&lt;ffffffffa0f74880&gt;] mdd_finish_unlink+0x210/0x3d0 [mdd]^M &lt;4&gt; [&lt;ffffffffa0f65d35&gt;] ? mdd_attr_check_set_internal+0x275/0x2c0 [mdd]^M &lt;4&gt; [&lt;ffffffffa0f75306&gt;] mdd_unlink+0x8c6/0xca0 [mdd]^M &lt;4&gt; [&lt;ffffffffa0e37788&gt;] mdo_unlink+0x18/0x50 [mdt]^M &lt;4&gt; [&lt;ffffffffa0e3b005&gt;] mdt_reint_unlink+0x835/0x1030 [mdt]^M &lt;4&gt; [&lt;ffffffffa0e37571&gt;] mdt_reint_rec+0x41/0xe0 [mdt]^M &lt;4&gt; [&lt;ffffffffa0e1ced3&gt;] mdt_reint_internal+0x4c3/0x780 [mdt]^M &lt;4&gt; [&lt;ffffffffa0e1d1d4&gt;] mdt_reint+0x44/0xe0 [mdt]^M &lt;4&gt; [&lt;ffffffffa0e1fada&gt;] mdt_handle_common+0x52a/0x1470 [mdt]^M &lt;4&gt; [&lt;ffffffffa0e5c5f5&gt;] mds_regular_handle+0x15/0x20 [mdt]^M &lt;4&gt; [&lt;ffffffffa07750c5&gt;] ptlrpc_server_handle_request+0x385/0xc00 [ptlrpc]^M &lt;4&gt; [&lt;ffffffffa048c5ae&gt;] ? cfs_timer_arm+0xe/0x10 [libcfs]^M &lt;4&gt; [&lt;ffffffffa049d8d5&gt;] ? lc_watchdog_touch+0x65/0x170 [libcfs]^M &lt;4&gt; [&lt;ffffffffa076da69&gt;] ? ptlrpc_wait_event+0xa9/0x2d0 [ptlrpc]^M &lt;4&gt; [&lt;ffffffff81057779&gt;] ? __wake_up_common+0x59/0x90^M &lt;4&gt; [&lt;ffffffffa077789d&gt;] ptlrpc_main+0xafd/0x1780 [ptlrpc]^M &lt;4&gt; [&lt;ffffffff8100c28a&gt;] child_rip+0xa/0x20^M &lt;4&gt; [&lt;ffffffffa0776da0&gt;] ? ptlrpc_main+0x0/0x1780 [ptlrpc]^M &lt;4&gt; [&lt;ffffffff8100c280&gt;] ? child_rip+0x0/0x20^M &lt;4&gt;^M &lt;0&gt;Kernel panic - not syncing: LBUG^M &lt;4&gt;Pid: 39313, comm: mdt02_049 Tainted: G --------------- T 2.6.32-504.30.3.el6.20151008.x86_64.lustre253 #1^M &lt;4&gt;Call Trace:^M &lt;4&gt; [&lt;ffffffff81564fb9&gt;] ? panic+0xa7/0x190^M &lt;4&gt; [&lt;ffffffffa048beeb&gt;] ? lbug_with_loc+0x9b/0xb0 [libcfs]^M &lt;4&gt; [&lt;ffffffffa05bed55&gt;] ? llog_osd_write_rec+0xfb5/0x1370 [obdclass]^M &lt;4&gt; [&lt;ffffffffa0d46ecb&gt;] ? dynlock_unlock+0x16b/0x1d0 [osd_ldiskfs]^M &lt;4&gt; [&lt;ffffffffa0d2e5d2&gt;] ? iam_path_release+0x42/0x70 [osd_ldiskfs]^M &lt;4&gt; [&lt;ffffffffa0590438&gt;] ? llog_write_rec+0xc8/0x290 [obdclass]^M &lt;4&gt; [&lt;ffffffffa059910d&gt;] ? llog_cat_add_rec+0xad/0x480 [obdclass]^M &lt;4&gt; [&lt;ffffffffa0590231&gt;] ? llog_add+0x91/0x1d0 [obdclass]^M &lt;4&gt; [&lt;ffffffffa0fd04f7&gt;] ? osp_sync_add_rec+0x247/0xad0 [osp]^M &lt;4&gt; [&lt;ffffffffa0fd0e2b&gt;] ? osp_sync_add+0x7b/0x80 [osp]^M &lt;4&gt; [&lt;ffffffffa0fc27d6&gt;] ? osp_object_destroy+0x106/0x150 [osp]^M &lt;4&gt; [&lt;ffffffffa0f068e7&gt;] ? lod_object_destroy+0x1a7/0x350 [lod]^M &lt;4&gt; [&lt;ffffffffa0f74880&gt;] ? mdd_finish_unlink+0x210/0x3d0 [mdd]^M &lt;4&gt; [&lt;ffffffffa0f65d35&gt;] ? mdd_attr_check_set_internal+0x275/0x2c0 [mdd]^M &lt;4&gt; [&lt;ffffffffa0f75306&gt;] ? mdd_unlink+0x8c6/0xca0 [mdd]^M &lt;4&gt; [&lt;ffffffffa0e37788&gt;] ? mdo_unlink+0x18/0x50 [mdt]^M &lt;4&gt; [&lt;ffffffffa0e3b005&gt;] ? mdt_reint_unlink+0x835/0x1030 [mdt]^M &lt;4&gt; [&lt;ffffffffa0e37571&gt;] ? mdt_reint_rec+0x41/0xe0 [mdt]^M &lt;4&gt; [&lt;ffffffffa0e1ced3&gt;] ? mdt_reint_internal+0x4c3/0x780 [mdt]^M &lt;4&gt; [&lt;ffffffffa0e1d1d4&gt;] ? mdt_reint+0x44/0xe0 [mdt]^M &lt;4&gt; [&lt;ffffffffa0e1fada&gt;] ? mdt_handle_common+0x52a/0x1470 [mdt]^M &lt;4&gt; [&lt;ffffffffa0e5c5f5&gt;] ? mds_regular_handle+0x15/0x20 [mdt]^M &lt;4&gt; [&lt;ffffffffa07750c5&gt;] ? ptlrpc_server_handle_request+0x385/0xc00 [ptlrpc]^M &lt;4&gt; [&lt;ffffffffa048c5ae&gt;] ? cfs_timer_arm+0xe/0x10 [libcfs]^M &lt;4&gt; [&lt;ffffffffa049d8d5&gt;] ? lc_watchdog_touch+0x65/0x170 [libcfs]^M &lt;4&gt; [&lt;ffffffffa076da69&gt;] ? ptlrpc_wait_event+0xa9/0x2d0 [ptlrpc]^M &lt;4&gt; [&lt;ffffffff81057779&gt;] ? __wake_up_common+0x59/0x90^M &lt;4&gt; [&lt;ffffffffa077789d&gt;] ? ptlrpc_main+0xafd/0x1780 [ptlrpc]^M &lt;4&gt; [&lt;ffffffff8100c28a&gt;] ? child_rip+0xa/0x20^M &lt;4&gt; [&lt;ffffffffa0776da0&gt;] ? ptlrpc_main+0x0/0x1780 [ptlrpc]^M &lt;4&gt; [&lt;ffffffff8100c280&gt;] ? child_rip+0x0/0x20^M </pre> </div></div> LU-8320

:(llog_osd.c:338:llog_osd_write_rec()) ASSERTION( llh ) failed:

Bug Critical Resolved Fixed Mikhail Pershin Mahmoud Hanafi Thu, 23 Jun 2016 18:13:10 +0000 Thu, 14 Jun 2018 21:41:19 +0000 Mon, 1 Aug 2016 05:45:12 +0000 Lustre 2.9.0 0 9 <p>After the reboot and remount we hit the same LBUG!</p> <p>This is lustre 2.5.3</p> <p>Looking into details supplied</p> <p>any other messages before this?</p> <p>no. </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">[-- MARK -- Thu Jun 23 09:00:00 2016]^M format at ldlm_pool.c:628:ldlm_pool_recalc doesn't end in newline^M LustreError: 39313:0:(llog_osd.c:338:llog_osd_write_rec()) ASSERTION( llh ) failed: ^M LustreError: 39313:0:(llog_osd.c:338:llog_osd_write_rec()) LBUG^M Pid: 39313, comm: mdt02_049^M ^M Call Trace:^M [&lt;ffffffffa048b895&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs]^M [&lt;ffffffffa048be97&gt;] lbug_with_loc+0x47/0xb0 [libcfs]^M ^M Entering kdb (current=0xffff8834a41b8ab0, pid 39313) on processor 16 Oops: (<span class="code-keyword">null</span>)^M due to oops @ 0x0^M kdba_dumpregs: pt_regs not available, use bt* or pid to select a different task^M </pre> </div></div> <p>Our git repo for 2.5.3 is at <br/> <a href="https://github.com/NASAEarthExchange/lustre-nas-fe/commits/nas-2.5.3" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/NASAEarthExchange/lustre-nas-fe/commits/nas-2.5.3</a></p> <p>Do you require any additional info?</p> <p>Any ideas on how we can at least get the mdt mounted and bring up the filesystem?</p> <p>can you see local variables easily?<br/> We wonder what is loghandle-&gt;lgh_id value?<br/> The current suspicion is you have a corrupted llog file somehow and in order to analyze and repair it we need to know what's the id so that we don't need to nuke all of them.</p> <p>What was the LU that the tool was part of?</p> <p><a href="https://jira.whamcloud.com/browse/LU-6696" title="ASSERTION( rc == 0 || rc == LLOG_PROC_BREAK ) failed: 0 changes, 0 in progress, 0 in flight: -5" class="issue-link" data-issue-key="LU-6696"><del>LU-6696</del></a> </p> <p>I am a bit confused how exactly to use the tool. You need me to run it on file in /O/?</p> <p>Yes, pretty much.<br/> If you can get the llog id from the variable - on that one.<br/> Otherwise - I guess on all of them (do you have many?) in some reverse data order, I imagine until it fixes something (it should print a corresponding message)</p> <p>Most of the files are returning this</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">nbp2-mds /mnt/lustre/nbp2-mdt/O/1/d2 # llog_catfix 12226 Header size : 8192 Time : Thu Jun 23 09:04:29 2016 <span class="code-object">Number</span> of records: 1544 ----------------------- Llog is not catalog, llh_size: 0, need 64 </pre> </div></div> <p>hm, yes, the tool only fixes catalog files.<br/> Anyway it can tell them apart.<br/> Do you have too many to go through all of them?</p> <p>yes there are 11k files under /O</p> <p>can we just delete all? will it not regenerate them. </p> <p>You can delete them but hat would lead to llog records not replayed - in other words - space leakage on your OSTs.</p> <p>The alternative is to find the llog id of the log that's bad - either from the crash or since it always crashes - you can just insert a printk there rebuild just for the MDS and run it again to get it?</p> <p>Can we assume it is one of the files modified in the last day or so that is the problem, and hit them all with something like this?</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>find /mnt/lustre/nbp2-mdt/O/1 -type f -mtime -2 | grep -v LAST_ID | xargs -t -I {} llog_catfix {} 2&gt;&amp;1 | tee /tmp/llog_catfix.out </pre> </div></div> <p>I think that's reasonable to assume, but there's no guarantee. The records might have been there since last reboot too.</p> <p>In the worst case you can always remove the CATALOGS file I imagine and it'll forget all about it (with the leaked space problem of course - so save that file too if you do that).<br/> Best of all is to find the llog id of the problematic llog and threat just that.</p> <p>so that lgh_id=</p> {...} <p> - what's inside?</p> <p>Actually, this must be a wrong one? lgh_hdr is not NULL here.</p> <p>Also, surprisingly the log record is a configuration one somehow:<br/> OBD_CFG_REC = LLOG_OP_MAGIC | 0x20000,</p> <p>I used systemtap to trace the function and this is when we hit the lbug</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> 0 llog_process_th(9252):-&gt;llog_osd_write_rec env={.le_ctx={...}, .le_ses=0xffff881c7aca7ba0} loghandle={.lgh_lock={...}, .lgh_hdr_lock={...}, .lgh_id={...}, .lgh_hdr=0x0, .lgh_obj=0xffff883e7ccb9d40, .lgh_last_idx=0, .lgh_cur_idx=0, .lgh_cur_offset=0, .lgh_ctxt=0xffff883e7d6e8640, .u={...}, .lgh_name=<span class="code-quote">"&lt;unknown&gt;"</span>, .private_data=0xffff883e8410a340, .lgh_logops=0xffffffffa1013a80, .lgh_refcount={...}} rec={.lrh_len=64, .lrh_index=0, .lrh_type=274989056, .lrh_id=0} reccookie={.lgc_lgl={...}, .lgc_subsys=0, .lgc_index=0, .lgc_padding=0} cookiecount=1 buf= </pre> </div></div> <p>Can you get the .lgh_id= content please? That's what should tell us the filename</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">0 llog_process_th(9553):-&gt;llog_osd_write_rec env={.le_ctx={.lc_tags=268435457, .lc_state=2, .lc_thread=0x0, .lc_value=0xffff881d24328600, .lc_remember={.next=0xffff883e8f777b78, .prev=0xffff883e8f777b78}, .lc_version=37, .lc_cookie=0}, .le_ses=0xffff883e8f777ba0} loghandle={.lgh_lock={.count=-4294967295, .wait_lock={.raw_lock={.slock=0}}, .wait_list={.next=0xffff881c6e34fbd0, .prev=0xffff881c6e34fbd0}}, .lgh_hdr_lock={.raw_lock={.slock=0}}, .lgh_id={.lgl_oi={&lt;union&gt;={.oi={.oi_id=12744, .oi_seq=1}, .oi_fid={.f_seq=12744, .f_oid=1, .f_ver=0}}}, .lgl_og </pre> </div></div> <p>So the file we are loking at is named either 12744 or 31c8.</p> <p>Try the fixing program on it and if it does not help, just move it away (please retain it still for further analysis.</p> <p>found this file </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">nbp2-mds /mnt/lustre/nbp2-mdt/O # find . -name <span class="code-quote">"12744"</span> ./1/d8/12744 </pre> </div></div> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> llog_catfix ./1/d8/12744 Header size : 8192 Time : Tue Feb 23 14:44:48 2016 <span class="code-object">Number</span> of records: 8 ----------------------- Llog is not catalog, llh_size: 0, need 64 </pre> </div></div> <p>removed the file and mounted without lbug, so far. osp-sync threads are running</p> <p>Filesystem is back up. Would you like to get a copy of the bad file?</p> <p>Since we have seen a form of this issue a couple times now, can we get the work that was started in <a href="https://jira.whamcloud.com/browse/LU-6696" title="ASSERTION( rc == 0 || rc == LLOG_PROC_BREAK ) failed: 0 changes, 0 in progress, 0 in flight: -5" class="issue-link" data-issue-key="LU-6696"><del>LU-6696</del></a> and <a href="https://jira.whamcloud.com/browse/LU-7011" title="Kernel part of llog subsystem can do self-repairing in some cases" class="issue-link" data-issue-key="LU-7011">LU-7011</a> updated and landed?</p> <p>Yes, please attach the file here</p> <p>FYI, it looks like we hit the same bug again:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>&lt;0&gt;LustreError: 37590:0:(llog_osd.c:338:llog_osd_write_rec()) ASSERTION( llh ) failed: &lt;0&gt;LustreError: 37590:0:(llog_osd.c:338:llog_osd_write_rec()) LBUG &lt;4&gt;Pid: 37590, comm: mdt03_120 &lt;4&gt; &lt;4&gt;Call Trace: &lt;4&gt; [&lt;ffffffffa0487895&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] &lt;4&gt; [&lt;ffffffffa0487e97&gt;] lbug_with_loc+0x47/0xb0 [libcfs] &lt;4&gt; [&lt;ffffffffa05bad55&gt;] llog_osd_write_rec+0xfb5/0x1370 [obdclass] &lt;4&gt; [&lt;ffffffffa0d49ecb&gt;] ? dynlock_unlock+0x16b/0x1d0 [osd_ldiskfs] &lt;4&gt; [&lt;ffffffffa0d315d2&gt;] ? iam_path_release+0x42/0x70 [osd_ldiskfs] &lt;4&gt; [&lt;ffffffffa058c438&gt;] llog_write_rec+0xc8/0x290 [obdclass] &lt;4&gt; [&lt;ffffffffa059510d&gt;] llog_cat_add_rec+0xad/0x480 [obdclass] &lt;4&gt; [&lt;ffffffffa058c231&gt;] llog_add+0x91/0x1d0 [obdclass] &lt;4&gt; [&lt;ffffffffa0fd34f7&gt;] osp_sync_add_rec+0x247/0xad0 [osp] &lt;4&gt; [&lt;ffffffffa0fd3e2b&gt;] osp_sync_add+0x7b/0x80 [osp] &lt;4&gt; [&lt;ffffffffa0fc57d6&gt;] osp_object_destroy+0x106/0x150 [osp] &lt;4&gt; [&lt;ffffffffa0f098e7&gt;] lod_object_destroy+0x1a7/0x350 [lod] &lt;4&gt; [&lt;ffffffffa0f77880&gt;] mdd_finish_unlink+0x210/0x3d0 [mdd] &lt;4&gt; [&lt;ffffffffa0f68d35&gt;] ? mdd_attr_check_set_internal+0x275/0x2c0 [mdd] &lt;4&gt; [&lt;ffffffffa0f78306&gt;] mdd_unlink+0x8c6/0xca0 [mdd] &lt;4&gt; [&lt;ffffffffa0e3a788&gt;] mdo_unlink+0x18/0x50 [mdt] &lt;4&gt; [&lt;ffffffffa0e3e005&gt;] mdt_reint_unlink+0x835/0x1030 [mdt] &lt;4&gt; [&lt;ffffffffa0e3a571&gt;] mdt_reint_rec+0x41/0xe0 [mdt] &lt;4&gt; [&lt;ffffffffa0e1fed3&gt;] mdt_reint_internal+0x4c3/0x780 [mdt] &lt;4&gt; [&lt;ffffffffa0e201d4&gt;] mdt_reint+0x44/0xe0 [mdt] &lt;4&gt; [&lt;ffffffffa0e22ada&gt;] mdt_handle_common+0x52a/0x1470 [mdt] &lt;4&gt; [&lt;ffffffffa0e5f5f5&gt;] mds_regular_handle+0x15/0x20 [mdt] &lt;4&gt; [&lt;ffffffffa07710c5&gt;] ptlrpc_server_handle_request+0x385/0xc00 [ptlrpc] &lt;4&gt; [&lt;ffffffffa04998d5&gt;] ? lc_watchdog_touch+0x65/0x170 [libcfs] &lt;4&gt; [&lt;ffffffffa0769a69&gt;] ? ptlrpc_wait_event+0xa9/0x2d0 [ptlrpc] &lt;4&gt; [&lt;ffffffffa077389d&gt;] ptlrpc_main+0xafd/0x1780 [ptlrpc] &lt;4&gt; [&lt;ffffffff8100c28a&gt;] child_rip+0xa/0x20 &lt;4&gt; [&lt;ffffffffa0772da0&gt;] ? ptlrpc_main+0x0/0x1780 [ptlrpc] &lt;4&gt; [&lt;ffffffff8100c280&gt;] ? child_rip+0x0/0x20 &lt;4&gt; &lt;0&gt;Kernel panic - not syncing: LBUG </pre> </div></div> <p>We will try to use yesterday's process to identify and fix a bad llog.</p> <p>Please stand by for more info and let us know if there are additional debug steps we should take.</p> <p>This is the file with bad record causing the LBUG.</p> <p>Today's crash was also from a file dated in Feb. Since we can't tell what files are bad I delete all files from Feb.</p> <p>I think we must have more of these corrupted files. Is there a quick method for finding them? </p> <p>Mike</p> <p>Please can you advise as to what is not handled in the attached llog file and how this could be better handled?</p> <p>Thanks</p> <p>Peter</p> <p>Would it make sense to deal with this assertion rather than just an LBUG. </p> <p>If llog_catfix can't read the file and errors out</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Llog is not catalog, llh_size: 0, need 64 </pre> </div></div> <p>means that the file is corrupted and can potentially trigger this lbug.</p> <p>First of all the llog_init_handle() should handle llog_read_header() errors correctly, not allowing the further use of such llog_handle. This will prevent similar lbugs(). It is the origin of that issue. </p> <p>Can we expect a patch? Any idea what is causing the corruption?</p> <p>Mahmoud, the message about 'Llog is not catalog' is not an error or corruption, it is just saying that llog file is not catalog as expected by llog_catfix tool. I am checking 12744.save now. It looks like plain llog with unlink records in it, I am still looking through it.</p> <p>yes, I am working on patch at least to handle llog_read_header() errors gracefully.</p> <p>Any updates?</p> <p>Hello, am I right that you are using 2.5.3 Lustre with additional patches? Are there patches related to llog subsystem? Could you make a list of them?</p> <p>I have found one suspicious thing in corrupted llog, checking the code now to understand how that may happen.</p> <p>Mike is setting up a github account. I will provide the details to Jay so he can grant the appropriate access to see NASA's tree</p> <p>FYI, we have a mix of 2.5.3, 2.7.1, and 2.7.2. We are moving all of them to 2.7.2 but that will take some time so we would need patch for 2.5.3 and 2.7.1. Jay will give you access to the repo.</p> <p>Mahmoud, in your very first comment about systemtap info, the llog_process_th is 9252 but in the next one it is 9553. Can that be this is another thread and related llog file is different from that one causing assertion? Do you still have any data related to that or similar lbug?</p> <p>The 9252 was for a previous crash where I didn't print out the full data structure. The second time (9553) I printing the correct structures. <br/> We did have a crash the next day where I used the systemtap script to find the file and delete it. But i didn't save a copy.</p> <p>OK, got it. Another question then - there should be CATALOG file with just list of catalog llog ids in it, could you look in it and provide those catalog llogs for inspection? No need to remove them, I want just check how big they are and what is inside.</p> <p>Mike Pershin (mike.pershin@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/21128" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/21128</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8320" title=":(llog_osd.c:338:llog_osd_write_rec()) ASSERTION( llh ) failed:" class="issue-link" data-issue-key="LU-8320"><del>LU-8320</del></a> llog: prevent llog ID re-use.<br/> Project: fs/lustre-release<br/> Branch: b2_5<br/> Current Patch Set: 1<br/> Commit: a051f8c40337d9a577bee88dab1b50477d84061d</p> <p>I think I've found the reason of this issue, it looks like the newly generated llog ID may cycle through zero and match some very old llog files. In that case it is considered as new llog (not initialized yet, but since it exists on disk several check are passed while they shouldn't and we have that assertion. That scenario fits well with what I see in llog file provided and assertion with llh == NULL, as well as with your comment that problem files are very old.</p> <p>Great!. do you still need to the CATALOGS file? Is the an issue with 2.7? if so we will need a patch for that as well.</p> <p>Why would we have old llog files still around?</p> <p>Also I'd recommend to apply patch from <a href="https://jira.whamcloud.com/browse/LU-5297" title="osp_sync_thread can&#39;t handle invalid record gracefully" class="issue-link" data-issue-key="LU-5297"><del>LU-5297</del></a> in addition to <a href="https://jira.whamcloud.com/browse/LU-7079" title="OSP shouldn&#39;t discard requests due to imp_peer_committed_transno" class="issue-link" data-issue-key="LU-7079"><del>LU-7079</del></a> (already in your tree) which might help to avoid having very old llogs.<br/> Consider also the patch from <a href="https://jira.whamcloud.com/browse/LU-4528" title="osd_trans_exec_op()) ASSERTION( oti-&gt;oti_declare_ops_rb[rb] &gt; 0 ) failed: rb = 0" class="issue-link" data-issue-key="LU-4528"><del>LU-4528</del></a> (<a href="http://review.whamcloud.com/#/c/11751/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/11751/</a>) which wasn't merged to 2.5 - it helps to avoid several types of corruptions we had in past. It is already in 2.7 so maybe this is not so critical if you are moving to 2.7.</p> <p>Mahmoud, there was a bug in OSP which caused some llogs to be not processed after some moment, so they stays forever. This was fixed by <a href="https://jira.whamcloud.com/browse/LU-7079" title="OSP shouldn&#39;t discard requests due to imp_peer_committed_transno" class="issue-link" data-issue-key="LU-7079"><del>LU-7079</del></a> which was merged in your tree at May 25, may it be so that failed node had no such patch applied? Also patch from <a href="https://jira.whamcloud.com/browse/LU-5297" title="osp_sync_thread can&#39;t handle invalid record gracefully" class="issue-link" data-issue-key="LU-5297"><del>LU-5297</del></a> solved similar problem which may cause stuck llog files.</p> <p>I don't need those CATALOGS file right now, but if you will have time, I'd like to look at llog files in it to check how many plain llogs are in use.</p> <p>Mike Pershin (mike.pershin@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/21130" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/21130</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8320" title=":(llog_osd.c:338:llog_osd_write_rec()) ASSERTION( llh ) failed:" class="issue-link" data-issue-key="LU-8320"><del>LU-8320</del></a> llog: prevent llog ID re-use.<br/> Project: fs/lustre-release<br/> Branch: b2_7<br/> Current Patch Set: 1<br/> Commit: d5e1cfbd9bd5bfdcd6d5c6b029e4cf578c9e75b8</p> <p>The 2.5.3 lustre server still running 2.5.3-6nasS, and <a href="https://jira.whamcloud.com/browse/LU-7079" title="OSP shouldn&#39;t discard requests due to imp_peer_committed_transno" class="issue-link" data-issue-key="LU-7079"><del>LU-7079</del></a> patch was included in 2.5.3-6.1nasS.</p> <p><a href="https://jira.whamcloud.com/browse/LU-5297" title="osp_sync_thread can&#39;t handle invalid record gracefully" class="issue-link" data-issue-key="LU-5297"><del>LU-5297</del></a> patch has not landed to b2_7_fe yet. I cherry-picked it to our nas-2.7.1 and nas-2.7.2 anyway (locally, not pushed to github yet.) OTOH, <a href="https://jira.whamcloud.com/browse/LU-5297" title="osp_sync_thread can&#39;t handle invalid record gracefully" class="issue-link" data-issue-key="LU-5297"><del>LU-5297</del></a> patch caused conflicts in nas-2.5.3. Since there is a workaround (ie, find the offending file and remove it), I think we do not need <a href="https://jira.whamcloud.com/browse/LU-5297" title="osp_sync_thread can&#39;t handle invalid record gracefully" class="issue-link" data-issue-key="LU-5297"><del>LU-5297</del></a> on nas-2.5.3.</p> <p>So, we have <a href="https://jira.whamcloud.com/browse/LU-4528" title="osd_trans_exec_op()) ASSERTION( oti-&gt;oti_declare_ops_rb[rb] &gt; 0 ) failed: rb = 0" class="issue-link" data-issue-key="LU-4528"><del>LU-4528</del></a>, <a href="https://jira.whamcloud.com/browse/LU-7079" title="OSP shouldn&#39;t discard requests due to imp_peer_committed_transno" class="issue-link" data-issue-key="LU-7079"><del>LU-7079</del></a>, <a href="https://jira.whamcloud.com/browse/LU-6696" title="ASSERTION( rc == 0 || rc == LLOG_PROC_BREAK ) failed: 0 changes, 0 in progress, 0 in flight: -5" class="issue-link" data-issue-key="LU-6696"><del>LU-6696</del></a>, and <a href="https://jira.whamcloud.com/browse/LU-5297" title="osp_sync_thread can&#39;t handle invalid record gracefully" class="issue-link" data-issue-key="LU-5297"><del>LU-5297</del></a> on nas-2.7.1 and nas-2.7.2. I tried <a href="https://jira.whamcloud.com/browse/LU-8320" title=":(llog_osd.c:338:llog_osd_write_rec()) ASSERTION( llh ) failed:" class="issue-link" data-issue-key="LU-8320"><del>LU-8320</del></a> and it was applied cleanly on nas-2.7.x, but I will wait until it gets reviews.</p> <p>Mike Pershin (mike.pershin@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/21144" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/21144</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8320" title=":(llog_osd.c:338:llog_osd_write_rec()) ASSERTION( llh ) failed:" class="issue-link" data-issue-key="LU-8320"><del>LU-8320</del></a> llog: prevent llog ID re-use.<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 722f308635f118d00a5c4a44fa72d18986ccdac9</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/21144/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/21144/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8320" title=":(llog_osd.c:338:llog_osd_write_rec()) ASSERTION( llh ) failed:" class="issue-link" data-issue-key="LU-8320"><del>LU-8320</del></a> llog: prevent llog ID re-use.<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: a93ede18ababa3fe1ae8f4a5f92e868589a58cb6</p> <p>Landed for 2.9</p> <p>Please re-open until the backport patch lands to 2.7 FE.</p> Related Development Rank 1|hzyflb: Rank (Obsolete) 9223372036854775807 Severity