https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9410 Lustre <p>We had 2 OSS and 3 different OST crash with bitmap corrupted messages.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Apr 3 18:38:16 nbp1-oss6 kernel: LDISKFS-fs error (device dm-42): ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap <span class="code-keyword">for</span> group 245659corrupted: 32768 blocks free in bitmap, 0 - in gd Apr 3 18:38:16 nbp1-oss6 kernel: Apr 3 18:38:16 nbp1-oss6 kernel: Aborting journal on device dm-3. Apr 3 18:38:16 nbp1-oss6 kernel: LDISKFS-fs (dm-42): Remounting filesystem read-only Apr 3 18:38:16 nbp1-oss6 kernel: LDISKFS-fs error (device dm-42): ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap <span class="code-keyword">for</span> group 245660corrupted: 32768 blocks free in bitmap, 0 - in gd </pre> </div></div> <p>These errors were on 2 different backend RAID devices. Note worthy  items: <br/> 1 .The filesystem was +90% full and 1/2 of the data was deleted.<br/> 2. OSTs are formatted with " -E packed_meta_blocks=1 "</p> LU-9410

on-disk bitmap corrupted

Bug Critical Resolved Fixed nasf Mahmoud Hanafi Thu, 27 Apr 2017 01:40:52 +0000 Thu, 22 Mar 2018 17:22:35 +0000 Mon, 28 Aug 2017 07:05:03 +0000 Lustre 2.7.0 Lustre 2.10.1 Lustre 2.11.0 0 13 <p>There is a patch <a href="https://review.whamcloud.com/16679" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/16679</a> "<a href="https://jira.whamcloud.com/browse/LU-7114" title="ldiskfs: corrupted bitmaps handling patches" class="issue-link" data-issue-key="LU-7114"><del>LU-7114</del></a> ldiskfs: corrupted bitmaps handling patches" that allows ldiskfs to handle this error more gracefully. </p> <p>We hit this this issue again. We are trying to determine root cause and eliminate Intel CAS as a possible source. Is FSCK expected to detected and fix these types of errors?</p> <p>Some background. We been running with 2.7 on all our OSS for sometime and haven't see this error. A few months ago we expanded 3 OSSes with additional 12 OSTs bring the total to 24 osts per OSS. These are the OSS that have hit this issue. This has occurred on different back-end raids and no errors are logged on the raids. Typically the errors is seen during high load on the OSS. </p> <p>We need a way to debug and find the root cause of the issue. We are open to installing a debug patch. After the last crash the OST was not scanned with fsck. If these errors are real corruption on the disk would an fsck find them?</p> <p>Please advise. </p> <p>Fan Yong</p> <p>Could you please advise here?</p> <p>Thanks</p> <p>Peter</p> <p>in ldiskfs_mb_init_cache between the time the bitmap is read and then checked in ldiskfs_mb_generate_from_pa isn't possible for the bitmaps to changed.</p> <p>We have ever hit similar troubles with similar message as following:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap for group xxx corrupted: mmmm blocks free in bitmap, nnnn - in gd </pre> </div></div> <p>For almost all of these cases, 'mmmm' &gt; 'nnnn'. That means the bitmap contains more available blocks than the statistic in the group desc. One suspected point is related with ldiskfs_ext_walk_space. Would you please to try the patch <a href="https://review.whamcloud.com/#/c/21603/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/21603/</a> from <a href="https://jira.whamcloud.com/browse/LU-8410" title="fiemap vs walk race" class="issue-link" data-issue-key="LU-8410"><del>LU-8410</del></a>?</p> <p>We hit this bug with the patch from <a href="https://jira.whamcloud.com/browse/LU-8410" title="fiemap vs walk race" class="issue-link" data-issue-key="LU-8410"><del>LU-8410</del></a>.</p> <p> </p> <p>Have you ever run <tt>e2fsck</tt> after the bitmap corruption? What does the <tt>e2fsck</tt> report?</p> <p>We had 2 crashes and we ran e2fsck both time. It didn't find any thing.</p> <p> </p> <p> </p> <p>Then what is your kernel version?</p> <p>we have hit it with</p> <p>2.6.32-642.15.1.el6</p> <p>and 2.6.32-642.13.1.el6</p> <p>We just recovered from a crash and fsck show nothing.</p> <p>(e2fsck -fp)</p> <p>It seems some in-RAM data corruption. We have hit similar trouble at other customer sites. Currently, we suspect that it is some kernel issue and may be fixed via the kernel patch "Addresses-Google-Bug: 2828254". One of our partners is verifying that whether the trouble can be fixed via such patch.</p> <p>Do we have the patch ported to 2.7?</p> <p> </p> <p>It is EXT4 itself patch, we are still verifying it.</p> <p>We had a crash were running fsck did find lots of wrong free block counts.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Free blocks count wrong <span class="code-keyword">for</span> group #326001 (0, counted=32768). Fix&lt;y&gt;? yes Free blocks count wrong <span class="code-keyword">for</span> group #326002 (0, counted=32768). Fix&lt;y&gt;? yes Free blocks count wrong <span class="code-keyword">for</span> group #326003 (0, counted=32768). Fix&lt;y&gt;? yes Free blocks count wrong <span class="code-keyword">for</span> group #326004 (0, counted=32768). </pre> </div></div> <p>Then the EXT4 patch may not cover your case, have to fix it via <tt>e2fsck</tt>.</p> <p>have you ported the patch to centos6.x yet. We are going to try to port the patch over to centos6 but if you have done so it would save us the work.</p> <p> </p> <p> </p> <p>You mean the EXT4 patch "Addresses-Google-Bug: 2828254", right? We have not ported such patch yet, we are not 100% sure that it is really useful for fixing our current in-RAM data corruption, still in verifying. But as you described in the comment <a href="https://jira.hpdd.intel.com/browse/LU-9410?focusedCommentId=203483&amp;page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-203483" class="external-link" target="_blank" rel="nofollow noopener">https://jira.hpdd.intel.com/browse/LU-9410?focusedCommentId=203483&amp;page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-203483</a>, means there is real on-disk data corruption. I am afraid that only porting such patch may be not enough for your case.</p> <p>What about a debug patch? We are hitting this every few hours on our production file system.</p> <p>I just got the latest feedback from our partner minutes ago, they still hit the in-RAM bitmap corruption after applying such EXT4 patch. So we have to make new investigation. Sorry for that.</p> <p>Would you please to describe what operations you did that may trigger your trouble few hour ago? Please upload the latest logs for the corruption. My understand is that the corruption still occurred on the new expended OSTs, right?</p> <p>The filesystem has 20 OSS and 360 OSTs. We have been seen bitmap corruption and remount to ro every few hours on different osts and OSSs. 9 of 10 time fsck doesn't report any errors.</p> <p>What type of logs would be helpfull.</p> <p>We set are mountoption to error=panic so we can get a crash dump right away.</p> <p> </p> <p>Both the crash dump and /var/log/messages may be helpful. You have mentioned that you did not hit the trouble before you expend your OSS, my understand is that the data corruption only happened on the new OSTs, not on the old ones, is it true?</p> <p>We have 2 filesystem that have been expanded and we are only seeing this on them. I need to double check to make sure we never had the corruption on the old osts. Since the new osts are empty they have higher utilization. </p> <p>I will upload crash dump.</p> <p>It will take sometime to upload the vmcore. But i have attached bt and dmesg.</p> <p>crash dumps uploaded. I have attached 2 more backtrace from 2 more crash dumps. We have had the same ost crash for the past 3 times.</p> <p>Fan Yong (fan.yong@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/28249" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28249</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9410" title="on-disk bitmap corrupted" class="issue-link" data-issue-key="LU-9410"><del>LU-9410</del></a> ldiskfs: read bitmap with lock<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: acbca29e7b28dcad35acdc4d59e3fafc833a572c</p> <p>Mahmoud,</p> <p>I made a kernel patch (<a href="https://review.whamcloud.com/28249" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28249</a>) based on el6.8 (2.6.32-642.15.1.el6) to resolve some in-RAM data corruption. Would you please to try such patch?</p> <p>Thanks!</p> <p>We deactivate one of the OST that kept crashing and we have been stable for 12hours. We will give the patch a try.</p> <p>I thought nasf wrote:<br/> "I just got the latest feedback from our partner minutes ago, they still hit the in-RAM bitmap corruption after applying such EXT4 patch. So we have to make new investigation. Sorry for that."</p> <p>One of our partners has ported the EXT4 patch by themselves, they finally told me that it does not resolve their issue but without showing me their patch. My patch 28249 is NOT the EXT4 patch to be ported, it prevents the bitmap readers accessing the bitmap if without lock the buffer head. It avoids complex logic of the EXT4 patch and may cover more corner cases. So I hope NASA site can try the patch. The side-effect of the patch 28249 is that it may affect the performance a bit, but very very little.</p> <p>Thanks, nasf! We will give it a try.</p> <p>The patch provided did not help with the bitmap errors! Did the crash dump provide any helpful info?</p> <p>What is the storage (hardware vendor) you are using?</p> <p>netapp E5500</p> <p>When we run fsck it sometimes will fix quota stuff but no bitmap.</p> <p>One thing I noticed during recover on a client that was active on the ost got this error</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">[190767.968385] LustreError: 3655:0:(<span class="code-keyword">import</span>.c:1261:ptlrpc_connect_interpret()) nbp2-OST0157_UUID went back in time (transno 17179894486 was previously committed, server now claims 12899642596)! See https:<span class="code-comment">//bugzilla.lustre.org/show_bug.cgi?id=9646</span> </pre> </div></div> <p>Mahmoud,</p> <p>I was told that your system may crash every few hours. I assume that they are similar bitmap corruption as the you described in the ticket summary, right?</p> <blockquote> <p>Apr 3 18:38:16 nbp1-oss6 kernel: LDISKFS-fs error (device dm-42): ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap for group 245659corrupted: 32768 blocks free in bitmap, 0 - in gd<br/> Apr 3 18:38:16 nbp1-oss6 kernel: <br/> Apr 3 18:38:16 nbp1-oss6 kernel: Aborting journal on device dm-3.<br/> Apr 3 18:38:16 nbp1-oss6 kernel: LDISKFS-fs (dm-42): Remounting filesystem read-only<br/> Apr 3 18:38:16 nbp1-oss6 kernel: LDISKFS-fs error (device dm-42): ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap for group 245660corrupted: 32768 blocks free in bitmap, 0 - in gd</p></blockquote> <p>And, when you run <tt>e2fsck</tt> after the corruption, the <tt>e2fsck</tt> will NOT report bitmap issues, instead, sometimes, nothing inconsistency found, sometimes, quota things repaired, right? Have I missed anything else for your current trouble?</p> <p>That is correct. Most of the time an fsck will report something like this</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">nbp2-OST0118: nbp2-OST0118 contains a file system with errors, check forced. nbp2-OST0118: nbp2-OST0118: 776296/74698752 files (20.6% non-contiguous), 2193846359/19122880512 blocks </pre> </div></div> <p> </p> <p> </p> <p>uploading lustre-log.1502322177.17192.txt.gz to ftp site /uploads/<a href="https://jira.whamcloud.com/browse/LU-9410" title="on-disk bitmap corrupted" class="issue-link" data-issue-key="LU-9410"><del>LU-9410</del></a></p> <p>This was taken at the time of OST0157 hitting bitmap error at 16:37:49 PDT</p> <p> </p> <p>please make sure ldiskfs/kernel_patches/series/ldiskfs-2.6-rhel6.8.series really contains rhel6.6/ext4-corrupted-inode-block-bitmaps-handling-patches.patch. I think you might be missing that patch for RHEL6.8 kernel.</p> <p>@shuichi Ihara,</p> <p>We looked at that patch and decided not to apply it because these don't appear to be real on disk corruptions.</p> <p>Yes, that's true. but, still detect and print corruption and you can fsck at maintance window instead of ReadOnly immediately. </p> <p>1. Most of the OSTs that hit this bug have Flex block group size=64 vs others set to Flex block group size=256. The back end raid is set for 1MB stripe size. (8 data disk with 128MB per disk stripe). And we pack all metadata blocks in the front of the LUN. Could this be a factor?</p> <p>2. Does fsck in fact check for bitmap corruption on disk? if we don't see it fixing anything does that confirm that these are in memory corruptions?</p> <p>3. If these are in memory corruption can we get a debug patch that will re-read from disk before marking the bitmap as bad.</p> <p>4. Can you provide any other debug patch to help narrow the root cause?</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=mhanafi" class="user-hover" rel="mhanafi">mhanafi</a>, would you please to show me the output:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>dumpe2fs -f $OST_device </pre> </div></div> <p>Thanks!</p> <p>We had a ost go read only today and I was able to gather some very useful info.</p> <p>I dump /proc/fs/ldiskfs/dm-13/mb_groups It had I/O error for block groups that the ost was complaining about.</p> <p>I dump (dumpe2fs) the block group info for the device and it did in fact show blockgroups with 0 free blocks but having free blocks. like this one that triggered the OST to readonly.</p> <p> Group 314421: (Blocks 10302947328-10302980095) <span class="error">&#91;INODE_UNINIT, BLOCK_UNINIT, ITABLE_ZEROED&#93;</span><br/>   Checksum 0x8d80, unused inodes 128<br/>   Block bitmap at 369141 (bg #11 + 8693), Inode bitmap at 970965 (bg #29 + 20693)<br/>   Inode table at 3773736-3773743 (bg #115 + 5416)<br/>   0 free blocks, 128 free inodes, 0 directories, 128 unused inodes<br/>   Free blocks: 10302947328-10302980095</p> <p> I recheck the blockgroups, using dumpe2fs after the fsck and it had fixed those blocks.<br/> Group 314421: (Blocks 10302947328-10302980095) <span class="error">&#91;INODE_UNINIT, BLOCK_UNINIT, ITABLE_ZEROED&#93;</span><br/>   Checksum 0x5b37, unused inodes 128<br/>   Block bitmap at 369141 (bg #11 + 8693), Inode bitmap at 970965 (bg #29 + 20693)<br/>   Inode table at 3773736-3773743 (bg #115 + 5416)<br/>   32768 free blocks, 128 free inodes, 0 directories, 128 unused inodes<br/>   Free blocks: 10302947328-10302980095</p> <p>I will attach the full dumpe2fs output to the case</p> <p>Fan Yong (fan.yong@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/28489" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28489</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9410" title="on-disk bitmap corrupted" class="issue-link" data-issue-key="LU-9410"><del>LU-9410</del></a> ldiskfs: enable mb debug<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: d4e2f024d91f375085c24906313b7bc522464c20</p> <blockquote> <p>1. Most of the OSTs that hit this bug have Flex block group size=64 vs others set to Flex block group size=256. The back end raid is set for 1MB stripe size. (8 data disk with 128MB per disk stripe). And we pack all metadata blocks in the front of the LUN. Could this be a factor?</p></blockquote> <p>I am not sure for this.</p> <blockquote> <p>2. Does fsck in fact check for bitmap corruption on disk? if we don't see it fixing anything does that confirm that these are in memory corruptions?</p></blockquote> <p>The <tt>e2fsck</tt> will verify the free blocks/inodes in the bitmap with the value recorded in the group descriptor. </p> <blockquote> <p>3. If these are in memory corruption can we get a debug patch that will re-read from disk before marking the bitmap as bad.</p> <p>4. Can you provide any other debug patch to help narrow the root cause?</p></blockquote> <p>I made a debug patch (<a href="https://review.whamcloud.com/28489" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28489</a>) with mb debug enable, please apply it with the former patch (<a href="https://review.whamcloud.com/#/c/28249/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/28249/</a>) together. Please NOTE: the mb debug switch is under /sys/kernel/debug/ldiskfs/mballoc-debug on the server node. It is disabled by default with the value 0. Please set it as '1' before you mount up the Lustre device.</p> <p>Thanks!</p> <p>we appear to be missing <a href="https://review.whamcloud.com/#/c/16312/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/16312/</a> (<a href="https://review.whamcloud.com/#/c/16679/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/16679/</a>) from 6.8 cent version. Does this cause any issue with debug patches or recommended action?</p> <p>We are planning to run with 16312 and suggested debug patches</p> <p>for some reason this patch never made in into the series/6.8 <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/sad.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Are we convinced this is a duplicate of <a href="https://jira.whamcloud.com/browse/LU-7114" title="ldiskfs: corrupted bitmaps handling patches" class="issue-link" data-issue-key="LU-7114"><del>LU-7114</del></a></p> <p>After the kernel and lustre rebuild/install we don't see /sys/kernel/debug/ldiskfs option.</p> <p> nbp2-oss1 /boot # cat config-2.6.32-642.15.1.el6.20170609.x86_64.lustre273 |grep CONFIG_EXT4_DEBUG<br/> CONFIG_EXT4_DEBUG=y<br/> nbp2-oss1 /boot # ls -l /sys/kernel/debug/<br/> total 0</p> <p>Kernel was rebuilt with CONFIG_EXT4_DEBUG on.</p> <blockquote> <p>After the kernel and lustre rebuild/install we don't see /sys/kernel/debug/ldiskfs option.</p></blockquote> <p>What is the output:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>find /proc /sys -name mballoc-debug </pre> </div></div> <blockquote> <p>Kernel was rebuilt with CONFIG_EXT4_DEBUG on.</p></blockquote> <p>Sorry, in my test, the CONFIG_EXT4_DEBUG is disabled by default, so I thought NASA may disable it by default also. Then to avoid rebuilding kernel, I made patch to remove the conditional compile for mb debug. On the other hand, I added more debug for initializing BG block bitmap case.</p> <p>Do we should get the debugging without the kernel recompile?</p> <p> </p> <p>The kernel was originally built with CONFIG_EXT4_DEBUG disabled and lustre server rpms were built for that.<br/> Mahmoud told me that he did not see /sys/kernel/debug/ldiskfs option, then I rebuilt the kernel and the lustre.</p> <p>So, we have both available for testing. If CONFIG_EXT4_DEBUG not needed in the kernel, how we can enable the mb-debug?</p> <blockquote> <p>Do we should get the debugging without the kernel recompile?</p></blockquote> <p>With my patch applied, you need NOT recompile the kernel. In fact, the CONFIG_EXT4_DEBUG is almost redundant since we can control the mb debug via the debug level.<br/> Considering the logs when corruption happened "32768 blocks free in bitmap", it seems that the BG was initialized by logic instead of loading from disk. On some degree, it can explain the in-ram corruption. So I added more debug information in my patch. I think it is worth to try.</p> <blockquote> <p>we appear to be missing <a href="https://review.whamcloud.com/#/c/16312/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/16312/</a> (<a href="https://review.whamcloud.com/#/c/16679/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/16679/</a>) from 6.8 cent version. Does this cause any issue with debug patches or recommended action?</p> <p>We are planning to run with 16312 and suggested debug patches</p> <p>for some reason this patch never made in into the series/6.8 </p> <p>Are we convinced this is a duplicate of <a href="https://jira.whamcloud.com/browse/LU-7114" title="ldiskfs: corrupted bitmaps handling patches" class="issue-link" data-issue-key="LU-7114"><del>LU-7114</del></a></p></blockquote> <p>It is true that we missed such patch, Andreas has pointed it out in the first comment:<br/> <a href="https://jira.hpdd.intel.com/browse/LU-9410?focusedCommentId=193803&amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-193803" class="external-link" target="_blank" rel="nofollow noopener">https://jira.hpdd.intel.com/browse/LU-9410?focusedCommentId=193803&amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-193803</a></p> <p>But this patch is mostly used for handling the case after the bitmap corruption happened. It allows the system to go ahead without failure right away, then the users can run <tt>e2fsck</tt> at the maintain windows. As <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=mhanafi" class="user-hover" rel="mhanafi">mhanafi</a> commented:<br/> <a href="https://jira.hpdd.intel.com/browse/LU-9410?focusedCommentId=205024&amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-205024" class="external-link" target="_blank" rel="nofollow noopener">https://jira.hpdd.intel.com/browse/LU-9410?focusedCommentId=205024&amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-205024</a>, it may not help too much for NASA case.</p> <p>OK, running kernel without CONFIG_EXT4_DEBUG and lustre with your patches, how do we enable debugging if we do not see /sys/kernel/debug/ldiskfs/ ? Please elaborate. Thanks.</p> <p>What is the output with ldiskfs.ko insmod:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>find /proc /sys -name mballoc-debug </pre> </div></div> <p>find /proc /sys -name mballoc-debug </p> <p>has not output</p> <p>Please attach the source file ldiskfs/mballoc.c, you can find it in your compile directory. Thanks!</p> <p>mballoc.c attached.</p> <p><a href="https://review.whamcloud.com/28489" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28489</a> is refreshed, please try again. Thanks!</p> <p>So haven't put patch debug 28489 in place but are now running with "<del><a href="https://jira.whamcloud.com/browse/LU-7114" title="ldiskfs: corrupted bitmaps handling patches" class="issue-link" data-issue-key="LU-7114"><del>LU-7114</del></a></del>" patch. It already has found bitmap errors.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">ug 12 01:05:43 nbp2-oss20 kernel: LDISKFS-fs warning (device dm-21): ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap <span class="code-keyword">for</span> group 275790 corrupted: 32768 blocks free in bitmap, 0 - in gd Aug 12 01:05:43 nbp2-oss20 kernel: Aug 12 01:05:43 nbp2-oss20 kernel: LDISKFS-fs warning (device dm-21): ldiskfs_mb_load_buddy: Error in loading buddy information <span class="code-keyword">for</span> 275790 Aug 12 01:05:43 nbp2-oss20 kernel: LDISKFS-fs warning (device dm-21): ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap <span class="code-keyword">for</span> group 275790 corrupted: 32768 blocks free in bitmap, 0 - in gd Aug 12 01:05:43 nbp2-oss20 kernel: Aug 12 01:05:43 nbp2-oss20 kernel: LDISKFS-fs warning (device dm-21): ldiskfs_mb_load_buddy: Error in loading buddy information <span class="code-keyword">for</span> 275790 Aug 12 01:05:44 nbp2-oss20 kernel: LDISKFS-fs warning (device dm-21): ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap <span class="code-keyword">for</span> group 275790 corrupted: 32768 blocks free in bitmap, 0 - in gd Aug 12 01:05:45 nbp2-oss20 kernel: Aug 12 01:05:45 nbp2-oss20 kernel: LDISKFS-fs warning (device dm-21): ldiskfs_mb_load_buddy: Error in loading buddy information <span class="code-keyword">for</span> 275790 Aug 12 01:05:45 nbp2-oss20 kernel: LDISKFS-fs warning (device dm-21): ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap <span class="code-keyword">for</span> group 275790 corrupted: 32768 blocks free in bitmap, 0 - in gd Aug 12 01:05:45 nbp2-oss20 kernel: Aug 12 01:05:45 nbp2-oss20 kernel: LDISKFS-fs warning (device dm-21): ldiskfs_mb_load_buddy: Error in loading buddy information <span class="code-keyword">for</span> 275790 Aug 12 01:05:46 nbp2-oss20 kernel: LDISKFS-fs warning (device dm-21): ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap <span class="code-keyword">for</span> group 275790 corrupted: 32768 blocks free in bitmap, 0 - in gd Aug 12 01:05:47 nbp2-oss20 kernel: Aug 12 01:05:47 nbp2-oss20 kernel: LDISKFS-fs warning (device dm-21): ldiskfs_mb_load_buddy: Error in loading buddy information <span class="code-keyword">for</span> 275790 Aug 12 01:05:47 nbp2-oss20 kernel: LDISKFS-fs warning (device dm-21): ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap <span class="code-keyword">for</span> group 275790 corrupted: 32768 blocks free in bitmap, 0 - in gd Aug 12 01:05:47 nbp2-oss20 kernel: Aug 12 01:05:47 nbp2-oss20 kernel: LDISKFS-fs warning (device dm-21): ldiskfs_mb_load_buddy: Error in loading buddy information <span class="code-keyword">for</span> 275790 Aug 12 01:05:49 nbp2-oss20 kernel: LDISKFS-fs warning (device dm-21): ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap <span class="code-keyword">for</span> group 275790 corrupted: 32768 blocks free in bitmap, 0 - in gd Aug 12 01:05:50 nbp2-oss20 kernel: Aug 12 01:05:50 nbp2-oss20 kernel: LDISKFS-fs warning (device dm-21): ldiskfs_mb_load_buddy: Error in loading buddy information <span class="code-keyword">for</span> 275790 Aug 12 01:05:50 nbp2-oss20 kernel: LDISKFS-fs warning (device dm-21): ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap <span class="code-keyword">for</span> group 275790 corrupted: 32768 blocks free in bitmap, 0 - in gd Aug 12 01:05:50 nbp2-oss20 kernel: Aug 12 01:05:50 nbp2-oss20 kernel: LDISKFS-fs warning (device dm-21): ldiskfs_mb_load_buddy: Error in loading buddy information <span class="code-keyword">for</span> 275790 Aug 12 01:05:53 nbp2-oss20 kernel: LDISKFS-fs warning (device dm-21): ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap <span class="code-keyword">for</span> group 275790 corrupted: 32768 blocks free in bitmap, 0 - in gd Aug 12 01:05:54 nbp2-oss20 kernel: Aug 12 01:05:54 nbp2-oss20 kernel: LDISKFS-fs warning (device dm-21): ldiskfs_mb_load_buddy: Error in loading buddy information <span class="code-keyword">for</span> 275790 Aug 12 01:05:54 nbp2-oss20 kernel: LDISKFS-fs warning (device dm-21): ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap <span class="code-keyword">for</span> group 275790 corrupted: 32768 blocks free in bitmap, 0 - in gd Aug 12 01:05:54 nbp2-oss20 kernel: Aug 12 01:05:54 nbp2-oss20 kernel: LDISKFS-fs warning (device dm-21): ldiskfs_mb_load_buddy: Error in loading buddy information <span class="code-keyword">for</span> 275790 Aug 12 01:05:59 nbp2-oss20 kernel: LDISKFS-fs warning (device dm-21): ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap <span class="code-keyword">for</span> group 275790 corrupted: 32768 blocks free in bitmap, 0 - in gd Aug 12 01:05:59 nbp2-oss20 kernel: Aug 12 01:05:59 nbp2-oss20 kernel: LDISKFS-fs warning (device dm-21): ldiskfs_mb_load_buddy: Error in loading buddy information <span class="code-keyword">for</span> 275790 Aug 12 01:05:59 nbp2-oss20 kernel: LDISKFS-fs warning (device dm-21): ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap <span class="code-keyword">for</span> group 275790 corrupted: 32768 blocks free in bitmap, 0 - in gd Aug 12 01:05:59 nbp2-oss20 kernel: Aug 12 01:05:59 nbp2-oss20 kernel: LDISKFS-fs warning (device dm-21): ldiskfs_mb_load_buddy: Error in loading buddy information <span class="code-keyword">for</span> 275790 Aug 12 01:06:05 nbp2-oss20 kernel: LDISKFS-fs warning (device dm-21): ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap <span class="code-keyword">for</span> group 275790 corrupted: 32768 blocks free in bitmap, 0 - in gd Aug 12 01:06:05 nbp2-oss20 kernel: Aug 12 01:06:05 nbp2-oss20 kernel: LDISKFS-fs warning (device dm-21): ldiskfs_mb_load_buddy: Error in loading buddy information <span class="code-keyword">for</span> 275790 Aug 12 01:06:05 nbp2-oss20 kernel: LDISKFS-fs warning (device dm-21): ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap <span class="code-keyword">for</span> group 275790 corrupted: 32768 blocks free in bitmap, 0 - in gd Aug 12 01:06:05 nbp2-oss20 kernel: Aug 12 01:06:05 nbp2-oss20 kernel: LDISKFS-fs warning (device dm-21): ldiskfs_mb_load_buddy: Error in loading buddy information <span class="code-keyword">for</span> 275790 Aug 12 01:06:12 nbp2-oss20 kernel: LDISKFS-fs warning (device dm-21): ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap <span class="code-keyword">for</span> group 275790 corrupted: 32768 blocks free in bitmap, 0 - in gd </pre> </div></div> <p>Some time later</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Aug 12 04:05:12 nbp2-oss20 kernel: LDISKFS-fs warning (device dm-21): ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap <span class="code-keyword">for</span> group 276684 corrupted: 32768 blocks free in bitmap, 0 - in gd Aug 12 04:05:12 nbp2-oss20 kernel: Aug 12 04:05:12 nbp2-oss20 kernel: LDISKFS-fs warning (device dm-21): ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap <span class="code-keyword">for</span> group 276685 corrupted: 32768 blocks free in bitmap, 0 - in gd Aug 12 04:07:56 nbp2-oss20 pcp-pmie[5801]: High 1-minute load average 354load@nbp2-oss20 Aug 12 04:07:56 nbp2-oss20 - in gd Aug 12 04:07:56 nbp2-oss20 kernel: Aug 12 04:07:56 nbp2-oss20 kernel: LDISKFS-fs warning (device dm-21): ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap <span class="code-keyword">for</span> group 304861 corrupted: 32768 blocks free in bitmap, 0 - in gd Aug 12 04:07:56 nbp2-oss20 kernel: Aug 12 04:07:56 nbp2-oss20 kernel: LDISKFS-fs warning (device dm-21): ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap <span class="code-keyword">for</span> group 304862 corrupted: 32768 blocks free in bitmap, 0 - in gd Aug 12 04:07:56 nbp2-oss20 kernel: Aug 12 04:07:56 nbp2-oss20 kernel: LDISKFS-fs warning (device dm-21): ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap <span class="code-keyword">for</span> group 304863 corrupted: 32768 blocks free in bitmap, 0 - in gd Aug 12 04:07:56 nbp2-oss20 kernel: Aug 12 04:07:56 nbp2-oss20 kernel: LDISKFS-fs warning (device dm-21): ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap <span class="code-keyword">for</span> group 304864 corrupted: 32768 blocks free in bitmap, 0 - in gd Aug 12 04:07:56 nbp2-oss20 kernel: ..... </pre> </div></div> <p><font color="#ff0000"><b>It has marked 6727 uniq groups as bad for dm-21(ost319)</b></font></p> <p> </p> <p><a href="https://jira.whamcloud.com/browse/LU-7114" title="ldiskfs: corrupted bitmaps handling patches" class="issue-link" data-issue-key="LU-7114"><del>LU-7114</del></a> will allow the system to go ahead without failure right away when found corrupted bitmap, but the corruption is still there. I would suggest to apply the patch <a href="https://review.whamcloud.com/#/c/28489/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/28489/</a>, it will give us more information the mb operations trace.</p> <p>With the new build are we suppose to have mballoc-debug in /proc or /sys?</p> <p>because the find doesn't find anything.</p> <p> </p> <p>Never mind I figured this out. We need to mount debugfs for it to show up.</p> <p>Got block group debug logs with corruption. Block group is #270808. I will attach full log file to the case. syslog.gp270808.error.gz</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Aug 14 18:37:14 nbp2-oss20 kernel: (/tmp/rpmbuild-lustre-jlan-PYDDD1xV/BUILD/lustre-2.7.3/ldiskfs/mballoc.c, 1103): ldiskfs_mb_load_buddy: load group 270808 Aug 14 18:37:14 nbp2-oss20 kernel: (/tmp/rpmbuild-lustre-jlan-PYDDD1xV/BUILD/lustre-2.7.3/ldiskfs/mballoc.c, 1032): ldiskfs_mb_init_group: init group 270808 Aug 14 18:37:14 nbp2-oss20 kernel: (/tmp/rpmbuild-lustre-jlan-PYDDD1xV/BUILD/lustre-2.7.3/ldiskfs/balloc.c, 179): ldiskfs_init_block_bitmap: #24877: init the group 270808 of total groups 583584: group_blocks 32768, free_blocks 32768, free_blocks_in_gdp 0, ret 32768 Aug 14 18:37:14 nbp2-oss20 kernel: (/tmp/rpmbuild-lustre-jlan-PYDDD1xV/BUILD/lustre-2.7.3/ldiskfs/mballoc.c, 927): ldiskfs_mb_init_cache: put bitmap <span class="code-keyword">for</span> group 270808 in page 541616/0 Aug 14 18:37:14 nbp2-oss20 kernel: on-disk bitmap <span class="code-keyword">for</span> group 270808 corrupted: 32768 blocks free in bitmap, 0 - in gd Aug 14 18:37:14 nbp2-oss20 kernel: LDISKFS-fs warning (device dm-21): ldiskfs_mb_load_buddy: Error in loading buddy information <span class="code-keyword">for</span> 270808 Aug 14 18:37:14 nbp2-oss20 kernel: (/tmp/rpmbuild-lustre-jlan-PYDDD1xV/BUILD/lustre-2.7.3/ldiskfs/mballoc.c, 1103): ldiskfs_mb_load_buddy: load group 270808 Aug 14 18:37:14 nbp2-oss20 kernel: (/tmp/rpmbuild-lustre-jlan-PYDDD1xV/BUILD/lustre-2.7.3/ldiskfs/mballoc.c, 1032): ldiskfs_mb_init_group: init group 270808 Aug 14 18:37:14 nbp2-oss20 kernel: (/tmp/rpmbuild-lustre-jlan-PYDDD1xV/BUILD/lustre-2.7.3/ldiskfs/mballoc.c, 927): ldiskfs_mb_init_cache: put bitmap <span class="code-keyword">for</span> group 270808 in page 541616/0 Aug 14 18:37:14 nbp2-oss20 kernel: LDISKFS-fs warning (device dm-21): ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap <span class="code-keyword">for</span> group 270808 corrupted: 32768 blocks free in bitmap, 0 - in gd Aug 14 18:37:14 nbp2-oss20 kernel: LDISKFS-fs warning (device dm-21): ldiskfs_mb_load_buddy: Error in loading buddy information <span class="code-keyword">for</span> 270808 Aug 14 18:37:15 nbp2-oss20 kernel: (/tmp/rpmbuild-lustre-jlan-PYDDD1xV/BUILD/lustre-2.7.3/ldiskfs/mballoc.c, 1103): ldiskfs_mb_load_buddy: load group 270808 Aug 14 18:37:15 nbp2-oss20 kernel: (/tmp/rpmbuild-lustre-jlan-PYDDD1xV/BUILD/lustre-2.7.3/ldiskfs/mballoc.c, 1032): ldiskfs_mb_init_group: init group 270808 Aug 14 18:37:15 nbp2-oss20 kernel: (/tmp/rpmbuild-lustre-jlan-PYDDD1xV/BUILD/lustre-2.7.3/ldiskfs/mballoc.c, 927): ldiskfs_mb_init_cache: put bitmap <span class="code-keyword">for</span> group 270808 in page 541616/0 Aug 14 18:37:15 nbp2-oss20 kernel: on-disk bitmap <span class="code-keyword">for</span> group 270808 corrupted: 32768 blocks free in bitmap, 0 - in gd Aug 14 18:37:15 nbp2-oss20 kernel: Error in loading buddy information <span class="code-keyword">for</span> 270808 Aug 14 18:37:15 nbp2-oss20 kernel: (/tmp/rpmbuild-lustre-jlan-PYDDD1xV/BUILD/lustre-2.7.3/ldiskfs/mballoc.c, 1103): ldiskfs_mb_load_buddy: load group 270808 Aug 14 18:37:15 nbp2-oss20 kernel: (/tmp/rpmbuild-lustre-jlan-PYDDD1xV/BUILD/lustre-2.7.3/ldiskfs/mballoc.c, 1032): ldiskfs_mb_init_group: init group 270808 Aug 14 18:37:15 nbp2-oss20 kernel: (/tmp/rpmbuild-lustre-jlan-PYDDD1xV/BUILD/lustre-2.7.3/ldiskfs/mballoc.c, 927): ldiskfs_mb_init_cache: put bitmap <span class="code-keyword">for</span> group 270808 in page 541616/0 Aug 14 18:37:15 nbp2-oss20 kernel: on-disk bitmap <span class="code-keyword">for</span> group 270808 corrupted: 32768 blocks free in bitmap, 0 - in gd Aug 14 18:37:15 nbp2-oss20 kernel: (/tmp/rpmbuild-lustre-jlan-PYDDD1xV/BUILD/lustre-2.7.3/ldiskfs/mballoc.c, 1103): ldiskfs_mb_load_buddy: Error in loading buddy information <span class="code-keyword">for</span> 270808 Aug 14 18:37:17 nbp2-oss20 kernel: (/tmp/rpmbuild-lustre-jlan-PYDDD1xV/BUILD/lustre-2.7.3/ldiskfs/mballoc.c, 1103): ldiskfs_mb_load_buddy: load group 270808 Aug 14 18:37:17 nbp2-oss20 kernel: (/tmp/rpmbuild-lustre-jlan-PYDDD1xV/BUILD/lustre-2.7.3/ldiskfs/mballoc.c, 1032): ldiskfs_mb_init_group: init group 270808 Aug 14 18:37:17 nbp2-oss20 kernel: (/tmp/rpmbuild-lustre-jlan-PYDDD1xV/BUILD/lustre-2.7.3/ldiskfs/mballoc.c, 927): ldiskfs_mb_init_cache: put bitmap <span class="code-keyword">for</span> group 270808 in page 541616/0 </pre> </div></div> <p> </p> <p> </p> <p>Fan Yong (fan.yong@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/28550" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28550</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9410" title="on-disk bitmap corrupted" class="issue-link" data-issue-key="LU-9410"><del>LU-9410</del></a> ldiskfs: handle unmatched bitmap<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 0a4199ad21c5ac23a4a4e7e07847610ad8ec7994</p> <blockquote> <p>Aug 14 18:37:14 nbp2-oss20 kernel: (/tmp/rpmbuild-lustre-jlan-PYDDD1xV/BUILD/lustre-2.7.3/ldiskfs/balloc.c, 179): ldiskfs_init_block_bitmap: #24877: init the group 270808 of total groups 583584: group_blocks 32768, free_blocks 32768, free_blocks_in_gdp 0, ret 32768</p></blockquote> <p>The logs shows that the ldiskfs_init_block_bitmap() initialized the bitmap, but the free blocks count in the group descriptor is still zero, that caused the subsequent ldiskfs_mb_check_ondisk_bitmap() failure. Currently, I can not say it is corruption, but more like logic issue. The patch will set the free block count based on the real free bits in the bitmap. It may be not the perfect solution, but we can try whether it can resolve your trouble or not.</p> <p>I used systemtap to catch one of these bad groups and dump out the ldiskfs_group_desc struct.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">mballoc.c:826: first_group: 274007 bg_free_blocks_count_hi: 0 bg_block_bitmap_hi: 0 bg_free_blocks_count_lo: 0 mballoc.c:826:$desc {.bg_block_bitmap_lo=328727, .bg_inode_bitmap_lo=930551, .bg_inode_table_lo=3450424, .bg_free_blocks_count_lo=0, .bg_free_inodes_count_lo=128, .bg_used_dirs_count_lo=0, .bg_flags=7, .bg_reserved=[...], .bg_itable_unused_lo=128, .bg_checksum=55256, .bg_block_bitmap_hi=0, .bg_inode_bitmap_hi=0, .bg_inode_table_hi=0, .bg_free_blocks_count_hi=0, .bg_free_inodes_count_hi=0, .bg_used_dirs_count_hi=0, .bg_itable_unused_hi=0, .bg_reserved2=[...]} </pre> </div></div> <p> </p> <p>It also seem odd that dumpe2fs can produce different results for unused block groups. Sometimes it will show block_bitmap!=free_blocks and other time it will be ok.</p> <p> ---</p> <p>in ldiskfs_valid_block_bitmap() I don't understand this</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">if</span> (LDISKFS_HAS_INCOMPAT_FEATURE(sb, LDISKFS_FEATURE_INCOMPAT_FLEX_BG)) { /* with FLEX_BG, the inode/block bitmaps and itable * blocks may not be in the group at all * so the bitmap validation will be skipped <span class="code-keyword">for</span> those groups * or it has to also read the block group where the bitmaps * are located to verify they are set. */ <span class="code-keyword">return</span> 1; } </pre> </div></div> <p>We have flex_bg enabled would this apply to us?</p> <p> </p> <p>For the OST that are prone to the bitmap errors cat /proc/fs/ldiskfs/dm*/mb_groups will reproduce the errors.</p> <p> </p> <p>Applied the new patch. After a full fsck mounting osts resulted in this many block groups getting corrected.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">---------------- service603 ---------------- 4549 dm-33): ---------------- service604 ---------------- 4425 dm-32): ---------------- service606 ---------------- 4658 dm-29): ---------------- service610 ---------------- 4631 dm-33): ---------------- service611 ---------------- 4616 dm-28): ---------------- service616 ---------------- 4652 dm-35): ---------------- service617 ---------------- 4501 dm-21): ---------------- service619 ---------------- 4657 dm-25): </pre> </div></div> <p>We need to rate limit the warnings. </p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=mhanafi" class="user-hover" rel="mhanafi">mhanafi</a><br/> It looks different from the original one, would you please to show me more logs (dmesg, /var/log/messages) about the latest corruption ? Is the system still accessible after above warning?</p> <p>here is part of dmesg. The high rate of messages caused the root drive scsi device to reset. But all but one server recovered. I had to turn down printk log level down to get the last one to recover.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">LDISKFS-fs warning (device dm-33): ldiskfs_init_block_bitmap: Set free blocks as 32768 <span class="code-keyword">for</span> group 262310 LDISKFS-fs warning (device dm-33): ldiskfs_init_block_bitmap: Set free blocks as 32768 <span class="code-keyword">for</span> group 262311 LDISKFS-fs warning (device dm-33): ldiskfs_init_block_bitmap: Set free blocks as 32768 <span class="code-keyword">for</span> group 262312 LDISKFS-fs warning (device dm-33): ldiskfs_init_block_bitmap: Set free blocks as 32768 <span class="code-keyword">for</span> group 262313 LDISKFS-fs warning (device dm-33): ldiskfs_init_block_bitmap: Set free blocks as 32768 <span class="code-keyword">for</span> group 262314 LNet: 12178:0:(lib-move.c:1487:lnet_parse_put()) Dropping PUT from 12345-10.149.2.156@o2ib313 portal 28 match 1575300167923792 offset 0 length 520: 4 LNet: 12178:0:(lib-move.c:1487:lnet_parse_put()) Skipped 978380 previous similar messages sd 0:0:1:0: attempting task abort! scmd(ffff880af433e0c0) sd 0:0:1:0: [sdb] CDB: Write(10): 2a 00 00 a0 08 08 00 00 08 00 scsi target0:0:1: handle(0x000a), sas_address(0x4433221102000000), phy(2) scsi target0:0:1: enclosure_logical_id(0x50030480198f7e01), slot(2) scsi target0:0:1: enclosure level(0x0000),connector name( ^C) sd 0:0:1:0: task abort: SUCCESS scmd(ffff880af433e0c0) sd 0:0:1:0: attempting task abort! scmd(ffff880a64ab46c0) sd 0:0:1:0: [sdb] CDB: Write(10): 2a 00 00 e0 08 08 00 00 08 00 scsi target0:0:1: handle(0x000a), sas_address(0x4433221102000000), phy(2) scsi target0:0:1: enclosure_logical_id(0x50030480198f7e01), slot(2) scsi target0:0:1: enclosure level(0x0000),connector name( ^C) sd 0:0:1:0: task abort: SUCCESS scmd(ffff880a64ab46c0) sd 0:0:1:0: attempting task abort! scmd(ffff880b21cec180) sd 0:0:1:0: [sdb] CDB: Write(10): 2a 00 00 c0 08 08 00 00 08 00 scsi target0:0:1: handle(0x000a), sas_address(0x4433221102000000), phy(2) DISKFS-fs (dm-23): mounted filesystem with ordered data mode. quota=on. Opts: LDISKFS-fs (dm-34): mounted filesystem with ordered data mode. quota=on. Opts: mounted filesystem with ordered data mode. quota=on. Opts: LDISKFS-fs (dm-29): mounted filesystem with ordered data mode. quota=on. Opts: LDISKFS-fs (dm-18): mounted filesystem with ordered data mode. quota=on. Opts: Lustre: nbp2-OST0081: Not available <span class="code-keyword">for</span> connect from 10.151.43.107@o2ib (not set up) Lustre: Skipped 3 previous similar messages Lustre: nbp2-OST0081: Not available <span class="code-keyword">for</span> connect from 10.151.29.130@o2ib (not set up) Lustre: Skipped 113 previous similar messages Lustre: nbp2-OST0081: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-450 Lustre: nbp2-OST0081: Will be in recovery <span class="code-keyword">for</span> at least 2:30, or until 14441 clients reconnect Lustre: nbp2-OST0081: Denying connection <span class="code-keyword">for</span> <span class="code-keyword">new</span> client 35b99837-9505-fc4d-270f-f2d1ca30372d (at 10.151.30.176@o2ib), waiting <span class="code-keyword">for</span> all 14441 known clients (44 recovered, 1 in progress, and 0 evicted) to recover in 5:10 </pre> </div></div> <p>Here is /var/log/messages</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Aug 11 17:58:25 nbp2-oss10 kernel: LNet: 12075:0:(lib-move.c:1487:lnet_parse_put()) Dropping PUT from 12345-10.151.30.120@o2ib portal 28 match 1575477031778096 offset 0 length 520: 4 Aug 11 17:58:25 nbp2-oss10 kernel: LNet: 12075:0:(lib-move.c:1487:lnet_parse_put()) Skipped 1037319 previous similar messages Aug 11 18:03:35 nbp2-oss10 kernel: LDISKFS-fs (dm-30): Aug 11 18:03:35 nbp2-oss10 kernel: LDISKFS-fs (dm-28): mounted filesystem with ordered data mode. quota=on. Opts: Aug 11 18:03:35 nbp2-oss10 kernel: LDISKFS-fs (dm-31): mounted filesystem with ordered data mode. quota=on. Opts: Aug 11 18:03:35 nbp2-oss10 kernel: LDISKFS-fs (dm-18): mounted filesystem with ordered data mode. quota=on. Opts: Aug 11 18:03:35 nbp2-oss10 kernel: LDISKFS-fs (dm-21): mounted filesystem with ordered data mode. quota=on. Opts: Aug 11 18:03:35 nbp2-oss10 kernel: LDISKFS-fs (dm-19): mounted filesystem with ordered data mode. quota=on. Opts: Aug 11 18:03:35 nbp2-oss10 kernel: LDISKFS-fs (dm-22): mounted filesystem with ordered data mode. quota=on. Opts: Aug 11 18:03:35 nbp2-oss10 kernel: LDISKFS-fs (dm-20): mounted filesystem with ordered data mode. quota=on. Opts: Aug 11 18:03:35 nbp2-oss10 kernel: LDISKFS-fs (dm-26): mounted filesystem with ordered data mode. quota=on. Opts: Aug 11 18:03:35 nbp2-oss10 kernel: LDISKFS-fs (dm-33): mounted filesystem with ordered data mode. quota=on. Opts: Aug 11 18:03:35 nbp2-oss10 kernel: mounted filesystem with ordered data mode. quota=on. Opts: Aug 11 18:03:35 nbp2-oss10 kernel: LDISKFS-fs (dm-23): mounted filesystem with ordered data mode. quota=on. Opts: Aug 11 18:03:35 nbp2-oss10 kernel: LDISKFS-fs (dm-32): mounted filesystem with ordered data mode. quota=on. Opts: Aug 11 18:03:40 nbp2-oss10 kernel: LDISKFS-fs (dm-34): mounted filesystem with ordered data mode. quota=on. Opts: Aug 11 18:03:40 nbp2-oss10 kernel: LDISKFS-fs (dm-24): mounted filesystem with ordered data mode. quota=on. Opts: Aug 11 18:03:40 nbp2-oss10 kernel: LDISKFS-fs (dm-25): mounted filesystem with ordered data mode. quota=on. Opts: Aug 11 18:03:40 nbp2-oss10 kernel: Aug 11 18:03:41 nbp2-oss10 kernel: LDISKFS-fs (dm-29): Aug 11 18:03:41 nbp2-oss10 kernel: LDISKFS-fs (dm-35): mounted filesystem with ordered data mode. quota=on. Opts: Aug 11 18:03:41 nbp2-oss10 kernel: mounted filesystem with ordered data mode. quota=on. Opts: Aug 11 18:03:49 nbp2-oss10 kernel: LDISKFS-fs (dm-27): mounted filesystem with ordered data mode. quota=on. Opts: Aug 11 18:03:50 nbp2-oss10 kernel: LustreError: 137-5: nbp2-OST0009_UUID: not available <span class="code-keyword">for</span> connect from 10.151.50.143@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server. Aug 11 18:03:50 nbp2-oss10 kernel: LustreError: Skipped 314 previous similar messages Aug 11 18:03:51 nbp2-oss10 kernel: Lustre: nbp2-OST00d1: Not available <span class="code-keyword">for</span> connect from 10.151.9.177@o2ib (not set up) Aug 11 18:03:51 nbp2-oss10 kernel: Lustre: Skipped 11 previous similar messages Aug 11 18:03:51 nbp2-oss10 kernel: LustreError: 137-5: nbp2-OST0009_UUID: not available <span class="code-keyword">for</span> connect from 10.151.8.85@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server. Aug 11 18:03:51 nbp2-oss10 kernel: LustreError: Skipped 3632 previous similar messages Aug 11 18:03:51 nbp2-oss10 kernel: Lustre: nbp2-OST00d1: Not available <span class="code-keyword">for</span> connect from 10.151.50.241@o2ib (not set up) Aug 11 18:03:51 nbp2-oss10 kernel: Lustre: Skipped 180 previous similar messages Aug 11 18:03:52 nbp2-oss10 kernel: LustreError: 137-5: nbp2-OST0135_UUID: not available <span class="code-keyword">for</span> connect from 10.151.48.113@o2ib (no target). If you are running an HA pair check that the target is mounted on the other server. Aug 11 18:03:52 nbp2-oss10 kernel: LustreError: Skipped 6273 previous similar messages Aug 11 18:03:52 nbp2-oss10 kernel: Lustre: nbp2-OST00d1: Not available <span class="code-keyword">for</span> connect from 10.151.7.158@o2ib (not set up) Aug 11 18:03:52 nbp2-oss10 kernel: Lustre: Skipped 402 previous similar messages Aug 11 18:03:52 nbp2-oss10 kernel: Lustre: nbp2-OST00d1: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-450 Aug 11 18:03:52 nbp2-oss10 kernel: Lustre: nbp2-OST00d1: Will be in recovery <span class="code-keyword">for</span> at least 2:30, or until 14452 clients reconnect </pre> </div></div> <p>Fan Yong (fan.yong@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/28566" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28566</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9410" title="on-disk bitmap corrupted" class="issue-link" data-issue-key="LU-9410"><del>LU-9410</del></a> ldiskfs: no check mb bitmap if flex_bg enabled<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 8332a30959750c603bc572db1fcde8bc92f82a40</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=mhanafi" class="user-hover" rel="mhanafi">mhanafi</a>, I have to say that this issue may be related with the improperly bitmap consistency verification in our ldiskfs patch without handling flex_bg case. I made a patch <a href="https://review.whamcloud.com/28566" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28566</a> to handle related issues. Would you pleas to try (no need other former patches). Thanks!</p> <p>I did a build with #28566 and #28550 yesterday. For testing purpose, do these two conflict?<br/> I will undo #28550, but if these two do not collide, we can do testing with the builds I did yesterday.</p> <p>Never mind. I just did another build with #28550 pulled out.</p> <p>The filesystem is stable with the workaround patch (<a href="https://review.whamcloud.com/#/c/28489/" class="external-link" target="_blank" rel="nofollow noopener">/28489/</a>). Can we run with this patch for sometime without any underlining filesystem issues? Or should we replace it with 28566 ASAP.</p> <p>The patch 28550 will take effect before 28566, so if 28550 is applied, then 28566 is meaningless. But 28550 may do more things than the necessary fixes. I am afraid of some penitential side-effect.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>The filesystem is stable with the workaround patch (/28489/). Can we run with this patch for sometime without any underlining filesystem issues? Or should we replace it with 28566 ASAP. </pre> </div></div> <p>It is interesting to know that. Because 28489 is just a debug patch, I cannot imagine how it can resolve your issue. It may because your system has jumped over the groups with "BLOCK_UNINIT" flag and zero free blocks in GDP. If it is true, then applying 28566 will not show you more benefit. Since your system is stable running, you can replace the patches with 28566 when it 'corrupted' next time.</p> <p>Sorry I typed the patch number. I wanted to say it is stable with 28550.</p> <p> </p> <blockquote> <p>Sorry I typed the patch number. I wanted to say it is stable with 28550.</p></blockquote> <p>Then it is reasonable. As I explained above, 28550 may do more than the necessary fixes. But since it runs stable, you can keep it until next 'corruption'.</p> <p>updated: we have applied <a href="https://review.whamcloud.com/28566" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28566</a> Friday and the filesystem has been stable.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=mhanafi" class="user-hover" rel="mhanafi">mhanafi</a> Thanks for the update.</p> <p>Does this patch require any changes to e2fsck?</p> <p>I think that there may be something can be improved for mke2fs, not e2fsck.</p> <p>Do I need this patch for 2.10.0?</p> <p>Yes, master also needs the patch 28566.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/28566/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28566/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9410" title="on-disk bitmap corrupted" class="issue-link" data-issue-key="LU-9410"><del>LU-9410</del></a> ldiskfs: no check mb bitmap if flex_bg enabled<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 5506c15a65b3eebb9f15000105e6eb7c02742a10</p> <p>Minh Diep (minh.diep@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/28765" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28765</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9410" title="on-disk bitmap corrupted" class="issue-link" data-issue-key="LU-9410"><del>LU-9410</del></a> ldiskfs: no check mb bitmap if flex_bg enabled<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: 53d836b1e5d255558639fe8e4eae78a87a176d04</p> <p>John L. Hammond (john.hammond@intel.com) merged in patch <a href="https://review.whamcloud.com/28765/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28765/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9410" title="on-disk bitmap corrupted" class="issue-link" data-issue-key="LU-9410"><del>LU-9410</del></a> ldiskfs: no check mb bitmap if flex_bg enabled<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: <br/> Commit: 27f5b8b16416b04a561d0b0121860e2a5188be4a</p> Duplicate LU-1026 LU-7114 Related LU-10837 Development Rank 1|hzzbbz: Rank (Obsolete) 9223372036854775807 Severity