https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11013 Lustre <p>Setting up Lustre testbed in ANL with: </p> <ul class="alternate" type="square"> <li>4 OSSs, total 16 OSTs ( 8 JBODs each with 60 HDDs)</li> <li>Hybrid lustre, MGT/MDT - mdraid - raid10 - ldiskfs, OST - zfs - dRaid</li> <li>MGT - 2 SSDs, raid10 - ldiskfs</li> <li>MDT0 - 12 SSDs, raid10 - ldiskfs</li> <li>MDT1 - 10 SSDs, raid10 - ldiskfs</li> <li>Each OST, 30 HDDs, zfs dRaid with 3*(8+1) + 1</li> </ul> <p> </p> <ul> <li>Filled up the fs to about 99%, we got data corruption problem after cleaned up fs and ran zfs scrub. Quite severe and ended up crashed the fs.</li> <li>Rebuild lustre dRaid fs, and test again in order to duplicate the problem.</li> <li>On first iteration of fill and clean up, the fs was holding up. Only got "One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected." on two dRaid zpool. So just clear up those errors.</li> <li>After 2nd iteration, finally able to reproduce the error, after emptied file system and run scrub, we got the same data corruption problem ("One or more devices has experienced an error resulting in data corruption. Application may be affected").</li> <li>Change the zpool to raidz2 with 3*(8+2) and we don't have this problem.</li> </ul> RHEL-7.4, in-kernel ofed, mellanox FDR10, Lustre-2.10.3, dRaid-(pull-7078), dm-multipath LU-11013

Data Corruption error on Lustre ZFS dRaid

Bug Minor Resolved Cannot Reproduce Isaac Huang Kurniawan Alfizah Thu, 10 May 2018 16:06:31 +0000 Tue, 11 May 2021 11:06:22 +0000 Tue, 11 May 2021 11:06:22 +0000 Lustre 2.10.3 0 5 <p>Have you tried this with native ZFS+dRAID to see if it hits the same corruption? That would isolate the problem to dRAID vs. an interaction between Lustre and dRAID. </p> <p>Note that you should make the native ZFS dataset the same way as Lustre, namely to enable <tt>recordsize=1024k</tt>, <tt>dnodesize=auto</tt>, <tt>multimount</tt>. It might be best to format the OST with <tt>mkfs.lustre</tt> as today, then set <tt>canmount=yes</tt> and mount it locally for your testing. </p> <p>When was the dRAID code last refreshed? I pushed quite some changes a couple of weeks ago - please make sure to run the latest code. Did you build zfs and spl with <em>--enable-debug</em> and set zfs module option draid_debug_lvl=5? Also, please see below on what debug information to gather for dRAID bugs:</p> <p><a href="https://github.com/zfsonlinux/zfs/wiki/dRAID-HOWTO#troubleshooting" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/zfsonlinux/zfs/wiki/dRAID-HOWTO#troubleshooting</a></p> <p>We're using this one '<a href="https://github.com/zfsonlinux/zfs/pull/7078" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/zfsonlinux/zfs/pull/7078</a>'  I cloned them around early March 18.</p> <p>Btw, following Andreas suggestion, I think I might be able to re-create the problem in our VM cluster. Created VM with 30 virtual hdds, filled them up to about 98% and then removed, I got data corruption. Same thing, with or without lustre. But I don't see the problem with raidz2.</p> <p>On wolf-16, created the draid with 30 hdds, and then filled them up, I even managed to crash ZFS itself. This one is Isaac built ZFS though, so could be different problem.</p> <p> </p> <p> </p> <p>This is presumably fixed in the ZFS 2.1 dRAID implementation upstream.</p> Development Rank 1|hzzx1j: Rank (Obsolete) 9223372036854775807 Severity