https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-12134 Lustre <p>I have been following <a href="https://jira.whamcloud.com/browse/LU-11205" title="Failure to clear the changelog for user 1 on MDT" class="issue-link" data-issue-key="LU-11205"><del>LU-11205</del></a> and am trying to reproduce it. I experimented with the <tt>llog_reader</tt>:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root]# lustre/tests/llmount.sh [root]# mkdir /mnt/ldiskfs-mdt0 [root]# mount -t ldiskfs /dev/mapper/mds1_flakey /mnt/ldiskfs-mdt0 [root]# lctl --device lustre-MDT0000 changelog_register lustre-MDT0000: Registered changelog userid 'cl1' [root]# lustre/utils/llog_reader /mnt/ldiskfs-mdt0/changelog_users llog_reader: Error allocating -16 bytes for recs_buf: Cannot allocate memory (12) llog_reader: Could not pack buffer.: Cannot allocate memory (12) [root]# lustre/utils/llog_reader /mnt/ldiskfs-mdt0/changelog_catalog llog_reader: File too small for llog header: need 8192, size 0 : Success (0) llog_reader: Could not pack buffer.: Input/output error (5) </pre> </div></div> <p>... Not ideal, but this is probably because there are not any changelog records yet. So I generate one:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root]# touch /mnt/lustre/file [root]# lustre/utils/llog_reader /mnt/ldiskfs-mdt0/changelog_users llog_reader: Error allocating -16 bytes for recs_buf: Cannot allocate memory (12) llog_reader: Could not pack buffer.: Cannot allocate memory (12) [root]# lustre/utils/llog_reader /mnt/ldiskfs-mdt0/changelog_catalog rec #1 type=1064553b len=64 offset 8192 Header size : 8192 Time : Fri Mar 29 14:12:30 2019 Number of records: 1 Target uuid : ----------------------- #01 (064)id=[0xa:0x1:0x0]:0 path=O/1/d10/10 </pre> </div></div> <p>Everything is fine now (well, the users changelog is still broken, but I do not really need it). But what happens if I try and generate a few more?</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root]# for _ in {0..15}; do touch /mnt/lustre/file-{0..1023}; rm /mnt/lustre/file-{0..1023}; done [root]# lustre/utils/llog_reader /mnt/ldiskfs-mdt0/changelog_catalog rec #1 type=1064553b len=64 offset 8192 off 8256 skip 8128 to next chunk. The log is corrupt (too big at 0) llog_reader: Could not pack buffer.: Invalid argument (22) </pre> </div></div> <p>Back to a broken changelog...<br/> <tt>lfs changelog</tt> seems to be doing fine though:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root]# lfs changelog lustre-MDT0000 1 01CREAT 14:14:51.291849812 2019.03.29 0x0 t=[0x200000401:0x12:0x0] j=touch.0 ef=0xf u=0:0 nid=0@lo p=[0x200000007:0x1:0x0] file 2 11CLOSE 14:14:51.293009818 2019.03.29 0x42 t=[0x200000401:0x12:0x0] j=touch.0 ef=0xf u=0:0 nid=0@lo 3 01CREAT 14:23:23.124863776 2019.03.29 0x0 t=[0x200000401:0x13:0x0] j=touch.0 ef=0xf u=0:0 nid=0@lo p=[0x200000007:0x1:0x0] file-0 4 11CLOSE 14:23:23.139933635 2019.03.29 0x42 t=[0x200000401:0x13:0x0] j=touch.0 ef=0xf u=0:0 nid=0@lo 5 01CREAT 14:23:23.140437103 2019.03.29 0x0 t=[0x200000401:0x14:0x0] j=touch.0 ef=0xf u=0:0 nid=0@lo p=[0x200000007:0x1:0x0] file-1 (...) 49153 06UNLNK 14:23:57.498329392 2019.03.29 0x1 t=[0x200000401:0x4011:0x0] j=rm.0 ef=0xf u=0:0 nid=0@lo p=[0x200000007:0x1:0x0] file-1022 49154 06UNLNK 14:23:57.499430628 2019.03.29 0x1 t=[0x200000401:0x4012:0x0] j=rm.0 ef=0xf u=0:0 nid=0@lo p=[0x200000007:0x1:0x0] file-1023 </pre> </div></div> <p>Which makes me hope the issue is only with the <tt>llog_reader</tt>.</p> LU-12134

llog_reader (incorrectly?) reports a corrupted changelog

Bug Minor Closed Done Mikhail Pershin CEA ORNL Fri, 29 Mar 2019 14:30:53 +0000 Sun, 16 Jan 2022 08:49:33 +0000 Sun, 16 Jan 2022 08:49:33 +0000 Upstream 0 6 <p>Mike</p> <p>Can you advise</p> <p>Thanks</p> <p>Peter</p> <p>try to remount local fs and check again?</p> <p>&gt; try to remount local fs and check again?</p> <p>Yes, thank you, this helped:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root]# lustre/utils/llog_reader /mnt/ldiskfs-mdt0/changelog_catalog llog_reader: Error allocating -16 bytes for recs_buf: Cannot allocate memory (12) llog_reader: Could not pack buffer.: Cannot allocate memory (12) [root]# umount /mnt/ldiskfs-mdt0 [root]# umount /mnt/lustre-mds1 [root]# mount -t lustre /dev/mapper/mds1_flakey /mnt/lustre-mds1/ [root]# mount -t ldiskfs /dev/mapper/mds1_flakey /mnt/ldiskfs-mdt0 [root]# lustre/utils/llog_reader /mnt/ldiskfs-mdt0/changelog_catalog rec #1 type=1064553b len=64 offset 8192 Header size : 8192 Time : Tue Apr 2 00:10:37 2019 Number of records: 1 Target uuid : ----------------------- #01 (064)id=[0x12:0x1:0x0]:0 path=O/1/d18/18 </pre> </div></div> <p>It would seem unmounting both the ldiskfs and the lustre fs before remounting them is mandatory, otherwise it doesn't work.</p> <p>Dropping caches seems to work too.</p> <p>Quentin, even if you are mounting the same device, you are using 2 different filesystems and 2 different mount points. Both of them have their own caching mechanisms. You cannot expect data written by Lustre being visible right away in the ldiskfs mount. If ldiskfs has already cached several blocks, data structure, etc. it does not expect somebody else changing them.</p> <p>Mounting the lustre device with ldiskfs works if you want to read data already present or if Lustre is unmount.</p> <p>As this is mostly a caching issue, dropping cache should definitely help.</p> <p>OK, thank you Aurélien!</p> Related LU-11205 Development Rank 1|i00e7b: Rank (Obsolete) 9223372036854775807 Severity