https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4395 Lustre <p>KIT has run into an issue where the MDT is creating files with objects that do not exist. Some of the symptoms look similar to <a href="https://jira.whamcloud.com/browse/LU-4034" title="Cannot allocate memory on clients with 2.4.X" class="issue-link" data-issue-key="LU-4034"><del>LU-4034</del></a>.</p> <p>On client:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">[root@client scc]# touch tmp/gaga touch: setting times of `tmp/gaga': No such file or directory [root@client scc]# lfs getstripe tmp/gaga tmp/gaga lmm_stripe_count: 4 lmm_stripe_size: 1048576 lmm_layout_gen: 0 lmm_stripe_offset: 5 obdidx objid objid group 5 65948624 0x3ee4bd0 0 25 66739551 0x3fa5d5f 0 9 65922640 0x3ede650 0 24 66084357 0x3f05e05 0 LustreError: 11-0: HC3WORK-OST0005-osc-ffff8804987dec00: Communicating with 172.26.3.138@o2ib, operation ldlm_enqueue failed with -12. [root@mds2 perftest]# ls -al ls: cannot access eaea: Cannot allocate memory ls: cannot access gaga: Cannot allocate memory total 12 drwxr-xr-x 3 er2341 scc 4096 Dec 12 21:40 . drwx------ 10 er2341 scc 4096 Sep 19 16:16 .. -rw-r--r-- 1 root root 0 Dec 12 21:41 e -????????? ? ? ? ? ? eaea -rw-r--r-- 1 root root 0 Dec 12 21:41 f -????????? ? ? ? ? ? gaga drwxr-xr-x 2 root root 4096 Dec 12 21:40 tmp </pre> </div></div> <p>on OSS:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Dec 12 22:25:05 oss1 kernel: : LustreError: 14167:0:(ldlm_resource.c:1165:ldlm_resource_get()) HC3WORK-OST0005: lvbo_init failed <span class="code-keyword">for</span> resource 0x3ee4bd0:0x0: rc = -2 </pre> </div></div> <p>On thing that's odd is that all the other OSTs on the system delete orphan objects around that object ID number, but not ost5:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"># echo $((0x3ee4bd0)) 65948624 Dec 12 15:01:02 oss1 kernel: : Lustre: HC3WORK-OST0002: deleting orphan objects from 0x0:66829746 to 0x0:66830014 Dec 12 15:01:02 oss1 kernel: : Lustre: HC3WORK-OST0006: deleting orphan objects from 0x0:66151265 to 0x0:66151535 Dec 12 15:01:02 oss1 kernel: : Lustre: HC3WORK-OST0000: deleting orphan objects from 0x0:66341886 to 0x0:66342155 Dec 12 15:01:02 oss1 kernel: : Lustre: HC3WORK-OST0004: deleting orphan objects from 0x0:65766910 to 0x0:65767207 Dec 12 15:01:02 oss1 kernel: : Lustre: HC3WORK-OST0003: deleting orphan objects from 0x0:66145109 to 0x0:66145379 </pre> </div></div> <p>Another weird thing is that the OSTs seem to delete the same objects repeatedly:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Dec 9 15:04:30 oss1 kernel: : Lustre: HC3WORK-OST0004: deleting orphan objects from 0x0:65766910 to 0x0:65767015 Dec 9 15:11:41 oss1 kernel: : Lustre: HC3WORK-OST0004: deleting orphan objects from 0x0:65766910 to 0x0:65767015 Dec 10 09:58:31 oss1 kernel: : Lustre: HC3WORK-OST0004: deleting orphan objects from 0x0:65766910 to 0x0:65767047 Dec 10 16:20:25 oss1 kernel: : Lustre: HC3WORK-OST0004: deleting orphan objects from 0x0:65766910 to 0x0:65767079 Dec 10 16:33:00 oss1 kernel: : Lustre: HC3WORK-OST0004: deleting orphan objects from 0x0:65766910 to 0x0:65767111 Dec 11 15:54:57 oss1 kernel: : Lustre: HC3WORK-OST0004: deleting orphan objects from 0x0:65766910 to 0x0:65767143 Dec 11 16:50:03 oss1 kernel: : Lustre: HC3WORK-OST0004: deleting orphan objects from 0x0:65766910 to 0x0:65767175 Dec 12 15:01:02 oss1 kernel: : Lustre: HC3WORK-OST0004: deleting orphan objects from 0x0:65766910 to 0x0:65767207 </pre> </div></div> <p>The filesystem was put back into production by disabling the OSTs that have this symptom. Are there any suggestions for what to look at in order to further debug this issue? Any logs we should get?</p> <p>Thanks,<br/> Kit</p> LU-4395

files created with non-existent objects

Bug Major Resolved Fixed Niu Yawei Kit Westneat Wed, 18 Dec 2013 18:08:27 +0000 Wed, 29 Jan 2014 15:05:14 +0000 Wed, 29 Jan 2014 15:05:14 +0000 Lustre 2.4.1 0 4 <p>I've been looking at the prealloc info on the mds and the LAST_ID on the OST:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">[root@mds1 ~]# grep . /proc/fs/lustre/osc/HC3WORK-OST0005-osc-MDT0000/pre* /proc/fs/lustre/osc/HC3WORK-OST0005-osc-MDT0000/prealloc_last_id:65948661 /proc/fs/lustre/osc/HC3WORK-OST0005-osc-MDT0000/prealloc_last_seq:0x100050000 /proc/fs/lustre/osc/HC3WORK-OST0005-osc-MDT0000/prealloc_next_id:65948632 /proc/fs/lustre/osc/HC3WORK-OST0005-osc-MDT0000/prealloc_next_seq:0x100050000 /proc/fs/lustre/osc/HC3WORK-OST0005-osc-MDT0000/prealloc_reserved:0 /proc/fs/lustre/osc/HC3WORK-OST0005-osc-MDT0000/prealloc_status:-19 # debugfs -c -R <span class="code-quote">'dump O/0/LAST_ID /tmp/LAST_ID'</span> /dev/mapper/ost_HC3WORK_5 # hexdump -e <span class="code-quote">'1/8 <span class="code-quote">"%d\n"</span>'</span> /tmp/LAST_ID 65988995 </pre> </div></div> <p>So it looks like the OST's LAST_ID is significantly larger than what the MDT sees. What could be causing that? A corrupted last_rcvd file on the MDT?</p> <p>Niu</p> <p>Could you please look into this one?</p> <p>Thanks</p> <p>Peter</p> <p>The LAST_ID on OST is much larger than the last_id on MDT, because the difference is even greater than the OST_MAX_PRECREATE (20000), the orhpan cleanup wasn't triggered on ost5.</p> <p>I'm not sure which LAST_ID is corrupted, was there any abnormal message on ost5 or MDT before this happened? That probably can give us some clue on how this happened.</p> <p>What's the id of last physical object on ost5? Could you have a check?</p> <p>The last physical object ID is 65988995, so it seems like a problem with the MDT. I'll upload the MDS and OSS logs. Do you think deleting the last_rcvd file on the MDT would fix the issue?</p> <p>mds2 is acting as the client in this case.</p> <p>Is the mds running 2.4? I see some 2.1 symbols in the mds log.</p> <p>I didn't see how this happened from the log, and yes, deleting the lov_objids on MDS or just changing the bad value in lov_objids may fix the problem.</p> <p>I think it was upgraded Dec 9, so it's possible that before then there are 2.1.x symbols. </p> <p>Do you know why it would keep deleting the same orphan objects? It seems like 65766910 is deleted at least 8 times.</p> <blockquote> <p>Do you know why it would keep deleting the same orphan objects? It seems like 65766910 is deleted at least 8 times.</p></blockquote> <p>You mean deleting orphan on ost4?</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Dec 9 15:04:30 oss1 kernel: : Lustre: HC3WORK-OST0004: deleting orphan objects from 0x0:65766910 to 0x0:65767015 ... Dec 9 15:11:41 oss1 kernel: : Lustre: HC3WORK-OST0004: deleting orphan objects from 0x0:65766910 to 0x0:65767015 ... Dec 10 09:58:31 oss1 kernel: : Lustre: HC3WORK-OST0004: deleting orphan objects from 0x0:65766910 to 0x0:65767047 ... Dec 10 16:20:25 oss1 kernel: : Lustre: HC3WORK-OST0004: deleting orphan objects from 0x0:65766910 to 0x0:65767079 ... Dec 10 16:33:00 oss1 kernel: : Lustre: HC3WORK-OST0004: deleting orphan objects from 0x0:65766910 to 0x0:65767111 ... Dec 11 15:54:57 oss1 kernel: : Lustre: HC3WORK-OST0004: deleting orphan objects from 0x0:65766910 to 0x0:65767143 ... Dec 11 16:50:03 oss1 kernel: : Lustre: HC3WORK-OST0004: deleting orphan objects from 0x0:65766910 to 0x0:65767175 ... Dec 12 15:01:02 oss1 kernel: : Lustre: HC3WORK-OST0004: deleting orphan objects from 0x0:65766910 to 0x0:65767207 </pre> </div></div> <p>I think it probably there wasn't any file striped on ost4 was created, so orphan cleanup always starts from 65766910 (all beyond 65766910 are precreated objects).</p> <p>Hmm, I think that is unlikely, as the filesystem is in production. I also don't understand why it would keep removing 65766910. Does it not keep track of orphans already deleted?</p> <p>I mean no files striped on ost4 was created during Dec 9 ~ Dec 12, is it possible? The deleted orhpan will be created by precreation, that's why it has to delete orhpan on each reboot.</p> <p>Kit, can you please tell us if Niu's idea might be right that ost4 did not see any create activity on it?</p> <p>It's possible, but I think unlikely. The filesystem was in use then. </p> <p>We were able to remove the last_rcvd file from the MDT, but that didn't work. I just reread your previous comment and realized that you were talking about the lov_objids file, oops. I'll try to get another downtime to remove the lov_objids file. </p> <p>Thanks,<br/> Kit</p> <p>Removing the lov_objids file seemed to fix the problem, so I think this ticket can be closed. </p> <p>ok thanks Kit!</p> Development Rank 1|hzwblb: Rank (Obsolete) 12059 Severity