https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-8100 Lustre <p>Our test setup contains 16 MDTs. Our clients only ever see 10 MDTs. All output below is from a client with the file system mounted. All were formatted at the same time with the same script and same mgs nid.</p> <p>Attempts to use any MDT that is not listed fails (i.e. lfs mkdir --index=10 will fail).</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@catalyst320:mdc]# lfs mkdir --index=12 /p/lustre/dinatale/testdir error on LL_IOC_LMV_SETSTRIPE '/p/lustre/dinatale/testdir' (3): No such device error: mkdir: create stripe dir '/p/lustre/dinatale/testdir' failed [root@catalyst320:mdc]# lfs mkdir --index=8 /p/lustre/dinatale/testdir [root@catalyst320:mdc]# lfs getdirstripe /p/lustre/dinatale/testdir/ /p/lustre/dinatale/testdir/ lmv_stripe_count: 0 lmv_stripe_offset: 8 </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@catalyst320:mdc]# cat /proc/fs/lustre/lmv/lustre-clilmv-ffff881003e14400/target_obd 0: lustre-MDT0000_UUID ACTIVE 1: lustre-MDT0001_UUID ACTIVE 2: lustre-MDT0002_UUID ACTIVE 3: lustre-MDT0003_UUID ACTIVE 4: lustre-MDT0004_UUID ACTIVE 5: lustre-MDT0005_UUID ACTIVE 6: lustre-MDT0006_UUID ACTIVE 7: lustre-MDT0007_UUID ACTIVE 8: lustre-MDT0008_UUID ACTIVE 9: lustre-MDT0009_UUID ACTIVE </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>ls /proc/fs/lustre/lmv/lustre-clilmv-ffff881003e14400/target_obds/ lustre-MDT0000-mdc-ffff881003e14400 lustre-MDT0006-mdc-ffff881003e14400 lustre-MDT0012-mdc-ffff881003e14400 lustre-MDT0001-mdc-ffff881003e14400 lustre-MDT0007-mdc-ffff881003e14400 lustre-MDT0013-mdc-ffff881003e14400 lustre-MDT0002-mdc-ffff881003e14400 lustre-MDT0008-mdc-ffff881003e14400 lustre-MDT0014-mdc-ffff881003e14400 lustre-MDT0003-mdc-ffff881003e14400 lustre-MDT0009-mdc-ffff881003e14400 lustre-MDT0015-mdc-ffff881003e14400 lustre-MDT0004-mdc-ffff881003e14400 lustre-MDT0010-mdc-ffff881003e14400 lustre-MDT0005-mdc-ffff881003e14400 lustre-MDT0011-mdc-ffff881003e14400 </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@catalyst320:mdc]# grep current_state */state lustre-MDT0000-mdc-ffff881003e14400/state:current_state: FULL lustre-MDT0001-mdc-ffff881003e14400/state:current_state: FULL lustre-MDT0002-mdc-ffff881003e14400/state:current_state: FULL lustre-MDT0003-mdc-ffff881003e14400/state:current_state: FULL lustre-MDT0004-mdc-ffff881003e14400/state:current_state: FULL lustre-MDT0005-mdc-ffff881003e14400/state:current_state: FULL lustre-MDT0006-mdc-ffff881003e14400/state:current_state: FULL lustre-MDT0007-mdc-ffff881003e14400/state:current_state: FULL lustre-MDT0008-mdc-ffff881003e14400/state:current_state: FULL lustre-MDT0009-mdc-ffff881003e14400/state:current_state: FULL lustre-MDT0010-mdc-ffff881003e14400/state:current_state: FULL lustre-MDT0011-mdc-ffff881003e14400/state:current_state: FULL lustre-MDT0012-mdc-ffff881003e14400/state:current_state: FULL lustre-MDT0013-mdc-ffff881003e14400/state:current_state: FULL lustre-MDT0014-mdc-ffff881003e14400/state:current_state: FULL lustre-MDT0015-mdc-ffff881003e14400/state:current_state: FULL </pre> </div></div> lustre-2.8.0_14_gd0cbf68-1.x86_64<br/> 2.6.32-573.22.1.1chaos.ch5.4.x86_64 LU-8100

Missing MDTs in /proc/fs/lustre/lmv/lustre-clilmv-.../target_obd

Bug Major Resolved Fixed Lai Siyao Giuseppe Di Natale llnl patch Wed, 4 May 2016 18:18:11 +0000 Mon, 27 Feb 2017 22:50:49 +0000 Wed, 21 Sep 2016 05:26:09 +0000 Lustre 2.8.0 Lustre 2.9.0 0 10 <p>Lai</p> <p>Could you please advise on this issue?</p> <p>Thanks</p> <p>Peter</p> <p>Note that when we formatted these MDTs, we accidentally named them</p> <p>0000 - 0009 and 0010 - 00015</p> <p>instead of</p> <p>0000 - 000F</p> <p>So there's a gap in the numbering system we used for the name. That shouldn't matter, as far as I'm aware, but I mention it because of the correlation with the symptom we saw.</p> <p>okay, I'll try to format system like yours and reproduce.</p> <p>at the same time, could you remount a client, `lfs mkdir --index=12 /p/lustre/dinatale/testdir` and then collect debug logs of this whole process?</p> <p>Unfortunately, the file system we were testing with was only a temporary set up on one of our clusters. It no longer exists, so we can't collect any logs for a client for that specific file system. I may be able to try and reproduce the issue today and get you logs if I am successful.</p> <p>Was able to reproduce the issue. Collected a log from a client involving a mount and the lfs command you requested. Let me know if you need anything else. The log file is called "debug_client_missing_mdts.log".</p> <p>I was able to confirm Olaf's speculation on the naming convention. It appears the naming may be the source of the problem. I went ahead and redeployed a test file system where the MDT names ranged from 0000-000F and the client was able to connect to all MDTs. For completeness, lots of info below.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@catalyst100:~]# lfs mkdir --index=12 /p/lustre/dinatale/testdir [root@catalyst100:~]# lfs getdirstripe /p/lustre/dinatale/testdir /p/lustre/dinatale/testdir lmv_stripe_count: 0 lmv_stripe_offset: 12 </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@catalyst100:mdc]# grep current_state */state lustre-MDT0000-mdc-ffff880fbb04f400/state:current_state: FULL lustre-MDT0001-mdc-ffff880fbb04f400/state:current_state: FULL lustre-MDT0002-mdc-ffff880fbb04f400/state:current_state: FULL lustre-MDT0003-mdc-ffff880fbb04f400/state:current_state: FULL lustre-MDT0004-mdc-ffff880fbb04f400/state:current_state: FULL lustre-MDT0005-mdc-ffff880fbb04f400/state:current_state: FULL lustre-MDT0006-mdc-ffff880fbb04f400/state:current_state: FULL lustre-MDT0007-mdc-ffff880fbb04f400/state:current_state: FULL lustre-MDT0008-mdc-ffff880fbb04f400/state:current_state: FULL lustre-MDT0009-mdc-ffff880fbb04f400/state:current_state: FULL lustre-MDT000a-mdc-ffff880fbb04f400/state:current_state: FULL lustre-MDT000b-mdc-ffff880fbb04f400/state:current_state: FULL lustre-MDT000c-mdc-ffff880fbb04f400/state:current_state: FULL lustre-MDT000d-mdc-ffff880fbb04f400/state:current_state: FULL lustre-MDT000e-mdc-ffff880fbb04f400/state:current_state: FULL lustre-MDT000f-mdc-ffff880fbb04f400/state:current_state: FULL </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@catalyst100:~]# ls /proc/fs/lustre/lmv/lustre-clilmv-ffff880fbb04f400/target_obds/ lustre-MDT0000-mdc-ffff880fbb04f400 lustre-MDT0004-mdc-ffff880fbb04f400 lustre-MDT0008-mdc-ffff880fbb04f400 lustre-MDT000c-mdc-ffff880fbb04f400 lustre-MDT0001-mdc-ffff880fbb04f400 lustre-MDT0005-mdc-ffff880fbb04f400 lustre-MDT0009-mdc-ffff880fbb04f400 lustre-MDT000d-mdc-ffff880fbb04f400 lustre-MDT0002-mdc-ffff880fbb04f400 lustre-MDT0006-mdc-ffff880fbb04f400 lustre-MDT000a-mdc-ffff880fbb04f400 lustre-MDT000e-mdc-ffff880fbb04f400 lustre-MDT0003-mdc-ffff880fbb04f400 lustre-MDT0007-mdc-ffff880fbb04f400 lustre-MDT000b-mdc-ffff880fbb04f400 lustre-MDT000f-mdc-ffff880fbb04f400 </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@catalyst100:~]# lfs mdts MDTS: 0: lustre-MDT0000_UUID ACTIVE 1: lustre-MDT0001_UUID ACTIVE 2: lustre-MDT0002_UUID ACTIVE 3: lustre-MDT0003_UUID ACTIVE 4: lustre-MDT0004_UUID ACTIVE 5: lustre-MDT0005_UUID ACTIVE 6: lustre-MDT0006_UUID ACTIVE 7: lustre-MDT0007_UUID ACTIVE 8: lustre-MDT0008_UUID ACTIVE 9: lustre-MDT0009_UUID ACTIVE 10: lustre-MDT000a_UUID ACTIVE 11: lustre-MDT000b_UUID ACTIVE 12: lustre-MDT000c_UUID ACTIVE 13: lustre-MDT000d_UUID ACTIVE 14: lustre-MDT000e_UUID ACTIVE 15: lustre-MDT000f_UUID ACTIVE </pre> </div></div> <p>This looks to be just as designed, because during format "--index" specifies the target index in the system. So in your original setup, `lfs mkdir --index 10 ...` will fail, but `lfs mkdir --index 16 ...` should succeed, because the MDT with index 10 doesn't exist, but 16 exists.</p> <p>Lai,</p> <p>I see what you mean. That explains why our mkdir failed. However the proc files seem not to be consistent, which seems like a separate problem.</p> <p>On the client, MDTs with indexes 0x10-0x15 are missing from the listing in /proc/fs/lustre/lmv/lustre-clilmv-ffff881003e14400/target_obd, even though they are all present in /proc/fs/lustre/lmv/lustre-clilmv-ffff881003e14400/target_obds/. You can see this in the description, above. Why would that be?</p> <p>thanks,<br/> Olaf</p> <p>Lai, I just launched a new test setup to do more testing. lfs mkdir was successful like you suggested it would be. Would this mean that there is a bug somewhere in the proc handler for /proc/fs/lustre/lmv/lustre-clilmv-.../target_obd since it doesn't contain all active MDTs in this case?</p> <p>yes, current LMV code stores targets in an array, and the index can not exceed total count, so only the targets whose index is below the total count are listed.</p> <p>I'll make a fix later.</p> <p>I found the portion of the code you are talking about. I might already have a patch.</p> <p>Just to make sure no one else is working on this, I'll be submitting a patch soon.</p> <p>No, thanks for your work!</p> <p>Giuseppe Di Natale (dinatale2@llnl.gov) uploaded a new patch: <a href="http://review.whamcloud.com/20336" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/20336</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8100" title="Missing MDTs in /proc/fs/lustre/lmv/lustre-clilmv-.../target_obd" class="issue-link" data-issue-key="LU-8100"><del>LU-8100</del></a> lmv: Correctly generate target_obd<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: d8c2209167efb0fcbba5a1a390e07e800e938fec</p> <p>Went ahead and submitted a patch. It works on a VM setup I have, but I'm not 100% it's correct. I would also like to introduce a test to check this file. The test would require MDTs to have non-consecutive indices. Is it possible within the test framework to manually assign indices to MDTs or can someone point me to an example test?</p> <p>It should be possible to generate non-consecutive MDT indices. See conf-sanity test_56 for an example of this with large OST indices. Maybe renaming that to test_56a and adding test_56b for large MDT indices is the right way to go?</p> <p>Giuseppe Di Natale (dinatale2@llnl.gov) uploaded a new patch: <a href="http://review.whamcloud.com/20546" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/20546</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8100" title="Missing MDTs in /proc/fs/lustre/lmv/lustre-clilmv-.../target_obd" class="issue-link" data-issue-key="LU-8100"><del>LU-8100</del></a> conf-sanity: Test the correctness of target_obd<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: e7cd83e1a36a6ea790233055222eb052670c4324</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/20336/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/20336/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8100" title="Missing MDTs in /proc/fs/lustre/lmv/lustre-clilmv-.../target_obd" class="issue-link" data-issue-key="LU-8100"><del>LU-8100</del></a> lmv: Correctly generate target_obd<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: e5bb5a373cb3548ec8f5989c7e08fa7a3c8259f3</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/20546/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/20546/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8100" title="Missing MDTs in /proc/fs/lustre/lmv/lustre-clilmv-.../target_obd" class="issue-link" data-issue-key="LU-8100"><del>LU-8100</del></a> tests: Test the correctness of target_obd<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 1a8c46a434b20c1083fafbded4d39e6dbbdff37a</p> <p>Landed for 2.9</p> Blocker Development End date Thu, 2 Jun 2016 18:18:11 +0000 Rank 1|hzyacf: Rank (Obsolete) 9223372036854775807 Severity Start date Wed, 4 May 2016 18:18:11 +0000