https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-16806 Lustre <p>Attempts to create directories are intermittently failing when using "mkdir".</p> <p>So far, "lfs mkdir" succeeds every time, so does file creation using "touch".</p> <p> </p> server (garter): <br/> toss 4.5-6rc6 <br/> 4.18.0-425.19.2.1toss.t4.x86_64 <br/> 2.14.0_19.llnl <br/> <br/> client (mutt): <br/> toss 4.5-6 <br/> 4.18.0-425.19.2.1toss.t4.x86_64 <br/> 2.15.2_5.llnl LU-16806

mkdir: cannot create directory ‘test5’: Object is remote

Bug Major Closed Fixed Lai Siyao Gian-Carlo Defazio llnl Tue, 9 May 2023 00:15:48 +0000 Fri, 2 Feb 2024 13:25:01 +0000 Wed, 5 Jul 2023 16:04:24 +0000 Lustre 2.14.0 Lustre 2.15.2 0 9 <p>For my notes, local issue is TOSS-6002</p> <p>the llnl lustre versions are at</p> <p><a href="https://github.com/LLNL/lustre/tree/2.14.0_19.llnl" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/LLNL/lustre/tree/2.14.0_19.llnl</a></p> <p><a href="https://github.com/LLNL/lustre/tree/2.15.2_5.llnl" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/LLNL/lustre/tree/2.15.2_5.llnl</a></p> <p> </p> <p>Here's an example of a few failures followed by a successful create</p> <p> </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>(mutt12):mdts_3_osts_3$ pwd /p/lflood/defazio1/mdtest/mdts_3_osts_3 (mutt12):mdts_3_osts_3$ mkdir junk20 mkdir: cannot create directory ‘junk20’: Object is remote (mutt12):mdts_3_osts_3(1)$ mkdir junk20 mkdir: cannot create directory ‘junk20’: Object is remote (mutt12):mdts_3_osts_3(1)$ mkdir junk20 mkdir: cannot create directory ‘junk20’: Object is remote (mutt12):mdts_3_osts_3(1)$ mkdir junk20 (mutt12):mdts_3_osts_3$ ls -ld junk20 drwx------ 2 defazio1 defazio1 26624 May  8 17:53 junk20 </pre> </div></div> <p> </p> <p> </p> <p>The parent directory is /p/lflood/defazio1/mdtest/mdts_3_osts_3</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>(mutt12):mdts_3_osts_3$ stat .   File: .   Size: 26624           Blocks: 52         IO Block: 131072 directory Device: 11602e98h/291516056d    Inode: 198161923999531372  Links: 16 Access: (0700/drwx------)  Uid: (28153/defazio1)   Gid: (28153/defazio1) Access: 2023-05-08 16:51:39.000000000 -0700 Modify: 2023-05-08 17:53:11.000000000 -0700 Change: 2023-05-08 17:53:11.000000000 -0700  Birth: 2023-05-08 11:43:54.000000000 -0700 </pre> </div></div> <p>it was created with the commands</p> <p> </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lfs setdirstripe --mdt-index 3 /p/lflood/defazio1/mdtest/mdts_3_osts_3 lfs setstripe --ost 3 /p/lflood/defazio1/mdtest/mdts_3_osts_3 chown defazio1:defazio1 /p/lflood/defazio1/mdtest/mdts_3_osts_3</pre> </div></div> <p> </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>(mutt12):mdts_3_osts_3(4)$ lfs  getdirstripe  . lmv_stripe_count: 0 lmv_stripe_offset: 3 lmv_hash_type: none (mutt12):mdts_3_osts_3$ lfs  getstripe  -d . stripe_count:  1 stripe_size:   1048576 pattern:       raid0 stripe_offset: 3 </pre> </div></div> <p> </p> <p>In this case some directories are created on the first try, but most fail</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>(mutt12):mdts_3_osts_3$ for x in {1..20}; do mkdir test_name$x; done mkdir: cannot create directory ‘test_name1’: Object is remote mkdir: cannot create directory ‘test_name2’: Object is remote mkdir: cannot create directory ‘test_name3’: Object is remote mkdir: cannot create directory ‘test_name4’: Object is remote mkdir: cannot create directory ‘test_name5’: Object is remote mkdir: cannot create directory ‘test_name6’: Object is remote mkdir: cannot create directory ‘test_name8’: Object is remote mkdir: cannot create directory ‘test_name9’: Object is remote mkdir: cannot create directory ‘test_name10’: Object is remote mkdir: cannot create directory ‘test_name11’: Object is remote mkdir: cannot create directory ‘test_name12’: Object is remote mkdir: cannot create directory ‘test_name13’: Object is remote mkdir: cannot create directory ‘test_name14’: Object is remote mkdir: cannot create directory ‘test_name15’: Object is remote mkdir: cannot create directory ‘test_name16’: Object is remote mkdir: cannot create directory ‘test_name18’: Object is remote mkdir: cannot create directory ‘test_name19’: Object is remote mkdir: cannot create directory ‘test_name20’: Object is remote (mutt12):mdts_3_osts_3(1)$ for x in {1..20}; do mkdir test_name$x; done mkdir: cannot create directory ‘test_name1’: Object is remote mkdir: cannot create directory ‘test_name2’: Object is remote mkdir: cannot create directory ‘test_name3’: Object is remote mkdir: cannot create directory ‘test_name4’: Object is remote mkdir: cannot create directory ‘test_name5’: Object is remote mkdir: cannot create directory ‘test_name7’: File exists mkdir: cannot create directory ‘test_name8’: Object is remote mkdir: cannot create directory ‘test_name9’: Object is remote mkdir: cannot create directory ‘test_name10’: Object is remote mkdir: cannot create directory ‘test_name12’: Object is remote mkdir: cannot create directory ‘test_name13’: Object is remote mkdir: cannot create directory ‘test_name14’: Object is remote mkdir: cannot create directory ‘test_name16’: Object is remote mkdir: cannot create directory ‘test_name17’: File exists mkdir: cannot create directory ‘test_name18’: Object is remote mkdir: cannot create directory ‘test_name19’: Object is remote mkdir: cannot create directory ‘test_name20’: Object is remote </pre> </div></div> <p> </p> <p>The directories that are created inherit the striping (this was done after more mkdir attempts, so more directories exist than the 2 indicated above)</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>(mutt12):mdts_3_osts_3$ lfs getdirstripe test_name* lmv_stripe_count: 0 lmv_stripe_offset: 3 lmv_hash_type: none lmv_stripe_count: 0 lmv_stripe_offset: 3 lmv_hash_type: none lmv_stripe_count: 0 lmv_stripe_offset: 3 lmv_hash_type: none lmv_stripe_count: 0 lmv_stripe_offset: 3 lmv_hash_type: none lmv_stripe_count: 0 lmv_stripe_offset: 3 lmv_hash_type: none lmv_stripe_count: 0 lmv_stripe_offset: 3 lmv_hash_type: none lmv_stripe_count: 0 lmv_stripe_offset: 3 lmv_hash_type: none lmv_stripe_count: 0 lmv_stripe_offset: 3 lmv_hash_type: none lmv_stripe_count: 0 lmv_stripe_offset: 3 lmv_hash_type: none</pre> </div></div> <p> </p> <p>However, using lfs mkdir seems to work fine</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>(mutt12):mdts_3_osts_3$ for x in {1..20}; do lfs mkdir -i $(($x % 4)) lfs_name$x; done (mutt12):mdts_3_osts_3$ ls lfs_name* lfs_name1: lfs_name10: lfs_name11: lfs_name12: lfs_name13: lfs_name14: lfs_name15: lfs_name16: lfs_name17: lfs_name18: lfs_name19: lfs_name2: lfs_name20: lfs_name3: lfs_name4: lfs_name5: lfs_name6: lfs_name7: lfs_name8: lfs_name9: </pre> </div></div> <p>I've uploaded debug logs for the attempted creation of the directory junk8</p> <p>debug log which contains failed attempt for junk8 is<br/> dk.mutt12.1683589208</p> <p>It then succeeded (after failing about 10 times) in<br/> dk.mutt12.1683589684</p> <p>Serguei</p> <p>Can you please advise?</p> <p>Thanks</p> <p>Peter</p> <p>It looks like this one needs attention of experts in higher-level Lustre. </p> <p>For example, is <a href="https://jira.whamcloud.com/browse/LUDOC-289" title="remote directory don&#39;t work for non-root users" class="issue-link" data-issue-key="LUDOC-289"><del>LUDOC-289</del></a> relevant here?</p> <p>Hi Serguei or Peter,</p> <p>Can you ask an appropriate Whamcloud-er to advise?  The two test clusters involved will be taken away from us for other testing on Tuesday or Wednesday of next week.</p> <p>thanks</p> <p>Could you collect MDS debug logs upon failure? I'm afraid some patch for backward compatibility is missing, maybe it's on server side. </p> <p>I've added mkdir-attempt-client-and-mds.tar.gz which has debug logs for the client and mds's with debug=-1.</p> <p>"Lai Siyao &lt;lai.siyao@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/51045" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/51045</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16806" title="mkdir: cannot create directory ‘test5’: Object is remote" class="issue-link" data-issue-key="LU-16806"><del>LU-16806</del></a> lod: access DMV when it's valid<br/> Project: fs/lustre-release<br/> Branch: b2_14<br/> Current Patch Set: 1<br/> Commit: 65887f6728d68d8b2e10969df25e5303603a4797</p> <p>This is caused by a bug in 2.14 code, but it's not an issue in the following release because related code has been removed.</p> <p>So LLNL's exposure to this issue will disappear once they upgrade to 2.15.x?</p> <p>Yes, upgrading to 2.15.x can get around this issue since related code has been removed.</p> <p>The patch fixed the issue on 2.14.</p> <p>I'm a bit confused by what the patch is doing.</p> <p>Is the idea that when (lds-&gt;lds_dir_def_striping_set == 0) that means that the parent directory doesn't have default striping set for its sub-directories, so when the sub-directory is created it should instead inherit from the root of the whole filesystem?</p> <p> </p> <p>Hi Lai,</p> <p>Please also confirm whether you believe 2.14 servers with this patch will work properly with both Lustre 2.12 and 2.15 clients.  As we work to switch to Lustre 2.15, we will have</p> <p>Clients: Lustre 2.12, 2.15<br/> Routers: Lustre 2.12, 2.15</p> <p>thanks</p> <p>Gian, these code is to check whether client sends mkdir request to wrong MDT. If parent has default LMV (2.14 code doesn't check this), and this default LMV is not space balanced, it's not allowed to mkdir on remote MDT.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">if</span> (hint-&gt;dah_parent &amp;&amp; dt_object_remote(hint-&gt;dah_parent) &amp;&amp; lds &amp;&amp; lds-&gt;lds_dir_def_striping_set &amp;&amp; lds-&gt;lds_dir_def_stripe_offset != LMV_OFFSET_DEFAULT) GOTO(out, rc = -EREMOTE); </pre> </div></div> <p>As for your question, yes, if parent doesn't have default LMV, the filesystem default LMV will be applied.</p> <p>Olaf, yes, it works with both 2.12 and 2.15 clients.</p> <p>Is there any further work outstanding here? AFAIK the b2_14 fix has been proven to work and this issue does not exist on more current releases, so I would think that we can close out the ticket...</p> <p>This issue is fixed for us on 2.14 and has been pulled into our local branch.</p> <p>reopening to remove topllnl</p> <p>reclosing after removing topllnl</p> Related Development Rank 1|i03ksv: Rank (Obsolete) 9223372036854775807 Severity