https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4592 Lustre <p>As part of preparation testing, the customer performed a failover tests. The customer rebooted the primary MDS in order to confirm the standby MDS would takeover and not interrupt the job. The job died when the client was unable to open a file.</p> <p>3 files attached.<br/> mds00.20140131.17 Primary MDS that was rebooted<br/> mds01.20140131.17 Secondary MDS that took over when mds00 went down<br/> client.807442 Client logs from the 2 compute nodes running the job (#807422) that failed. (The two nodes are mu0104 and mu0105)</p> <p>Error reported on MDS01 -<br/> Jan 31 17:07:12 l1-mds01 kernel: : LustreError: 18626:0:(mdt_open.c:1314:mdt_reint_open()) @@@ OPEN &amp; CREAT not in open replay. req@ffff881006dda400 x1458783605491287/t0(30064772087) o101-&gt;8eb15a41-9744-ff91-d294-57256d6605bc@10.11.16.104@tcp:0/0 lens 544/4552 e 0 to 0 dl 1391213274 ref 1 fl Interpret:/4/0 rc 0/0</p> <p>ERRORs on client -<br/> Jan 31 17:07:12 mu0104 kernel: : LustreError: 2376:0:(client.c:2634:ptlrpc_replay_interpret()) @@@ status <del>116, old was 0 req@ffff88025258e400 x1458783605491285/t30064772084(30064772084) o35</del>&gt;l1-MDT0000-mdc-ffff8804014a9000@10.1.15.2@o2ib5:23/10 lens 360/424 e 0 to 0 dl 1391213270 ref 2 fl Interpret:R/4/0 rc -116/-116 <br/> Jan 31 17:07:13 mu0104 kernel: : LustreError: 2376:0:(client.c:2634:ptlrpc_replay_interpret()) @@@ status <del>116, old was 0 req@ffff88017c924400 x1458783605697158/t30064772108(30064772108) o35</del>&gt;l1-MDT0000-mdc-ffff8804014a9000@10.1.15.2@o2ib5:23/10 lens 360/424 e 0 to 0 dl 1391213270 ref 2 fl Interpret:R/4/0 rc -116/-116 </p> Lustre 2.1.5 servesr, LLNL Chaos clients LU-4592

mdt_reint_open()) @@@ OPEN & CREAT not in open replay

Bug Major Resolved Done Hongchao Zhang Oz Rentas Wed, 5 Feb 2014 23:45:35 +0000 Sat, 23 Jan 2016 01:29:16 +0000 Sat, 23 Jan 2016 01:29:16 +0000 Lustre 2.1.5 0 8 <p>Hongchao</p> <p>Could you please advise on this one?</p> <p>Thanks</p> <p>Peter</p> <p>Hi Oz,</p> <p>do you mount the Lustre with ACL enabled and disabled the "identity_upcall"?</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Jan 31 17:17:55 l1-mds01 kernel: : Lustre: 22128:0:(mdt_lproc.c:414:lprocfs_wr_identity_upcall()) l1-MDT0000: disable "identity_upcall" with ACL enabled maybe cause unexpected "EACCESS" Jan 31 17:17:55 l1-mds01 kernel: : Lustre: 22128:0:(mdt_lproc.c:416:lprocfs_wr_identity_upcall()) l1-MDT0000: identity upcall set to NONE </pre> </div></div> <p>the problem in the job is just -EACCESS,</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>... Rank 26 Host mu0104.localdomain FATAL ERROR 1391213863: Unable to open file /lustre/lscratch1/atorrez/out.1391213561.26 for read. (errno=Permission denied) (MPI_Error = 42) Rank 28 Host mu0104.localdomain FATAL ERROR 1391213863: Unable to open file /lustre/lscratch1/atorrez/out.1391213561.28 for read. (errno=Permission denied) (MPI_Error = 42) Rank 29 Host mu0104.localdomain FATAL ERROR 1391213863: Unable to open file /lustre/lscratch1/atorrez/out.1391213561.29 for read. (errno=Permission denied) (MPI_Error = 42) Rank 47 Host mu0104.localdomain FATAL ERROR 1391213863: Unable to open file /lustre/lscratch1/atorrez/out.1391213561.47 for read. (errno=Permission denied) (MPI_Error = 42) ... </pre> </div></div> <p>could you please test it without ACL to check whether it is the cause? Thanks!</p> <p>The customer reports they are not mounting with ACL support, as seen here:<br/> /dev/mapper/vg_l1-mdt on /lustre/l1/mdt type lustre (rw)</p> <p>Any other suggestions on where we can look?</p> <p>Side note - On my system I was able to duplicate the error they received by setting upcall_identity to NONE, and mounting with ACL:<br/> Feb 11 09:56:12 es0 kernel: : Lustre: 7799:0:(mdt_lproc.c:372:lprocfs_wr_identity_upcall()) testfs-MDT0000: disable "identity_upcall" with ACL enabled maybe cause unexpected "EACCESS"</p> <p><span class="error">&#91;root@es0 ~&#93;</span># mount |grep mdt<br/> /dev/mapper/vg_testfs-mdt on /lustre/testfs/mdt type lustre (rw,acl)</p> <p>Any updates on this one? </p> <p>sorry for delayed response.</p> <p>from the code, this debug line is only printed when mounted with ACL</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-keyword">static</span> <span class="code-object">int</span> lprocfs_wr_identity_upcall(struct file *file, <span class="code-keyword">const</span> <span class="code-object">char</span> *buffer, unsigned <span class="code-object">long</span> count, void *data) { ... <span class="code-keyword">if</span> (strcmp(hash-&gt;uc_upcall, <span class="code-quote">"NONE"</span>) == 0 &amp;&amp; mdt-&gt;mdt_opts.mo_acl) &lt;---- here, <span class="code-quote">"mo_acl is 1"</span> CWARN(<span class="code-quote">"%s: disable \"</span>identity_upcall\<span class="code-quote">" with ACL enabled maybe "</span> <span class="code-quote">"cause unexpected \"</span>EACCESS\<span class="code-quote">"\n"</span>, mdt_obd_name(mdt)); ... } </pre> </div></div> <p>Is it possible that mds00 mounts without ACL but mds01 with it?</p> <p>Thanks!</p> <p>After being onsite with customer I can confirm that when running the mount command that the system appears to NOT be mounting with acls.</p> <p>/dev/mapper/vg_l1-mdt on /lustre/l1/mdt type lustre (rw)</p> <p>Re-asking to Oz's question, is there another place that it may show as being mounted with acl's that I can check the next time I'm onsite?</p> <p>currently, the mount options is not printed when showing the mount info if the mount type is "lustre" (and it will show when mounting it with "ldiskfs" type)</p> <p>the default mount options could contain "ACL" (it's the case in my local node RHEL6.5/x86_64),<br/> could you please mount the MDT with "-o noacl" explicitly and retest it, Thanks</p> <p>Hi Oz,<br/> Do you need more works on this ticket? Or can we close it?<br/> Thanks</p> <p>yes, it can be closed. thanks.</p> <p>Thanks Oz and Hongchao.</p> <p>~ jfc.</p> Development Rank 1|hzween: Rank (Obsolete) 12545 Severity