https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1597 Lustre <p>We're currently seeing a user's reads and writes failing with -13 (-EACCES) errors. The errors are coming from a set of clients from a single cluster, but are using multiple different filesystems. From what I can tell, the -EACCES is coming from this part of the server code:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>filter_capa.c: </pre> </div></div> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">138 <span class="code-keyword">if</span> (capa == NULL) { 139 <span class="code-keyword">if</span> (fid) 140 CERROR(<span class="code-quote">"seq/fid/opc "</span>LPU64<span class="code-quote">"/"</span>DFID<span class="code-quote">"/"</span>LPX64 141 <span class="code-quote">": no capability has been passed\n"</span>, 142 seq, PFID(fid), opc); 143 <span class="code-keyword">else</span> 144 CERROR(<span class="code-quote">"seq/opc "</span>LPU64<span class="code-quote">"/"</span>LPX64 145 <span class="code-quote">": no capability has been passed\n"</span>, 146 seq, opc); 147 RETURN(-EACCES); 148 } </pre> </div></div> <p>The message on the client is:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Jul 3 13:26:50 ansel242 kernel: LustreError: 11-0: lsc-OST00b4-osc-ffff8806244c3800: Communicating with 172.19.1.113@o2ib100, operation ost_read failed with -13. Jul 3 13:26:50 ansel242 kernel: LustreError: Skipped 3495061 previous similar messages </pre> </div></div> <p>And there are corresponding messages on the server:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Jul 3 13:26:51 sumom13 kernel: LustreError: 24607:0:(filter_capa.c:146:filter_auth_capa()) seq/opc 0/0x40: no capability has been passed Jul 3 13:26:51 sumom13 kernel: LustreError: 24607:0:(filter_capa.c:146:filter_auth_capa()) Skipped 3495057 previous similar messages </pre> </div></div> <p>It appears the for each "ost_</p> {read|write} <p> failed" message on the client, there is a "no capability" message on the server.</p> <p>I'm unsure why the capability isn't being set by the client, but it seems that is causing the -EACCES error to get propagated to the clients.</p> <p>Lustre versions:</p> <p>Client: lustre-modules-2.1.1-13chaos_2.6.32_220.17.1.3chaos.ch5.x86_64.x86_64<br/> Server: lustre-modules-2.1.1-4chaos_2.6.32_220.7.1.7chaos.ch5.x86_64.x86_64</p> Client: lustre-modules-2.1.1-13chaos_2.6.32_220.17.1.3chaos.ch5.x86_64.x86_64<br/> Server: lustre-modules-2.1.1-4chaos_2.6.32_220.7.1.7chaos.ch5.x86_64.x86_64 LU-1597

Reads and Writes failing with -13 (-EACCES)

Bug Blocker Closed Fixed Zhenyu Xu Prakash Surya Tue, 3 Jul 2012 16:39:36 +0000 Wed, 30 Apr 2014 23:02:11 +0000 Thu, 27 Sep 2012 18:44:53 +0000 Lustre 2.3.0 Lustre 2.1.4 0 9 <p>I previously saw some patches from LLNL in the security/capability code. Any chance this is due to a delta between 2.1.1-4chaos and 2.1.1-13chaos, or are there other clients running 2.1.1-13chaos that are working correctly?</p> <p>Can the user access the same files correctly from other clients, and conversely do other users on the 2.1.1-13chaos clients run without problems?</p> <p>Andreas, perhaps you are thinking of <a href="https://jira.whamcloud.com/browse/LU-1102" title="NULL pointer dereference in capa_encrypt_id+0x8b/0x3e0" class="issue-link" data-issue-key="LU-1102"><del>LU-1102</del></a>? We had questions about capa stuff there that were never answered. We STILL don't know why Oleg claims capa is off by default when it appears to be on by default.<br/> We wound up with a patch that avoids an assertion, but that is about all.</p> <p>We have other 2.1.1-13chaos clients that are talking to 2.1.1-4chaos servers without this error (as far as I know...). But as far as I understand it the <a href="https://jira.whamcloud.com/browse/LU-1102" title="NULL pointer dereference in capa_encrypt_id+0x8b/0x3e0" class="issue-link" data-issue-key="LU-1102"><del>LU-1102</del></a> patch was to address a server-side crash, so we're not even using that in production yet.</p> <p>From what I was told by the admins, the user is able to access the files from another cluster running the same lustre version (2.1.1-13chaos) just fine. The problem was only seen on this specific cluster (ansel).</p> <p>After rebooting the clients, we are still seeing the -13 errors persist. Can we raise the priority as this is currently affecting production.</p> <p>EDIT: Sorry, I was wrong. It looks like the nodes which were rebooted are no longer hitting the issue. Why the reboot helped is still an open question.</p> <p>Bobijam</p> <p>Can you please look into this one?</p> <p>Peter</p> <p>Is there any update on this? We're now seeing this on one of our SCF clusters.</p> <p>Whamcloud, please update bug to blocker status. This is taking our largest cluster out of action at the moment. Thanks.</p> <p>So the -13 error happens on some clients, and after reboot, the clients are no longer hitting the issue? Can you please upload MDS/OSS and affected clients logs here?</p> <p>No, the current problems are on the SCF, so no logs are available.</p> <p>Why is the server complaining about capabilities? Can you explain the listed error messages?</p> <p>On one of the OSS nodes, it looks to me like the the ll_ost_io threads will say a message like one of the following:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>(filter_capa.c:146:filter_auth_capa()) seq/opc 0/0x20: no capability has been passed (filter_capa.c:146:filter_auth_capa()) seq/opc 0/0x280: no capability has been passed </pre> </div></div> <p>and the ldlm_cn thread will print these two lines together:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>(filter_capa.c:146:filter_auth_capa()) seq/opc 0/0x20: no capability has been passed (ost_handler.c:1534:ost_blocking_ast()) Error -13 syncing data on lock cancel </pre> </div></div> <p>I don't see anything else in the OSS's log at our normal log level except for some noise about a few router nodes that are down.</p> <p>Our default client log level isn't showing much there:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>ptlrpc_check_status...Communicating with X, operation ost_write failed with -13 </pre> </div></div> <p>also see ost_read failures with -13.</p> <p>And these messages too:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>vvp_io_commit_write()) Write page X of inode Y failed -13 </pre> </div></div> <blockquote><p>So the -13 error happens on some clients, and after reboot, the clients are no longer hitting the issue?</p></blockquote> <p>We believe that rebooting made it go away on clients previously. But we don't understand what is going on yet, so I can't claim definitively that a client reboot clears the problem. We are going to drain the 300 or so nodes that are printing the errors over night, and then reboot them in the morning (leaving a few drained for investigation).</p> <p>But that is just a wild attempt to restore normality. We really need to get to the bottom of the cause.</p> <p>We are investigating it, a patch will be out soon.</p> <p>it looks like ost_blocking_ast() hasn't set capa when calling obd_sync().</p> <p>patch tracking at <a href="http://review.whamcloud.com/3372" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/3372</a></p> <p>obdfilter: set default capa for OST</p> <p>A capability should be set for filter_sync(), and when the operation<br/> is come from OSS itself, the capability check can be passed.</p> <p>If clients do not support capability, the server capability check<br/> should be bypassed.</p> <p>Some of the opc make me think that there are other problematic areas. For instance, filter_setattr() seems to be one of the call paths triggering the "no capability has been passed" error.</p> <p>I see that there is now a ticket and patch to disable capa by default: <a href="http://jira.whamcloud.com/browse/LU-1621" class="external-link" rel="nofollow">http://jira.whamcloud.com/browse/LU-1621</a>.</p> <p>So it sounds like our quick work-around is to just disable capa.</p> <p>unfortunately capability feature is not done yet, we need disable it for now.</p> <p>Disabling capa on the servers got rid of the server error messages as expected, since filter_auth_capa() will now just always return 0.</p> <p>However, our clients are still throwing many -13 errors. So the question now is if this is something that is negotiate at mount time, and now that we've disabled it on servers the clients are going to be unhappy until we force them to reconnect?</p> <p>I'd rather not tell the admins to reboot 15000 nodes until I'm sure it will actually fix the problem.</p> <p>Hello?</p> <p>It looks like the clients are throwing the following two messages, not always at the same time:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Communicating with X, operation ost_write failed with -13 vvp_io_commit_write() write page X of inode Y failed -13 </pre> </div></div> <p>The "Communicating" error is thrown by client.c:ptlrpc_check_status(), which seems to be checking the rpc reply status. That would seem to imply that the server returned this error without a console message.</p> <p>I caught a log on a client with rpctrace enabled. I can't share it, but there didn't seem to be too much info there. Server logs might be better if I can get them. But that is harder without an automated trigger.</p> <p>Don't reconnect clients for now, we'll try to fix it on the server side.</p> <p>updated <a href="http://review.whamcloud.com/3372" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/3372</a></p> <p>obdfilter: fix some capa code for OST</p> <ul> <li>A capability should be set for filter_sync(), and when the operation<br/> is come from OSS itself, the capability check can be passed.</li> </ul> <ul> <li>filter_capa_fixoa() need check whether filter enabled capability.</li> </ul> <p>Ah, great. We'll be trying that today on our test system.</p> <p>See <a href="https://github.com/chaos/lustre/tree/2.1.1-17chaos" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/chaos/lustre/tree/2.1.1-17chaos</a> for our production solution. We took the one fix from <a href="http://review.whamcloud.com/3372" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/3372</a> to honor the capablities disabled flag, and then took the patch from <a href="https://jira.whamcloud.com/browse/LU-1621" title="Disable lustre capa by force" class="issue-link" data-issue-key="LU-1621"><del>LU-1621</del></a> to disable capabilities by default.</p> <p>Please let me know if this needs to be reopened.</p> <p>I see the following on worker nodes:</p> <p> Oct 3 08:50:58 wn612 kernel: LustreError: 22347:0:(vvp_io.c:1018:vvp_io_commit_write()) Write page 0 of inode ffff8101eae5f010 failed -13</p> <p>and on servers:</p> <p> Oct 3 08:57:43 oss6 kernel: LustreError: 5343:0:(filter_capa.c:151:filter_auth_capa()) seq/opc 0/0x20: no capability has been passed</p> <p>On both sides I have:</p> <p> lustre: 2.1.5<br/> kernel: patchless_client<br/> build: v2_1_5_0--PRISTINE-2.6.18-348.3.1.el5</p> <p>On worker nodes are scientific linux 5.9 and OFED-1.5.3.2, on servers pure centos 6.3.</p> <p>Is this the same bug ?</p> <p>We are seeing the same problem.<br/> Client is 2.1.4-5chaos.</p> <p>Has a solution been found?<br/> It is killing jobs here.</p> <p>Susan, the solution that I described in described for Lustre 2.1.1-17chaos is still in the Lustre 2.1.4-*chaos releases. You should open a new ticket describing your issue.</p> Related LU-1621 Development Rank 1|hzv2zb: Rank (Obsolete) 3982 Severity