https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5371 Lustre <p>This issue was created by maloo for sarah &lt;sarah@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.hpdd.intel.com/test_sets/c8156066-0dc7-11e4-af8b-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/c8156066-0dc7-11e4-af8b-5254006e85c2</a>.</p> <p>The sub-test test_simul failed with the following error:</p> <blockquote> <p>simul failed! 1</p></blockquote> <p>test log:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>22:35:15: FAILED in simul_rmdir: too many operations succeeded (2). </pre> </div></div> client and server: lustre-b2_6-rc2 RHEL6 ldiskfs LU-5371

Failure on test suite parallel-scale test_simul

Bug Blocker Resolved Fixed Di Wang Maloo Fri, 18 Jul 2014 20:28:44 +0000 Fri, 17 Oct 2014 17:59:40 +0000 Thu, 21 Aug 2014 16:48:17 +0000 Lustre 2.6.0 Lustre 2.7.0 0 4 <p>Di,<br/> Could you please look into and comment on this one?<br/> Thank you!</p> <p>Also hit this issue in interop test between 2.5.2 server and b2_6-rc2 testing</p> <p><a href="https://testing.hpdd.intel.com/test_sets/3c3c2f74-0d82-11e4-b3f5-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/3c3c2f74-0d82-11e4-b3f5-5254006e85c2</a></p> <p>Hmm, I checked the log, it seems failures start to occur around July 3th. (build 2548)</p> <p>It seems simul test tries to rmdir a single directory from multiple clients(multiple threads), and only 1 thread should succeed. But two threads succeed in the test, that is why it causes failure. </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>04:50:57: FAILED in simul_rmdir: too many operations succeeded (2). </pre> </div></div> <p>Hmm, I checked the debug log on both clients and MDT side.</p> <p>Client1 (succeeds for rmdir)</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000080:00200000:1.0:1405575315.343857:0:29632:0:(namei.c:1137:ll_rmdir_generic()) VFS Op:name=simul_rmdir.0, dir=[0x2000013a8:0x143db:0x0](ffff880060e42138) 00010000:00010000:1.0:1405575315.343863:0:29632:0:(ldlm_request.c:1113:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff88005dc3e000 lock: ffff88001f3b3500/0x4c7dc53d5dce65cc lrc: 2/0,0 mode: PR/PR res: [0x2000013a7:0x1c43f:0x0].0 bits 0x13 rrc: 2 type: IBT flags: 0x8400000000 nid: local remote: 0xce57c426058b2c04 expref: -99 pid: 29632 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1405575315.343874:0:29632:0:(ldlm_request.c:1172:ldlm_cancel_pack()) ### packing ns: lustre-MDT0000-mdc-ffff88005dc3e000 lock: ffff88001f3b3500/0x4c7dc53d5dce65cc lrc: 1/0,0 mode: --/PR res: [0x2000013a7:0x1c43f:0x0].0 bits 0x13 rrc: 1 type: IBT flags: 0x4809400000000 nid: local remote: 0xce57c426058b2c04 expref: -99 pid: 29632 timeout: 0 lvb_type: 0 00010000:00010000:1.0:1405575315.343877:0:29632:0:(ldlm_request.c:1176:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1405575315.343877:0:29632:0:(ldlm_lock.c:219:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff88005dc3e000 lock: ffff88001f3b3500/0x4c7dc53d5dce65cc lrc: 0/0,0 mode: --/PR res: [0x2000013a7:0x1c43f:0x0].0 bits 0x13 rrc: 1 type: IBT flags: 0x4809400000000 nid: local remote: 0xce57c426058b2c04 expref: -99 pid: 29632 timeout: 0 lvb_type: 0 00000100:00100000:1.0:1405575315.343885:0:29632:0:(client.c:1480:ptlrpc_send_new_req()) Sending RPC pname:cluuid:pid:xid:nid:opc simul:49ab2d2b-0df9-c120-4bb6-1f0147383ab7:29632:1473843428301100:10.2.4.203@tcp:36 00000100:00100000:1.0:1405575315.343909:0:29632:0:(client.c:2146:ptlrpc_set_wait()) set ffff88001f1b0440 going to sleep for 43 seconds 00000100:00100000:1.0:1405575315.345000:0:29632:0:(client.c:1863:ptlrpc_check_set()) Completed RPC pname:cluuid:pid:xid:nid:opc simul:49ab2d2b-0df9-c120-4bb6-1f0147383ab7:29632:1473843428301100:10.2.4.203@tcp:36 00000080:00200000:1.0:1405575315.345007:0:29632:0:(llite_lib.c:1425:ll_clear_inode()) VFS Op:inode=[0x2000013a7:0x1c43f:0x0](ffff8800616f1178) </pre> </div></div> <p>Client2 (also succeeds with rmdir}</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000080:00200000:0.0:1405575315.345956:0:18310:0:(namei.c:1137:ll_rmdir_generic()) VFS Op:name=simul_rmdir.0, dir=[0x2000013a8:0x143db:0x0](ffff88005d0bebb8) 00010000:00010000:0.0:1405575315.345961:0:18310:0:(ldlm_request.c:1113:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff88007dbfd000 lock: ffff88002e7eed40/0x366918de5153be5b lrc: 2/0,0 mode: PR/PR res: [0x2000013a7:0x1c43f:0x0].0 bits 0x13 rrc: 2 type: IBT flags: 0x8400000000 nid: local remote: 0xce57c426058b2c20 expref: -99 pid: 18310 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1405575315.345970:0:18310:0:(ldlm_request.c:1172:ldlm_cancel_pack()) ### packing ns: lustre-MDT0000-mdc-ffff88007dbfd000 lock: ffff88002e7eed40/0x366918de5153be5b lrc: 1/0,0 mode: --/PR res: [0x2000013a7:0x1c43f:0x0].0 bits 0x13 rrc: 1 type: IBT flags: 0x4809400000000 nid: local remote: 0xce57c426058b2c20 expref: -99 pid: 18310 timeout: 0 lvb_type: 0 00010000:00010000:0.0:1405575315.345974:0:18310:0:(ldlm_request.c:1176:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1405575315.345975:0:18310:0:(ldlm_lock.c:219:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff88007dbfd000 lock: ffff88002e7eed40/0x366918de5153be5b lrc: 0/0,0 mode: --/PR res: [0x2000013a7:0x1c43f:0x0].0 bits 0x13 rrc: 1 type: IBT flags: 0x4809400000000 nid: local remote: 0xce57c426058b2c20 expref: -99 pid: 18310 timeout: 0 lvb_type: 0 00000100:00100000:0.0:1405575315.345981:0:18310:0:(client.c:1480:ptlrpc_send_new_req()) Sending RPC pname:cluuid:pid:xid:nid:opc simul:992018ad-6c7b-3f3b-15cf-42922a66aa9a:18310:1473838828202428:10.2.4.203@tcp:36 00000100:00100000:0.0:1405575315.345989:0:18310:0:(client.c:2146:ptlrpc_set_wait()) set ffff88002c1cf200 going to sleep for 43 seconds 00000100:00100000:0.0:1405575315.347314:0:18310:0:(client.c:1863:ptlrpc_check_set()) Completed RPC pname:cluuid:pid:xid:nid:opc simul:992018ad-6c7b-3f3b-15cf-42922a66aa9a:18310:1473838828202428:10.2.4.203@tcp:36 00000080:00200000:0.0:1405575315.347322:0:18310:0:(llite_lib.c:1425:ll_clear_inode()) VFS Op:inode=[0x2000013a7:0x1c43f:0x0](ffff88003ab0c1b8) </pre> </div></div> <p>It is interesting when MDT handle the request from client2, it is already return ENOENT, but it seems client2 ignore that</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000100:00100000:1.0:1405575315.353820:0:25427:0:(service.c:2144:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc mdt00_003:992018ad-6c7b-3f3b-15cf-42922a66aa9a+675:18310:x1473838828202428:12345-10.2.4.205@tcp:36 Request procesed in 988us (1036us total) trans 0 rc -2/-2 </pre> </div></div> <p>Sigh there are no enough debug log tell me what happen there, I will keep dig.</p> <p><a href="http://review.whamcloud.com/11170" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/11170</a> </p> <p>Sorry, this is brought in by <br/> commit 3ea78dd02a57211ae9b55111323d14cfbc71bc43<br/> Author: Wang Di &lt;di.wang@intel.com&gt;<br/> Date: Wed Jun 25 22:35:52 2014 -0700</p> <p> <a href="https://jira.whamcloud.com/browse/LU-4921" title="DNE clients should fall back to &quot;try all stripes&quot; for lookups in directories with unknown hash functions " class="issue-link" data-issue-key="LU-4921"><del>LU-4921</del></a> lmv: try all stripes for unknown hash functions</p> <p> For unknown hash type, LMV should try all stripes to locate<br/> the name entry. But it will only for lookup and unlink, i.e.<br/> we can only list and unlink entries under striped dir with<br/> unknown hash type.</p> <p> Signed-off-by: wang di &lt;di.wang@intel.com&gt;<br/> Change-Id: Ifeed7131c24e48277a6cc8fd4c09b7534e31079f<br/> Reviewed-on: <a href="http://review.whamcloud.com/10041" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/10041</a><br/> Tested-by: Jenkins<br/> Reviewed-by: John L. Hammond &lt;john.hammond@intel.com&gt;<br/> Reviewed-by: Andreas Dilger &lt;andreas.dilger@intel.com&gt;<br/> Tested-by: Maloo &lt;hpdd-maloo@intel.com&gt;</p> <p>Patch landed to Master.</p> <p>Hit this error in interop test between 2.6.0 client and master server, patch needs to be back ported to b2_6 fix the error</p> <p><a href="https://testing.hpdd.intel.com/test_sets/336e7e88-5558-11e4-a570-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/336e7e88-5558-11e4-a570-5254006e85c2</a></p> Related LU-4921 Development Rank 1|hzwrsn: Rank (Obsolete) 14975 Severity