[LU-7865] conf-sanity test_70a: @@@@@@ FAIL: delete dir fail

• Type: Bug
• Resolution: Cannot Reproduce
• Priority: Critical
• Labels: dne
• Severity: 3
• Assignee: Yang Sheng
• Reporter: Yang Sheng

Description

https://testing.hpdd.intel.com/test_sets/5d140f12-e631-11e5-abcf-5254006e85c2

CMD: trevis-42vm3 e2label /dev/lvm-Role_MDS/P4 2>/dev/null
Started lustre-MDT0003
mount lustre on /mnt/lustre.....
Starting client: trevis-42vm1.trevis.hpdd.intel.com:  -o user_xattr,flock trevis-42vm7@tcp:/lustre /mnt/lustre
CMD: trevis-42vm1.trevis.hpdd.intel.com mkdir -p /mnt/lustre
CMD: trevis-42vm1.trevis.hpdd.intel.com mount -t lustre -o user_xattr,flock trevis-42vm7@tcp:/lustre /mnt/lustre
rm: cannot remove `/mnt/lustre/d70a.conf-sanity': Directory not empty
 conf-sanity test_70a: @@@@@@ FAIL: delete dir fail 
  Trace dump:
  = /usr/lib64/lustre/tests/test-framework.sh:4670:error_noexit()
  = /usr/lib64/lustre/tests/test-framework.sh:4704:error()
  = /usr/lib64/lustre/tests/conf-sanity.sh:4679:test_70a()
  = /usr/lib64/lustre/tests/test-framework.sh:4951:run_one()
  = /usr/lib64/lustre/tests/test-framework.sh:4988:run_one_logged()
  = /usr/lib64/lustre/tests/test-framework.sh:4853:run_test()

From client log:

00000100:00000001:0.0:1457547974.439172:0:24095:0:(client.c:2537:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1)
00000100:00000001:0.0:1457547974.439174:0:24095:0:(client.c:1301:after_reply()) Process entered
02000000:00000001:0.0:1457547974.439175:0:24095:0:(sec.c:1025:do_cli_unwrap_reply()) Process entered
00000100:00000001:0.0:1457547974.439176:0:24095:0:(pack_generic.c:577:__lustre_unpack_msg()) Process entered
00000100:00000001:0.0:1457547974.439178:0:24095:0:(pack_generic.c:596:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0)
02000000:00000001:0.0:1457547974.439180:0:24095:0:(sec.c:1079:do_cli_unwrap_reply()) Process leaving (rc=0 : 0 : 0)
00000100:00001000:0.0:1457547974.439183:0:24095:0:(import.c:1662:at_measured()) add 1 to ffff880059b5bbe8 time=0 v=1 (1 0 0 0)
00000100:00001000:0.0:1457547974.439186:0:24095:0:(import.c:1662:at_measured()) add 1 to ffff880059b5bbb0 time=0 v=1 (1 0 0 0)
00010000:00000010:0.1:1457547974.439212:0:24095:0:(ldlm_lock.c:444:lock_handle_free()) slab-freed 'lock': 504 at ffff88006268c280.
00000100:00000001:0.0:1457547974.439216:0:24095:0:(client.c:1218:ptlrpc_check_status()) Process entered
00000100:00000040:0.0:1457547974.439219:0:24095:0:(client.c:1236:ptlrpc_check_status()) @@@ status is -39  req@ffff880037f31080 x1528349813899424/t0(0) o36->lustre-MDT0000-mdc-ffff88007b9cf000@10.9.5.252@tcp:12/10 lens 624/424 e 0 to 0 dl 1457547981 ref 2 fl Rpc:R/0/0 rc 0/-39
00000100:00000001:0.0:1457547974.439224:0:24095:0:(client.c:1242:ptlrpc_check_status()) Process leaving (rc=18446744073709551577 : -39 : ffffffffffffffd9)
00000100:00000001:0.0:1457547974.439227:0:24095:0:(client.c:2627:ptlrpc_free_committed()) Process entered
00000100:00000040:0.0:1457547974.439228:0:24095:0:(client.c:2635:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff88007b9cf000: skip recheck: last_committed 8589934603
00000100:00000001:0.0:1457547974.439230:0:24095:0:(client.c:2636:ptlrpc_free_committed()) Process leaving
00000100:00000001:0.0:1457547974.439231:0:24095:0:(client.c:1482:after_reply()) Process leaving (rc=18446744073709551577 : -39 : ffffffffffffffd9)
00000100:00000040:0.0:1457547974.439234:0:24095:0:(lustre_net.h:2442:ptlrpc_rqphase_move()) @@@ move req "Rpc" -> "Interpret"  req@ffff880037f31080 x1528349813899424/t0(0) o36->lustre-MDT0000-mdc-ffff88007b9cf000@10.9.5.252@tcp:12/10 lens 624/424 e 0 to 0 dl 1457547981 ref 2 fl Rpc:R/0/0 rc -39/-39
00000100:00000001:0.0:1457547974.439239:0:24095:0:(client.c:1940:ptlrpc_check_set()) Process leaving via interpret (rc=18446744073709551577 : -39 : 0xffffffffffffffd9)

But MDS log not found any useful information.

Comments

[Yang Sheng]

Don't hit it a long time. So close first. Please feel free to reopen it.

[Gerrit Updater]

Yang Sheng ([e-mail]) uploaded a new patch: http://review.whamcloud.com/19131
Subject: LU-7865 lod: debug patch
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 42623d2c54938b3b9813413cb7c1fde58bbde9b8

[Richard Henwood (Inactive)]

And another example, from Master running review-dne-part-1:

https://testing.hpdd.intel.com/test_sets/29a4fde6-f03e-11e5-8202-5254006e85c2

[Richard Henwood (Inactive)]

Another example, from Master running review-dne-part-1

https://testing.hpdd.intel.com/test_sets/372f8082-ef74-11e5-8ddc-5254006e85c2