[LU-5294] mdd_unlink() returning -7

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Affects Version/s: Lustre 2.4.3
• Environment: RHEL 6.4, kernel 2.6.32_358.23.2.el6,
• Severity: 3
• Assignee: Oleg Drokin
• Reporter: Blake Caldwell

Description

Any client on this filesystem will get back -7 from an unlink or rm. Creates, reads, writes work fine. We tried multiple users, and found no difference. Upcall is successful:

1. /usr/sbin/l_getidentity linkfarm-MDT0000 0

+trace and +rpctrace debugs were captured on the MDT while performing the unlink on a client 10.36.226.85@o2ib

I can upload a full debug log. This is easy to recreate and capture, so just let me know which debug flags would useful.

00000040:00000001:19.0:1404405371.756302:0:13436:0:(llog_osd.c:317:llog_osd_declare_write_rec()) Process leaving (rc=0 : 0 : 0)
00000040:00000001:19.0:1404405371.756303:0:13436:0:(llog.c:714:llog_declare_write_rec()) Process leaving (rc=0 : 0 : 0)
00000040:00000001:19.0:1404405371.756303:0:13436:0:(llog_cat.c:443:llog_cat_declare_add_rec()) Process leaving (rc=0 : 0 : 0)
00000040:00000001:19.0:1404405371.756304:0:13436:0:(llog.c:790:llog_declare_add()) Process leaving (rc=0 : 0 : 0)
00000004:00000001:19.0:1404405371.756305:0:13436:0:(osp_sync.c:206:osp_sync_declare_add()) Process leaving (rc=0 : 0 : 0)
00000004:00000001:19.0:1404405371.756306:0:13436:0:(osp_object.c:322:osp_declare_object_destroy()) Process leaving (rc=0 : 0 : 0)
00000004:00000001:19.0:1404405371.756307:0:13436:0:(lod_object.c:1044:lod_declare_object_destroy()) Process leaving (rc=0 : 0 : 0)
00000004:00000001:19.0:1404405371.756309:0:13436:0:(lod_object.c:434:lod_declare_xattr_set()) Process entered
00000004:00000001:19.0:1404405371.756310:0:13436:0:(lod_object.c:464:lod_declare_xattr_set()) Process leaving (rc=18446744073709551609 : -7 : fffffffffffffff9)
00000004:00000001:19.0:1404405371.756311:0:13436:0:(mdd_dir.c:1422:mdd_unlink()) Process leaving via stop (rc=18446744073709551609 : -7 : 0xfffffffffffffff9)
00000004:00000001:19.0:1404405371.756313:0:13436:0:(osd_handler.c:915:osd_trans_stop()) Process entered
00040000:00000001:19.0:1404405371.756314:0:13436:0:(qsd_handler.c:1074:qsd_op_end()) Process entered
00040000:00000001:19.0:1404405371.756315:0:13436:0:(qsd_handler.c:1102:qsd_op_end()) Process leaving
00000004:00000001:19.0:1404405371.756316:0:13436:0:(osd_handler.c:968:osd_trans_stop()) Process leaving (rc=0 : 0 : 0)
00000004:00000001:19.0:1404405371.756317:0:13436:0:(mdt_reint.c:868:mdt_reint_unlink()) Process leaving
00000004:00000001:19.0:1404405371.756317:0:13436:0:(mdt_handler.c:2791:mdt_object_unlock()) Process entered
00000004:00000001:19.0:1404405371.756318:0:13436:0:(mdt_handler.c:2739:mdt_save_lock()) Process entered

...
00000004:00000001:19.0:1404405371.756396:0:13436:0:(mdt_internal.h:584:mdt_object_put()) Process entered
00000020:00000001:19.0:1404405371.756397:0:13436:0:(lustre_fid.h:715:fid_flatten32()) Process leaving (rc=4196255 : 4196255 : 40079f)
00000004:00000001:19.0:1404405371.756398:0:13436:0:(mdt_internal.h:586:mdt_object_put()) Process leaving
00000004:00000001:19.0:1404405371.756398:0:13436:0:(mdt_reint.c:1375:mdt_reint_rec()) Process leaving (rc=18446744073709551609 : -7 : fffffffffffffff9)
00000004:00000001:19.0:1404405371.756399:0:13436:0:(mdt_handler.c:1832:mdt_reint_internal()) Process leaving
02000000:00000001:19.0:1404405371.756400:0:13436:0:(upcall_cache.c:276:upcall_cache_put_entry()) Process entered
02000000:00000001:19.0:1404405371.756400:0:13436:0:(upcall_cache.c:287:upcall_cache_put_entry()) Process leaving
00000004:00000001:19.0:1404405371.756401:0:13436:0:(mdt_handler.c:429:mdt_client_compatibility()) Process entered
00000004:00000001:19.0:1404405371.756401:0:13436:0:(mdt_handler.c:433:mdt_client_compatibility()) Process leaving
00000004:00000001:19.0:1404405371.756402:0:13436:0:(mdt_lib.c:572:mdt_fix_reply()) Process entered
00000004:00000001:19.0:1404405371.756403:0:13436:0:(mdt_lib.c:671:mdt_fix_reply()) Process leaving (rc=0 : 0 : 0)
00000004:00000001:19.0:1404405371.756404:0:13436:0:(mdt_handler.c:1898:mdt_reint()) Process leaving (rc=18446744073709551609 : -7 : fffffffffffffff9)
00010000:00000001:19.0:1404405371.756405:0:13436:0:(ldlm_lib.c:2440:target_send_reply()) Process entered
00010000:00000001:19.0:1404405371.756406:0:13436:0:(ldlm_lib.c:2393:target_pack_pool_reply()) Process entered
00010000:00000001:19.0:1404405371.756407:0:13436:0:(ldlm_lib.c:2412:target_pack_pool_reply()) Process leaving (rc=0 : 0 : 0)

...
00010000:00000001:19.0:1404405371.756416:0:13436:0:(ldlm_lib.c:2452:target_send_reply()) Process leaving
00000004:00000001:19.0:1404405371.756417:0:13436:0:(mdt_handler.c:3103:mdt_req_handle()) Process leaving (rc=0 : 0 : 0)
00000004:00000001:19.0:1404405371.756417:0:13436:0:(mdt_handler.c:3429:mdt_handle0()) Process leaving (rc=0 : 0 : 0)
00000004:00000001:19.0:1404405371.756418:0:13436:0:(mdt_handler.c:3463:mdt_handle_common()) Process leaving (rc=0 : 0 : 0)
00000100:00100000:19.0:1404405371.756421:0:13436:0:(service.c:2055:ptlrpc_server_handle_request()) Handled RPC pname:cluuid+ref:pid:xid:nid:opc mdt03_035:12cfc1a2-8c1a-ebe6-ebdb-eb076741d9d3+16:27222:x1472624384278960:12345-10.36.226.85@o2ib:36 Request procesed in 395us (422us total) trans 0 rc -7/-7
00000100:00100000:19.0:1404405371.756424:0:13436:0:(nrs_fifo.c:244:nrs_fifo_req_stop()) NRS stop fifo request from 12345-10.36.226.85@o2ib, seq: 2243836