[LU-6849] sanity-quota test 22 LBUG with “ASSERTION( orro->oo_ref == 0 ) failed: Busy NRS TRR policy object for OST”

• Type: Bug
• Resolution: Unresolved
• Priority: Minor
• Affects Version/s: Lustre 2.8.0, Lustre 2.9.0
• Environment: review-zfs-part-1 test group in autotest
• Severity: 3
• Assignee: Li Xi
• Reporter: James Nunez (Inactive)

Description

sanity-quota test 22 failed with 'server shadow-14vm11 environments are insane!'. Logs for this failure are:
2015-07-13 16:36:38 - https://testing.hpdd.intel.com/test_sets/ba5b90c2-29a9-11e5-b0c2-5254006e85c2

From the OST console:

17:35:27:Lustre: DEBUG MARKER: umount -d -f /mnt/ost2
17:35:27:Lustre: 4055:0:(client.c:2018:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1436808903/real 1436808903]  req@ffff88007da416c0 x1506591584180392/t0(0) o250->MGC10.1.4.170@tcp@10.1.4.170@tcp:26/25 lens 520/544 e 0 to 1 dl 1436808914 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
17:35:27:Lustre: server umount lustre-OST0001 complete
17:35:27:LustreError: 4041:0:(nrs_orr.c:483:nrs_trr_hop_exit()) ASSERTION( orro->oo_ref == 0 ) failed: Busy NRS TRR policy object for OST with index 1, with 1 refs
17:35:27:LustreError: 4041:0:(nrs_orr.c:483:nrs_trr_hop_exit()) LBUG
17:35:27:Pid: 4041, comm: obd_zombid
17:35:27:
17:35:27:Call Trace:
17:35:27: [<ffffffffa0709875>] libcfs_debug_dumpstack+0x55/0x80 [libcfs]
17:35:27: [<ffffffffa0709e77>] lbug_with_loc+0x47/0xb0 [libcfs]
17:35:27: [<ffffffffa0ada51c>] nrs_trr_hop_exit+0x1cc/0x1d0 [ptlrpc]
17:35:27: [<ffffffffa071bf46>] cfs_hash_putref+0x1a6/0x480 [libcfs]
17:35:27: [<ffffffffa0721dbb>] ? cfs_binheap_destroy+0x3db/0x810 [libcfs]
17:35:27: [<ffffffffa0adb61f>] nrs_orr_stop+0x6f/0x440 [ptlrpc]
17:35:27: [<ffffffffa0ad11af>] nrs_policy_stop0+0x4f/0x1f0 [ptlrpc]
17:35:27: [<ffffffffa0ad14f8>] nrs_policy_stop_locked+0x1a8/0x230 [ptlrpc]
17:35:27: [<ffffffffa0ad2854>] nrs_policy_unregister+0xb4/0x670 [ptlrpc]
17:35:27: [<ffffffffa0ad2f32>] ptlrpc_service_nrs_cleanup+0x122/0x5b0 [ptlrpc]
17:35:27: [<ffffffffa0a9a617>] ptlrpc_unregister_service+0x417/0xe10 [ptlrpc]
17:35:27: [<ffffffff8105bd73>] ? __wake_up+0x53/0x70
17:35:27: [<ffffffffa0fdc12a>] ost_cleanup+0x6a/0x1b0 [ost]
17:35:27: [<ffffffffa084cf55>] class_decref+0x1e5/0x4d0 [obdclass]
17:35:27: [<ffffffffa0715c01>] ? libcfs_debug_msg+0x41/0x50 [libcfs]
17:35:27: [<ffffffffa083753c>] obd_zombie_impexp_cull+0x61c/0xac0 [obdclass]
17:35:27: [<ffffffffa0837a45>] obd_zombie_impexp_thread+0x65/0x190 [obdclass]
17:35:27: [<ffffffff81064be0>] ? default_wake_function+0x0/0x20
17:35:27: [<ffffffffa08379e0>] ? obd_zombie_impexp_thread+0x0/0x190 [obdclass]
17:35:27: [<ffffffff8109e78e>] kthread+0x9e/0xc0
17:35:27: [<ffffffff8100c28a>] child_rip+0xa/0x20
17:35:27: [<ffffffff8109e6f0>] ? kthread+0x0/0xc0
17:35:27: [<ffffffff8100c280>] ? child_rip+0x0/0x20
17:35:27:
17:35:27:Kernel panic - not syncing: LBUG
17:35:27:Pid: 4041, comm: obd_zombid Tainted: P           ---------------    2.6.32-504.23.4.el6_lustre.g2382eb0.x86_64 #1
17:35:27:Call Trace:
17:35:27: [<ffffffff81529bbc>] ? panic+0xa7/0x16f
17:35:27: [<ffffffffa0709ecb>] ? lbug_with_loc+0x9b/0xb0 [libcfs]
17:35:27: [<ffffffffa0ada51c>] ? nrs_trr_hop_exit+0x1cc/0x1d0 [ptlrpc]
17:35:27: [<ffffffffa071bf46>] ? cfs_hash_putref+0x1a6/0x480 [libcfs]
17:35:27: [<ffffffffa0721dbb>] ? cfs_binheap_destroy+0x3db/0x810 [libcfs]
17:35:27: [<ffffffffa0adb61f>] ? nrs_orr_stop+0x6f/0x440 [ptlrpc]
17:35:27: [<ffffffffa0ad11af>] ? nrs_policy_stop0+0x4f/0x1f0 [ptlrpc]
17:35:27: [mory mce=off disable_cpu_apicid=0 memmap=exactmap memmap=627K@4K memmap=131449K@49779K elfcorehdr=181228K memmap=4K$0K memmap=9K$631K memmap=64K$960K memmap=12K$2097140K memmap=272K$4194032K
17:35:27:KERNEL supported cpus:

Issue Links

• duplicates: LU-16253 sanityn: ASSERTION( orro->oo_ref == 0 ) in 77d