[LU-3238] ASSERTION( (!(moving_req ? CFS_ALLOC_ATOMIC : CFS_ALLOC_IO != CFS_ALLOC_ATOMIC)

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.4.0
• Fix Version/s: Lustre 2.4.0
• Type: Bug
• Priority: Major
• Reporter: Minh Diep
• Assignee: Liang Zhen (Inactive)
• Resolution: Fixed

Environment

server and client run lustre-master build 1441

https://maloo.whamcloud.com/test_sessions/749d166a-aefd-11e2-901b-52540035b04c

• Severity: 3

Description

LustreError: 9518:0:(nrs_orr.c:888:nrs_orr_res_get()) ASSERTION( (Unable to render embedded object: File ( CFS_ALLOC_IO != CFS_ALLOC_ATOMIC) ) not found.(((current_thread_info()->preempt_count) & ((((1UL << (10))-1) << ((0 + 8) + 8)) | (((1UL << (8))-1) << (0 + 8)) | (((1UL << (1))-1) << (((0 + 8) + 8) + 10))))))) ) failed:
LustreError: 9518:0:(nrs_orr.c:888:nrs_orr_res_get()) LBUG
Kernel panic - not syncing: LBUG in interrupt.

Pid: 9518, comm: ll_ost02_041 Not tainted 2.6.32-358.2.1.el6_lustre.g53d4286.x86_64 #1
Call Trace:
[<ffffffff8150d648>] ? panic+0xa7/0x16f
[<ffffffffa044eedd>] ? lbug_with_loc+0x8d/0xb0 [libcfs]
[<ffffffffa0829472>] ? nrs_orr_res_get+0xab2/0xb80 [ptlrpc]
[<ffffffffa081f306>] ? nrs_resource_get+0x56/0x110 [ptlrpc]
[<ffffffffa081fccb>] ? nrs_resource_get_safe+0x8b/0x100 [ptlrpc]
[<ffffffffa0822308>] ? ptlrpc_nrs_req_hp_move+0x68/0x210 [ptlrpc]
[<ffffffffa07bc908>] ? ldlm_server_blocking_ast+0x228/0x880 [ptlrpc]
[<ffffffffa029f7e5>] ? lprocfs_counter_add+0x125/0x182 [lvfs]
[<ffffffffa0dd5683>] ? ost_blocking_ast+0x193/0x10f0 [ost]
[<ffffffffa0791e9d>] ? ldlm_work_bl_ast_lock+0xdd/0x290 [ptlrpc]
[<ffffffffa07d29ac>] ? ptlrpc_set_wait+0x6c/0x8c0 [ptlrpc]
[<ffffffffa044f805>] ? cfs_waitq_init+0x15/0x20 [libcfs]
[<ffffffffa07cf259>] ? ptlrpc_prep_set+0xf9/0x2e0 [ptlrpc]
[<ffffffffa0791dc0>] ? ldlm_work_bl_ast_lock+0x0/0x290 [ptlrpc]
[<ffffffffa079427b>] ? ldlm_run_ast_work+0x1db/0x490 [ptlrpc]
[<ffffffffa07abd5d>] ? ldlm_process_extent_lock+0x13d/0xa80 [ptlrpc]
[<ffffffffa07953f5>] ? ldlm_lock_enqueue+0x405/0x8d0 [ptlrpc]
[<ffffffffa07bb3f7>] ? ldlm_handle_enqueue0+0x4f7/0x10b0 [ptlrpc]
[<ffffffffa07bc016>] ? ldlm_handle_enqueue+0x66/0x70 [ptlrpc]
[<ffffffffa07bc020>] ? ldlm_server_completion_ast+0x0/0x6c0 [ptlrpc]
[<ffffffffa0dd54f0>] ? ost_blocking_ast+0x0/0x10f0 [ost]
[<ffffffffa07b8a30>] ? ldlm_server_glimpse_ast+0x0/0x3b0 [ptlrpc]
[<ffffffffa0dde498>] ? ost_handle+0x1e28/0x4790 [ost]
[<ffffffffa045b0e4>] ? libcfs_id2str+0x74/0xb0 [libcfs]
[<ffffffffa07ed5b8>] ? ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc]
[<ffffffffa044f5de>] ? cfs_timer_arm+0xe/0x10 [libcfs]
[<ffffffffa0460d8f>] ? lc_watchdog_touch+0x6f/0x170 [libcfs]
[<ffffffffa07e4919>] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc]
[<ffffffff81055ab3>] ? __wake_up+0x53/0x70
[<ffffffffa07ee93e>] ? ptlrpc_main+0xabe/0x16f0 [ptlrpc]
[<ffffffffa07ede80>] ? ptlrpc_main+0x0/0x16f0 [ptlrpc]
[<ffffffff8100c0ca>] ? child_rip+0xa/0x20
[<ffffffffa07ede80>] ? ptlrpc_main+0x0/0x16f0 [ptlrpc]
[<ffffffffa07ede80>] ? ptlrpc_main+0x0/0x16f0 [ptlrpc]
[<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Initializing cgroup subsys cpuset

To reproduce:
Run parallel-scale test while run below script on mds

#!/bin/bash

count=1
while true; do
        echo "`date`====== run $count ======"
        for a in fifo crrn orr trr; do
                echo ==================
                for i in mds02 oss02 oss03; do
                        echo "set nrs policy $a on $i"
                        pdsh -w $i "/usr/sbin/lctl set_param *.*.*.nrs_policies=$a"
                        sleep 5
                        echo ++++++++++++
                        echo "view nrs policies $a on $i"
                        pdsh -w $i "/usr/sbin/lctl get_param *.*.*.nrs_policies"
                done
        done
        sleep 300
        ((count=count+1))
done

Issue Links

• is related to: LU-398 NRS (Network Request Scheduler )

Comments

[Liang Zhen (Inactive)]

patch is here: http://review.whamcloud.com/#change,6190

[Peter Jones]

Landed for 2.4