[LU-12907] LNet routers: LNetError: 14141:0:(lib-msg.c:894:lnet_finalize()) ASSERTION( !(((current_thread_info()->preempt_count) & ((((1UL << (10))-1) << ((0 + 8) + 8)) | (((1UL << (8))-1) << (0 + 8)) | (((1UL << (1))-1) << (((0 + 8) + 8) + 10))))) - Whamcloud Community JIRA

Details

Type: Bug
Resolution: Duplicate
Priority: Major
Fix Version/s: None
Affects Version/s: Lustre 2.12.3
Labels:
None
Environment:
CentOS 7.6

Severity:
2
Rank (Obsolete):
9223372036854775807

Description

We have been upgrading our Lnet routers recently to 2.12.3 and all of them crashed simultaneously tonight with the following assertion:

[39140.467535] LNetError: 14141:0:(lib-msg.c:894:lnet_finalize()) ASSERTION( !(((current_thread_info()->preempt_count) & ((((1UL << (10))-1) << ((0 + 8) + 8)) | (((1UL << (8))-1) << (0 + 8)) | (((1UL << (1))-1) << (((0 + 8) + 8) + 10)))))
[39140.491917] general protection fault: 0000 [#1] SMP 
[39140.491969] Modules linked in: ko2iblnd(OE) lnet(OE) libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx5_ib(OE) mlx5_core(OE) mlxfw(OE) mlx4_en(OE) mlx4_ib(OE) ib_uverbsm
[39140.491977]  crct10dif_pclmul crct10dif_common tg3 libahci megaraid_sas ptp libata crc32c_intel pps_core [last unloaded: mlx_compat]
[39140.491982] CPU: 0 PID: 14141 Comm: kiblnd_connd Kdump: loaded Tainted: G           OE  ------------   3.10.0-957.27.2.el7.x86_64 #1
[39140.491983] Hardware name: Dell Inc. PowerEdge R630/02C2CP, BIOS 2.10.5 07/25/2019
[39140.491985] task: ffff90b918b1a080 ti: ffff90b8fa518000 task.ti: ffff90b8fa518000
[39140.491995] RIP: 0010:[<ffffffff886f3875>]  [<ffffffff886f3875>] cpuacct_charge+0x35/0x50
[39140.491997] RSP: 0018:ffff90b91c603dd0  EFLAGS: 00010006
[39140.491998] RAX: 18244c8948c18cb8 RBX: ffff90b918b1a0e8 RCX: 000000000000ffff
[39140.492000] RDX: ffffffff8925b640 RSI: 0000000001743e28 RDI: ffff90b918b1a080
[39140.492002] RBP: ffff90b91c603dd0 R08: ffffffffffffb820 R09: 000000000000040f
[39140.492003] R10: 0000000000000004 R11: 0000000000000005 R12: 0000000001743e28
[39140.492005] R13: ffff90b91c61ac00 R14: ffff90b918b1a080 R15: 0000000000000000
[39140.492008] FS:  0000000000000000(0000) GS:ffff90b91c600000(0000) knlGS:0000000000000000
[39140.492010] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[39140.492011] CR2: 00007fd46cd96248 CR3: 0000000154c10000 CR4: 00000000003607f0
[39140.492013] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[39140.492015] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[39140.492016] Call Trace:
[39140.492025]  <IRQ> 
[39140.492025]  [<ffffffff886e143c>] update_curr+0x14c/0x1e0
[39140.492029]  [<ffffffff886e295d>] task_tick_fair+0x2bd/0x660
[39140.492034]  [<ffffffff88634919>] ? sched_clock+0x9/0x10
[39140.492038]  [<ffffffff886db1f5>] ? sched_clock_cpu+0x85/0xc0
[39140.492041]  [<ffffffff886d60ad>] scheduler_tick+0xcd/0x150
[39140.492046]  [<ffffffff8870c160>] ? tick_sched_do_timer+0x50/0x50
[39140.492051]  [<ffffffff886ac3a5>] update_process_times+0x65/0x80
[39140.492055]  [<ffffffff8870bed0>] tick_sched_handle+0x30/0x70
[39140.492058]  [<ffffffff8870c199>] tick_sched_timer+0x39/0x80
[39140.492065]  [<ffffffff886c71e3>] __hrtimer_run_queues+0xf3/0x270
[39140.492069]  [<ffffffff886c776f>] hrtimer_interrupt+0xaf/0x1d0
[39140.492076]  [<ffffffff8865a61b>] local_apic_timer_interrupt+0x3b/0x60
[39140.492081]  [<ffffffff88d7b6e3>] smp_apic_timer_interrupt+0x43/0x60
[39140.492087]  [<ffffffff88d77df2>] apic_timer_interrupt+0x162/0x170
[39140.492111]  <EOI> 
[39140.492111]  [<ffffffffc0ac3f9d>] ? lnet_finalize+0x98d/0x9a0 [lnet]
[39140.492127]  [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
[39140.492156]  [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet]
[39140.492171]  [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet]
[39140.492184]  [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet]
[39140.492196]  [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet]
[39140.492206]  [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
[39140.492219]  [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet]
[39140.492232]  [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet]
[39140.492243]  [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet]
[39140.492254]  [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet]
[39140.492264]  [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
[39140.492276]  [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet]
[39140.492288]  [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet]
[39140.492299]  [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet]
[39140.492309]  [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet]
[39140.492319]  [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
[39140.492330]  [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet]
[39140.492341]  [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet]
[39140.492353]  [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet]
[39140.492363]  [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet]
[39140.492372]  [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
[39140.492383]  [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet]
[39140.492395]  [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet]
[39140.492406]  [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet]
[39140.492416]  [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet]
[39140.492425]  [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
[39140.492436]  [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet]
[39140.492447]  [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet]
[39140.492457]  [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet]
[39140.492468]  [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet]
[39140.492476]  [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
[39140.492487]  [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet]
[39140.492501]  [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet]
[39140.492512]  [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet]
[39140.492522]  [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet]
[39140.492530]  [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
[39140.492541]  [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet]
[39140.492552]  [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet]
[39140.492563]  [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet]
[39140.492573]  [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet]
[39140.492582]  [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
[39140.492592]  [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet]
[39140.492603]  [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet]
[39140.492614]  [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet]
[39140.492624]  [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet]
[39140.492632]  [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
[39140.492642]  [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet]
[39140.492653]  [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet]
[39140.492664]  [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet]
[39140.492674]  [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet]
[39140.492682]  [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
[39140.492693]  [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet]
[39140.492704]  [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet]
[39140.492714]  [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet]
[39140.492724]  [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet]
[39140.492732]  [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
[39140.492743]  [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet]
[39140.492753]  [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet]
[39140.492764]  [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet]
[39140.492774]  [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet]
[39140.492782]  [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
[39140.492792]  [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet]
[39140.492803]  [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet]
[39140.492813]  [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet]
[39140.492823]  [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet]
[39140.492831]  [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
[39140.492842]  [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet]
[39140.492853]  [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet]
[39140.492863]  [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet]
[39140.492873]  [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet]
[39140.492881]  [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
[39140.492892]  [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet]
[39140.492902]  [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet]
[39140.492913]  [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet]
[39140.492923]  [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet]
[39140.492931]  [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
[39140.492941]  [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet]
[39140.492952]  [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet]
[39140.492962]  [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet]
[39140.492972]  [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet]
[39140.492980]  [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
[39140.492991]  [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet]
[39140.493002]  [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet]
[39140.493012]  [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet]
[39140.493022]  [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet]
[39140.493030]  [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
[39140.493040]  [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet]
[39140.493051]  [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet]
[39140.493061]  [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet]
[39140.493071]  [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet]
[39140.493079]  [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
[39140.493089]  [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet]
[39140.493100]  [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet]
[39140.493111]  [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet]
[39140.493121]  [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet]
[39140.493128]  [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
[39140.493139]  [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet]
[39140.493150]  [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet]
[39140.493160]  [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet]
[39140.493170]  [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet]
[39140.493178]  [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
[39140.493188]  [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet]
[39140.493199]  [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet]
[39140.493209]  [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet]
[39140.493219]  [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet]
[39140.493227]  [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
[39140.493237]  [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet]
[39140.493248]  [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet]
[39140.493258]  [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet]
[39140.493268]  [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet]
[39140.493276]  [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
[39140.493287]  [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet]
[39140.493297]  [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet]
[39140.493308]  [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet]
[39140.493318]  [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet]
[39140.493325]  [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
[39140.493336]  [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet]
[39140.493347]  [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet]
[39140.493357]  [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet]
[39140.493367]  [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet]
[39140.493375]  [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
[39140.493385]  [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet]
[39140.493396]  [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet]
[39140.493406]  [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet]
[39140.493416]  [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet]
[39140.493424]  [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
[39140.493434]  [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet]
[39140.493445]  [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet]
[39140.493455]  [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet]
[39140.493466]  [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet]
[39140.493473]  [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
[39140.493484]  [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet]
[39140.493496]  [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet]
[39140.493508]  [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet]
[39140.493518]  [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet]
[39140.493525]  [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
[39140.493536]  [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet]
[39140.493547]  [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet]
[39140.493557]  [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet]
[39140.493567]  [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet]
[39140.493575]  [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
[39140.493585]  [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet]
[39140.493596]  [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet]
[39140.493606]  [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet]
[39140.493616]  [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet]
[39140.493624]  [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
[39140.493634]  [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet]
[39140.493645]  [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet]
[39140.493655]  [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet]
[39140.493665]  [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet]
[39140.493673]  [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
[39140.493684]  [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet]
[39140.493694]  [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet]
[39140.493704]  [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet]
[39140.493714]  [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet]
[39140.493722]  [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
[39140.493733]  [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet]
[39140.493743]  [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet]
[39140.493753]  [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet]
[39140.493763]  [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet]
[39140.493771]  [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
[39140.493782]  [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet]
[39140.493792]  [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet]
[39140.493802]  [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet]
[39140.493813]  [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet]
[39140.493820]  [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
[39140.493831]  [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet]
[39140.493842]  [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet]
[39140.493852]  [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet]
[39140.493862]  [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet]
[39140.493869]  [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
[39140.493880]  [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet]
[39140.493891]  [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet]
[39140.493901]  [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet]
[39140.493911]  [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet]
[39140.493918]  [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
[39140.493929]  [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet]
[39140.493940]  [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet]
[39140.493950]  [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet]
[39140.493960]  [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet]
[39140.493967]  [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
[39140.493978]  [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet]
[39140.493989]  [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet]
[39140.493999]  [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet]
[39140.494009]  [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet]
[39140.494017]  [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
[39140.494027]  [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet]
[39140.494038]  [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet]
[39140.494048]  [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet]
[39140.494058]  [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet]
[39140.494065]  [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
[39140.494076]  [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet]
[39140.494087]  [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet]
[39140.494097]  [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet]
[39140.494107]  [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet]
[39140.494114]  [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
[39140.494125]  [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet]
[39140.494136]  [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet]
[39140.494146]  [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet]
[39140.494156]  [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet]
[39140.494163]  [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
[39140.494174]  [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet]
[39140.494185]  [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet]
[39140.494195]  [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet]
[39140.494205]  [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet]
[39140.494212]  [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
[39140.494223]  [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet]
[39140.494233]  [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet]
[39140.494243]  [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet]
[39140.494253]  [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet]
[39140.494261]  [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
[39140.494271]  [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet]
[39140.494282]  [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet]
[39140.494292]  [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet]
[39140.494302]  [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet]
[39140.494310]  [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
[39140.494320]  [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet]
[39140.494332]  [<ffffffffc0ac0082>] ? libcfs_nid2str_r+0xe2/0x130 [lnet]
[39140.494343]  [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet]
[39140.494353]  [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet]
[39140.494363]  [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet]
[39140.494372]  [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs]
[39140.494382]  [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet]
[39140.494391]  [<ffffffffc090fae8>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs]
[39140.494402]  [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet]
[39140.494412]  [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet]
[39140.494423]  [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet]
[39140.494432]  [<ffffffffc0babd22>] ? kiblnd_pool_free_node+0x82/0x170 [ko2iblnd]
[39140.494440]  [<ffffffffc0bb561d>] ? kiblnd_tx_done+0x10d/0x3e0 [ko2iblnd]
[39140.494447]  [<ffffffffc0bb593b>] ? kiblnd_txlist_done+0x4b/0x60 [ko2iblnd]
[39140.494454]  [<ffffffffc0bbab83>] ? kiblnd_check_conns+0x553/0x880 [ko2iblnd]
[39140.494465]  [<ffffffffc09213ba>] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs]
[39140.494473]  [<ffffffffc0bbfc1b>] ? kiblnd_connd+0x83b/0xa00 [ko2iblnd]
[39140.494476]  [<ffffffff886d7c40>] ? wake_up_state+0x20/0x20
[39140.494484]  [<ffffffffc0bbf3e0>] ? kiblnd_cm_callback+0x2380/0x2380 [ko2iblnd]
[39140.494487]  [<ffffffff886c2e81>] ? kthread+0xd1/0xe0
[39140.494490]  [<ffffffff886c2db0>] ? insert_kthread_work+0x40/0x40
[39140.494495]  [<ffffffff88d76c37>] ? ret_from_fork_nospec_begin+0x21/0x21
[39140.494499]  [<ffffffff886c2db0>] ? insert_kthread_work+0x40/0x40
[39140.494536] Code: 48 89 e5 48 63 48 18 48 8b 87 40 09 00 00 48 8b 50 48 eb 0b 66 90 48 8b 50 68 48 85 d2 74 1b 48 8b 42 40 48 03 04 cd a0 bf 34 89 <48> 01 30 48 8b 02 48 8b 40 40 48 85 c0 75 dc 5d c3 66 2e 0f 1f 
[39140.494541] RIP  [<ffffffff886f3875>] cpuacct_charge+0x35/0x50
[39140.494541]  RSP <ffff90b91c603dd0>

[root@sh-rtr-fir-1-1 127.0.0.1-2019-10-25-21:01:51]# rpm -qa | grep lustre
lustre-client-2.12.3-1.el7.x86_64
lustre-client-dkms-2.12.3-1.el7.noarch
[root@sh-rtr-fir-1-1 127.0.0.1-2019-10-25-21:01:51]# uname -a
Linux sh-rtr-fir-1-1.int 3.10.0-957.27.2.el7.x86_64 #1 SMP Mon Jul 29 17:46:05 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

vmcore-dmesg-sh-rtr-fir-1-1.log
156 kB
26/Oct/19 4:23 AM

Issue Links

is related to

LU-12568 LNetError: 28086:0:(lib-move.c:2862:lnet_detach_rsp_tracker()) ASSERTION( rspt->rspt_cpt == cpt ) failed

Resolved

LU-12441 Response tracker is not detached on router ping reply

Resolved

Activity

[LU-12907] LNet routers: LNetError: 14141:0:(lib-msg.c:894:lnet_finalize()) ASSERTION( !(((current_thread_info()->preempt_count) & ((((1UL << (10))-1) << ((0 + 8) + 8)) | (((1UL << (8))-1) << (0 + 8)) | (((1UL << (1))-1) << (((0 + 8) + 8) + 10)))))

Peter Jones made changes - 07/Dec/19 2:59 PM

Link

Original: This issue is related to JFC-21 [ JFC-21 ]

Peter Jones made changes - 07/Dec/19 2:58 PM

Resolution		New: Duplicate [ 3 ]
Status	Original: Open [ 1 ]	New: Resolved [ 5 ]

Peter Jones made changes - 09/Nov/19 2:43 PM

Link

Original: This issue is related to LU-12241 [ LU-12241 ]

Peter Jones made changes - 09/Nov/19 2:43 PM

Link

New: This issue is related to ~~LU-12441~~ [ ~~LU-12441~~ ]

Peter Jones made changes - 09/Nov/19 2:42 PM

Link

New: This issue is related to LU-12241 [ LU-12241 ]

Peter Jones made changes - 09/Nov/19 2:42 PM

Link

New: This issue is related to ~~LU-12568~~ [ ~~LU-12568~~ ]

Peter Jones made changes - 08/Nov/19 9:51 PM

Link

New: This issue is related to JFC-21 [ JFC-21 ]

Stephane Thiell made changes - 04/Nov/19 7:53 PM

Comment

[ More info on some of the other clients found in that state (we can see that it's not always the same targets that are impacted). The rest of the nodes on Sherlock (1000+) are apparently fine.
* sh-31-09 {{10.8.31.9@o2ib6}}

{noformat}
[root@sh-31-09 ~]# lfs df -v /scratch
UUID 1K-blocks Used Available Use% Mounted on
fir-MDT0000_UUID 18287292984 8939092268 8411591900 52% /scratch[MDT:0] f
fir-MDT0001_UUID 18287292984 4277026904 13073799168 25% /scratch[MDT:1] f
fir-MDT0002_UUID 18287292984 7080879448 10270132756 41% /scratch[MDT:2] f
fir-MDT0003_UUID 18287292984 3658330152 13692645740 22% /scratch[MDT:3] f
fir-OST0000_UUID 61986877596 30949957496 30411312792 51% /scratch[OST:0]
fir-OST0001_UUID 61986877596 30215977912 31145205600 50% /scratch[OST:1]
fir-OST0002_UUID 61986877596 30709166112 30652406664 51% /scratch[OST:2]
fir-OST0003_UUID 61986877596 30355727296 31005500224 50% /scratch[OST:3]
fir-OST0004_UUID 61986877596 30509331704 30851976060 50% /scratch[OST:4]
fir-OST0005_UUID 61986877596 30557864356 30803397248 50% /scratch[OST:5]
fir-OST0006_UUID 61986877596 30077579320 31283927464 50% /scratch[OST:6]
fir-OST0007_UUID 61986877596 30750589336 30610882152 51% /scratch[OST:7]
fir-OST0008_UUID 61986877596 30497811536 30863485612 50% /scratch[OST:8]
fir-OST0009_UUID 61986877596 30652300488 30709035008 50% /scratch[OST:9]
fir-OST000a_UUID 61986877596 29670740024 31690554904 49% /scratch[OST:10]
OST000b : inactive device
fir-OST000c_UUID 61986877596 29597506152 31763777956 49% /scratch[OST:12]
OST000d : inactive device
fir-OST000e_UUID 61986877596 30748544848 30612822608 51% /scratch[OST:14]
fir-OST000f_UUID 61986877596 29817693992 31543549068 49% /scratch[OST:15]
fir-OST0010_UUID 61986877596 30348718584 31012342584 50% /scratch[OST:16]
fir-OST0011_UUID 61986877596 30520160012 30841155676 50% /scratch[OST:17]
fir-OST0012_UUID 61986877596 30373509384 30987808224 50% /scratch[OST:18]
fir-OST0013_UUID 61986877596 30322939000 31038011344 50% /scratch[OST:19]
fir-OST0014_UUID 61986877596 30114752004 31246434104 50% /scratch[OST:20]
fir-OST0015_UUID 61986877596 31000512848 30360742432 51% /scratch[OST:21]
fir-OST0016_UUID 61986877596 30500010868 30861224276 50% /scratch[OST:22]
OST0017 : inactive device
fir-OST0018_UUID 61986877596 30472950396 30888381476 50% /scratch[OST:24]
fir-OST0019_UUID 61986877596 30111104796 31250328840 50% /scratch[OST:25]
fir-OST001a_UUID 61986877596 30363747204 30997493896 50% /scratch[OST:26]
fir-OST001b_UUID 61986877596 29733122628 31628229248 49% /scratch[OST:27]
OST001c : inactive device
fir-OST001d_UUID 61986877596 30963470920 30397754836 51% /scratch[OST:29]
OST001e : inactive device
fir-OST001f_UUID 61986877596 30339614880 31021547564 50% /scratch[OST:31]
fir-OST0020_UUID 61986877596 30931807172 30429259448 51% /scratch[OST:32]
fir-OST0021_UUID 61986877596 31373213420 29987923052 52% /scratch[OST:33]
fir-OST0022_UUID 61986877596 30770502444 30590982164 51% /scratch[OST:34]
fir-OST0023_UUID 61986877596 30702780792 30658609016 51% /scratch[OST:35]
fir-OST0024_UUID 61986877596 30586817044 30774501748 50% /scratch[OST:36]
fir-OST0025_UUID 61986877596 30231628608 31129708148 50% /scratch[OST:37]
fir-OST0026_UUID 61986877596 30277966200 31083417480 50% /scratch[OST:38]
fir-OST0027_UUID 61986877596 29900758936 31460310460 49% /scratch[OST:39]
fir-OST0028_UUID 61986877596 30454670188 30906596984 50% /scratch[OST:40]
fir-OST0029_UUID 61986877596 30640732804 30720696176 50% /scratch[OST:41]
fir-OST002a_UUID 61986877596 30465828080 30895358200 50% /scratch[OST:42]
fir-OST002b_UUID 61986877596 30265299116 31095790364 50% /scratch[OST:43]
fir-OST002c_UUID 61986877596 31641994748 29719055956 52% /scratch[OST:44]
fir-OST002d_UUID 61986877596 30225079240 31136143492 50% /scratch[OST:45]
fir-OST002e_UUID 61986877596 29676602976 31684845376 49% /scratch[OST:46]
fir-OST002f_UUID 61986877596 30111759352 31249700184 50% /scratch[OST:47]
^C
{noformat}

* sh-07-13 {{10.8.7.13@o2ib6}}

{noformat}
[root@sh-07-13 ~]# lfs df -v /scratch
UUID 1K-blocks Used Available Use% Mounted on
fir-MDT0000_UUID 18287292984 8939631940 8411027920 52% /scratch[MDT:0] f
MDT0001 : inactive device
fir-MDT0002_UUID 18287292984 7080858324 10270158788 41% /scratch[MDT:2] f
fir-MDT0003_UUID 18287292984 3658379376 13692600816 22% /scratch[MDT:3] f
fir-OST0000_UUID 61986877596 30950524024 30410748772 51% /scratch[OST:0]
fir-OST0001_UUID 61986877596 30216446884 31144824708 50% /scratch[OST:1]
fir-OST0002_UUID 61986877596 30708442648 30652995932 51% /scratch[OST:2]
fir-OST0003_UUID 61986877596 30358659584 31002689780 50% /scratch[OST:3]
fir-OST0004_UUID 61986877596 30510265848 30851049420 50% /scratch[OST:4]
fir-OST0005_UUID 61986877596 30558835640 30802448684 50% /scratch[OST:5]
fir-OST0006_UUID 61986877596 30077941048 31283596728 50% /scratch[OST:6]
fir-OST0007_UUID 61986877596 30751074652 30610370076 51% /scratch[OST:7]
fir-OST0008_UUID 61986877596 30498183648 30863067520 50% /scratch[OST:8]
fir-OST0009_UUID 61986877596 30652427468 30708879776 50% /scratch[OST:9]
fir-OST000a_UUID 61986877596 29671105172 31690193220 49% /scratch[OST:10]
fir-OST000b_UUID 61986877596 29838310432 31522943840 49% /scratch[OST:11]
fir-OST000c_UUID 61986877596 29597981568 31763376016 49% /scratch[OST:12]
fir-OST000d_UUID 61986877596 30086491724 31274838296 50% /scratch[OST:13]
fir-OST000e_UUID 61986877596 30749480900 30611916020 51% /scratch[OST:14]
fir-OST000f_UUID 61986877596 29818145704 31542983876 49% /scratch[OST:15]
fir-OST0010_UUID 61986877596 30349236812 31011795604 50% /scratch[OST:16]
fir-OST0011_UUID 61986877596 30520504868 30840821676 50% /scratch[OST:17]
fir-OST0012_UUID 61986877596 30373853832 30987511120 50% /scratch[OST:18]
fir-OST0013_UUID 61986877596 30325330096 31035949288 50% /scratch[OST:19]
fir-OST0014_UUID 61986877596 30114932684 31246391948 50% /scratch[OST:20]
fir-OST0015_UUID 61986877596 31000968752 30360330336 51% /scratch[OST:21]
fir-OST0016_UUID 61986877596 30500671836 30860592848 50% /scratch[OST:22]
fir-OST0017_UUID 61986877596 30038865812 31322727020 49% /scratch[OST:23]
fir-OST0018_UUID 61986877596 30473121860 30888170896 50% /scratch[OST:24]
fir-OST0019_UUID 61986877596 30081927088 31279522992 50% /scratch[OST:25]
fir-OST001a_UUID 61986877596 30364483800 30996714244 50% /scratch[OST:26]
fir-OST001b_UUID 61986877596 29733478820 31627825708 49% /scratch[OST:27]
fir-OST001c_UUID 61986877596 31203094848 30158089048 51% /scratch[OST:28]
fir-OST001d_UUID 61986877596 30964160192 30397130396 51% /scratch[OST:29]
fir-OST001e_UUID 61986877596 29980976316 31380369756 49% /scratch[OST:30]
fir-OST001f_UUID 61986877596 30339848284 31021319936 50% /scratch[OST:31]
fir-OST0020_UUID 61986877596 30932449068 30428590864 51% /scratch[OST:32]
fir-OST0021_UUID 61986877596 31373885068 29987384752 52% /scratch[OST:33]
^C
{noformat}
* sh-103-33 {{10.9.103.33@o2ib4}} with inactive MDT0

{noformat}
[root@sh-103-33 ~]# lfs df -v /scratch
error: invalid path '/scratch': Cannot send after transport endpoint shutdown
[root@sh-103-33 ~]# lctl dl | grep fir
  1 UP lov fir-clilov-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 3
  2 UP lmv fir-clilmv-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
  3 IN mdc fir-MDT0000-mdc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
  4 UP mdc fir-MDT0001-mdc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
  5 UP mdc fir-MDT0002-mdc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
  6 UP mdc fir-MDT0003-mdc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
  7 UP osc fir-OST0000-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
  8 UP osc fir-OST0001-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
  9 UP osc fir-OST0002-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
10 IN osc fir-OST0003-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
11 UP osc fir-OST0004-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
12 UP osc fir-OST0005-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
13 UP osc fir-OST0006-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
14 UP osc fir-OST0007-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
15 UP osc fir-OST0008-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
16 UP osc fir-OST0009-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
17 UP osc fir-OST000a-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
18 UP osc fir-OST000b-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
19 UP osc fir-OST000c-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
20 UP osc fir-OST000d-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
21 UP osc fir-OST000e-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
22 UP osc fir-OST000f-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
23 UP osc fir-OST0010-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
24 UP osc fir-OST0011-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
25 UP osc fir-OST0012-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
26 UP osc fir-OST0013-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
27 UP osc fir-OST0014-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
28 UP osc fir-OST0015-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
29 UP osc fir-OST0016-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
30 UP osc fir-OST0017-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
31 UP osc fir-OST0018-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
32 UP osc fir-OST0019-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
33 UP osc fir-OST001a-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
34 UP osc fir-OST001b-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
35 IN osc fir-OST001c-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
36 UP osc fir-OST001d-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
37 UP osc fir-OST001e-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
38 UP osc fir-OST001f-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
39 UP osc fir-OST0020-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
40 IN osc fir-OST0021-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
41 UP osc fir-OST0022-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
42 UP osc fir-OST0023-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
43 UP osc fir-OST0024-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
44 UP osc fir-OST0025-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
45 UP osc fir-OST0026-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
46 UP osc fir-OST0027-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
47 IN osc fir-OST0028-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
48 UP osc fir-OST0029-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
49 UP osc fir-OST002a-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
50 UP osc fir-OST002b-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
51 UP osc fir-OST002c-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
52 UP osc fir-OST002d-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
53 UP osc fir-OST002e-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
54 UP osc fir-OST002f-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
55 UP osc fir-OST0030-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
56 UP osc fir-OST0031-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
57 UP osc fir-OST0032-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
58 UP osc fir-OST0033-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
59 IN osc fir-OST0034-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
60 UP osc fir-OST0035-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
61 IN osc fir-OST0036-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
62 UP osc fir-OST0037-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
63 UP osc fir-OST0038-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
64 UP osc fir-OST0039-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
65 UP osc fir-OST003a-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
66 IN osc fir-OST003b-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
67 UP osc fir-OST003c-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
68 IN osc fir-OST003d-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
69 UP osc fir-OST003e-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
70 IN osc fir-OST003f-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
71 UP osc fir-OST0040-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
72 UP osc fir-OST0041-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
73 UP osc fir-OST0042-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
74 UP osc fir-OST0043-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
75 UP osc fir-OST0044-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
76 UP osc fir-OST0045-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
77 UP osc fir-OST0046-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
78 IN osc fir-OST0047-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
79 UP osc fir-OST0048-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
80 UP osc fir-OST0049-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
81 UP osc fir-OST004a-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
82 UP osc fir-OST004b-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
83 IN osc fir-OST004c-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
84 UP osc fir-OST004d-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
85 UP osc fir-OST004e-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
86 IN osc fir-OST004f-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
87 IN osc fir-OST0050-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
88 UP osc fir-OST0051-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
89 UP osc fir-OST0052-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
90 IN osc fir-OST0053-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
91 UP osc fir-OST0054-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
92 UP osc fir-OST0055-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
93 UP osc fir-OST0056-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
94 UP osc fir-OST0057-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
95 UP osc fir-OST0058-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
96 IN osc fir-OST0059-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
97 IN osc fir-OST005a-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
98 IN osc fir-OST005b-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
99 UP osc fir-OST005c-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
100 UP osc fir-OST005d-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
101 UP osc fir-OST005e-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
102 IN osc fir-OST005f-osc-ffff9b9fe1a9e800 535246f0-9862-c25f-7814-3ac6259d8dbc 4
{noformat}
* sh-103-68 {{10.9.103.68@o2ib4}}

{noformat}
[root@sh-103-68 ~]# lfs df -v /scratch
error: invalid path '/scratch': Cannot send after transport endpoint shutdown
[root@sh-103-68 ~]# lctl dl | grep fir
  1 UP lov fir-clilov-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 3
  2 UP lmv fir-clilmv-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
  3 IN mdc fir-MDT0000-mdc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
  4 UP mdc fir-MDT0001-mdc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
  5 UP mdc fir-MDT0002-mdc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
  6 UP mdc fir-MDT0003-mdc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
  7 UP osc fir-OST0000-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
  8 IN osc fir-OST0001-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
  9 IN osc fir-OST0002-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
10 UP osc fir-OST0003-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
11 UP osc fir-OST0004-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
12 UP osc fir-OST0005-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
13 UP osc fir-OST0006-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
14 UP osc fir-OST0007-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
15 UP osc fir-OST0008-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
16 UP osc fir-OST0009-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
17 UP osc fir-OST000a-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
18 IN osc fir-OST000b-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
19 IN osc fir-OST000c-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
20 UP osc fir-OST000d-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
21 IN osc fir-OST000e-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
22 UP osc fir-OST000f-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
23 UP osc fir-OST0010-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
24 UP osc fir-OST0011-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
25 UP osc fir-OST0012-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
26 UP osc fir-OST0013-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
27 UP osc fir-OST0014-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
28 UP osc fir-OST0015-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
29 UP osc fir-OST0016-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
30 UP osc fir-OST0017-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
31 UP osc fir-OST0018-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
32 UP osc fir-OST0019-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
33 IN osc fir-OST001a-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
34 IN osc fir-OST001b-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
35 UP osc fir-OST001c-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
36 IN osc fir-OST001d-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
37 UP osc fir-OST001e-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
38 UP osc fir-OST001f-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
39 UP osc fir-OST0020-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
40 UP osc fir-OST0021-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
41 IN osc fir-OST0022-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
42 UP osc fir-OST0023-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
43 UP osc fir-OST0024-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
44 IN osc fir-OST0025-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
45 UP osc fir-OST0026-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
46 UP osc fir-OST0027-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
47 UP osc fir-OST0028-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
48 UP osc fir-OST0029-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
49 UP osc fir-OST002a-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
50 UP osc fir-OST002b-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
51 UP osc fir-OST002c-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
52 UP osc fir-OST002d-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
53 UP osc fir-OST002e-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
54 IN osc fir-OST002f-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
55 IN osc fir-OST0030-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
56 UP osc fir-OST0031-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
57 UP osc fir-OST0032-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
58 UP osc fir-OST0033-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
59 UP osc fir-OST0034-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
60 IN osc fir-OST0035-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
61 UP osc fir-OST0036-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
62 IN osc fir-OST0037-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
63 IN osc fir-OST0038-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
64 UP osc fir-OST0039-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
65 UP osc fir-OST003a-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
66 UP osc fir-OST003b-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
67 IN osc fir-OST003c-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
68 IN osc fir-OST003d-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
69 UP osc fir-OST003e-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
70 UP osc fir-OST003f-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
71 IN osc fir-OST0040-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
72 UP osc fir-OST0041-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
73 UP osc fir-OST0042-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
74 UP osc fir-OST0043-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
75 IN osc fir-OST0044-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
76 UP osc fir-OST0045-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
77 UP osc fir-OST0046-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
78 UP osc fir-OST0047-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
79 UP osc fir-OST0048-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
80 UP osc fir-OST0049-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
81 IN osc fir-OST004a-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
82 IN osc fir-OST004b-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
83 UP osc fir-OST004c-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
84 IN osc fir-OST004d-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
85 UP osc fir-OST004e-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
86 UP osc fir-OST004f-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
87 IN osc fir-OST0050-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
88 IN osc fir-OST0051-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
89 UP osc fir-OST0052-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
90 UP osc fir-OST0053-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
91 UP osc fir-OST0054-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
92 UP osc fir-OST0055-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
93 IN osc fir-OST0056-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
94 UP osc fir-OST0057-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
95 UP osc fir-OST0058-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
96 UP osc fir-OST0059-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
97 IN osc fir-OST005a-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
98 UP osc fir-OST005b-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
99 UP osc fir-OST005c-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
100 UP osc fir-OST005d-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
101 UP osc fir-OST005e-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
102 UP osc fir-OST005f-osc-ffff916185740800 0793d310-98b7-3c69-61fc-4fc3ff0f31a5 4
{noformat} ]

Peter Jones made changes - 26/Oct/19 2:36 PM

Assignee

Original: WC Triage [ wc-triage ]

New: Amir Shehata [ ashehata ]

Stephane Thiell created issue - 26/Oct/19 4:24 AM

People

Assignee:: Amir Shehata (Inactive)

Reporter:: Stephane Thiell

Votes:: 0 Vote for this issue

Watchers:: 4 Start watching this issue

Dates

Created:: 26/Oct/19 4:24 AM

Updated:: 07/Dec/19 7:13 PM

Resolved:: 07/Dec/19 2:58 PM