https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4499 Lustre <p>Hit LBUG and crach on OSS during IOR SSF(Single shared file) test with striping setting on all OSTs(lfs setstripe -c -1).</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>&lt;0&gt;LustreError: 28757:0:(hash.c:1252:cfs_hash_find_or_add()) ASSERTION( hlist_unhashed(hnode) ) failed: &lt;0&gt;LustreError: 28757:0:(hash.c:1252:cfs_hash_find_or_add()) LBUG &lt;0&gt;Kernel panic - not syncing: LBUG in interrupt. &lt;0&gt; &lt;4&gt;Pid: 28757, comm: ll_ost01_008 Not tainted 2.6.32-358.23.2.el6_lustre.ge975b1c.x86_64 #1 &lt;4&gt;Call Trace: &lt;4&gt; [&lt;ffffffff8150deec&gt;] ? panic+0xa7/0x16f &lt;4&gt; [&lt;ffffffffa065aedd&gt;] ? lbug_with_loc+0x8d/0xb0 [libcfs] &lt;4&gt; [&lt;ffffffffa0672d80&gt;] ? cfs_hash_findadd_unique+0x0/0x30 [libcfs] &lt;4&gt; [&lt;ffffffffa0672d98&gt;] ? cfs_hash_findadd_unique+0x18/0x30 [libcfs] &lt;4&gt; [&lt;ffffffffa0c83c76&gt;] ? nrs_orr_res_get+0x696/0xb90 [ptlrpc] &lt;4&gt; [&lt;ffffffff81055ad3&gt;] ? __wake_up+0x53/0x70 &lt;4&gt; [&lt;ffffffffa0c79e36&gt;] ? nrs_resource_get+0x56/0x110 [ptlrpc] &lt;4&gt; [&lt;ffffffffa0c37d95&gt;] ? lustre_msg_buf+0x55/0x60 [ptlrpc] &lt;4&gt; [&lt;ffffffffa0c7a7fb&gt;] ? nrs_resource_get_safe+0x8b/0x100 [ptlrpc] &lt;4&gt; [&lt;ffffffffa0c7ce38&gt;] ? ptlrpc_nrs_req_hp_move+0x68/0x210 [ptlrpc] &lt;4&gt; [&lt;ffffffffa0c5f845&gt;] ? req_capsule_client_get+0x15/0x20 [ptlrpc] &lt;4&gt; [&lt;ffffffffa0c1a158&gt;] ? ldlm_server_blocking_ast+0x228/0x880 [ptlrpc] &lt;4&gt; [&lt;ffffffffa0c8f65b&gt;] ? tgt_blocking_ast+0x7b/0x5e0 [ptlrpc] &lt;4&gt; [&lt;ffffffffa0beb1ba&gt;] ? ldlm_add_bl_work_item+0x8a/0x1e0 [ptlrpc] &lt;4&gt; [&lt;ffffffffa0bee405&gt;] ? ldlm_add_ast_work_item+0x55/0x180 [ptlrpc] &lt;4&gt; [&lt;ffffffffa0bed38d&gt;] ? ldlm_work_bl_ast_lock+0xdd/0x290 [ptlrpc] &lt;4&gt; [&lt;ffffffffa0c2e3bc&gt;] ? ptlrpc_set_wait+0x6c/0x860 [ptlrpc] &lt;4&gt; [&lt;ffffffff811685ac&gt;] ? __kmalloc+0x20c/0x220 &lt;4&gt; [&lt;ffffffffa0c2b06a&gt;] ? ptlrpc_prep_set+0xfa/0x2f0 [ptlrpc] &lt;4&gt; [&lt;ffffffffa0bed2b0&gt;] ? ldlm_work_bl_ast_lock+0x0/0x290 [ptlrpc] &lt;4&gt; [&lt;ffffffffa0bf006b&gt;] ? ldlm_run_ast_work+0x1bb/0x470 [ptlrpc] &lt;4&gt; [&lt;ffffffffa0c070ad&gt;] ? ldlm_process_extent_lock+0x13d/0xa90 [ptlrpc] &lt;4&gt; [&lt;ffffffffa0bef5ab&gt;] ? ldlm_lock_enqueue+0x3fb/0x920 [ptlrpc] &lt;4&gt; [&lt;ffffffffa0c18c4f&gt;] ? ldlm_handle_enqueue0+0x4ef/0x10a0 [ptlrpc] &lt;4&gt; [&lt;ffffffffa0c92562&gt;] ? tgt_enqueue+0x62/0x1d0 [ptlrpc] &lt;4&gt; [&lt;ffffffffa0c94f5a&gt;] ? tgt_handle_request0+0x2ea/0x1490 [ptlrpc] &lt;4&gt; [&lt;ffffffffa065b4ce&gt;] ? cfs_timer_arm+0xe/0x10 [libcfs] &lt;4&gt; [&lt;ffffffffa066c3af&gt;] ? lc_watchdog_touch+0x6f/0x170 [libcfs] &lt;4&gt; [&lt;ffffffffa0c3792c&gt;] ? lustre_msg_get_opc+0x9c/0x110 [ptlrpc] &lt;4&gt; [&lt;ffffffffa0c9653a&gt;] ? tgt_request_handle+0x43a/0x980 [ptlrpc] &lt;4&gt; [&lt;ffffffffa0c4a295&gt;] ? ptlrpc_main+0xd25/0x1970 [ptlrpc] &lt;4&gt; [&lt;ffffffff810096f0&gt;] ? __switch_to+0xd0/0x320 &lt;4&gt; [&lt;ffffffff8150e600&gt;] ? thread_return+0x4e/0x76e &lt;4&gt; [&lt;ffffffffa0c49570&gt;] ? ptlrpc_main+0x0/0x1970 [ptlrpc] &lt;4&gt; [&lt;ffffffff81096a36&gt;] ? kthread+0x96/0xa0 &lt;4&gt; [&lt;ffffffff8100c0ca&gt;] ? child_rip+0xa/0x20 &lt;4&gt; [&lt;ffffffff810969a0&gt;] ? kthread+0x0/0xa0 &lt;4&gt; [&lt;ffffffff8100c0c0&gt;] ? child_rip+0x0/0x20 </pre> </div></div> Lustre-2.5.52 (server), Lustre-2.5.53(Client) LU-4499

NRS ORR cfs_hash_find_or_add() LBUG

Bug Critical Resolved Fixed Jinshan Xiong Shuichi Ihara cea Thu, 16 Jan 2014 16:32:59 +0000 Tue, 19 Mar 2019 15:45:28 +0000 Wed, 26 Aug 2015 17:40:05 +0000 Lustre 2.5.0 Lustre 2.6.0 Lustre 2.8.0 Lustre 2.8.0 0 21 <p>ORR(Object-Based Round Robin) is turned on as NRS policy in this testing.</p> <p>Lai</p> <p>could you please help with this one?</p> <p>Thanks</p> <p>Peter</p> <p>This doesn't look likely to happen, because though orro-&gt;oo_hnode is not initialise before use, it's zeroed at allocation time by default. Anyway I composed a patch to initialise it, could you patch it and test on your system?</p> <p>Patch is on <a href="http://review.whamcloud.com/#/c/9046/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/9046/</a></p> <p>patch <a href="http://review.whamcloud.com/#/c/9046/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/9046/</a> doesn't help. We hit another crach here. <br/> Just chnage NRS policy to ORR(lctl set_param ost.OSS.ost_io.nrs_policies=orr), run IOR from client, then hit crach on OSS.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>&lt;4&gt;------------[ cut here ]------------ &lt;2&gt;kernel BUG at mm/slab.c:2835! &lt;4&gt;invalid opcode: 0000 [#1] SMP &lt;4&gt;last sysfs file: /sys/devices/pci0000:00/0000:00:03.0/0000:05:00.0/host7/target7:0:0/7:0:0:2/state &lt;4&gt;CPU 3 &lt;4&gt;Modules linked in: osp(U) ofd(U) ost(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) lquota(U) jbd2 mdd(U) fid(U) fld(U) ptlrpc(U) ko2iblnd(U) obdc lass(U) lnet(U) lvfs(U) sha512_generic sha256_generic crc32c_intel libcfs(U) ib_srp(U) scsi_transport_srp(U) bridge stp llc ipmi_devintf dell_rbu nfs lockd fscache auth_rpcgss nfs_acl sunrpc rdma_ucm(U) ib_ucm(U) rdma_cm(U) iw_cm(U) ib_ipoib(U) ib_cm(U) ib_uverbs(U) ib_umad(U) mlx5_ib(U) mlx5_core(U) mlx4_en(U) mlx4_ib(U) ib_sa(U) ib_mad(U) ib_core(U) ib_addr(U) ipv6 mlx4_core(U) compat(U) dm_round_robin dm_multipath vhost_net macvtap macvlan tun kvm knem(U) power _meter ses enclosure sg shpchp tg3 dcdbas microcode iTCO_wdt iTCO_vendor_support ext3 jbd mbcache sr_mod cdrom sd_mod crc_t10dif ahci wmi megaraid_sas dm_mi rror dm_region_hash dm_log dm_mod [last unloaded: speedstep_lib] &lt;4&gt; &lt;4&gt;Pid: 1868, comm: ll_ost_io01_002 Not tainted 2.6.32-358.23.2.el6_lustre.x86_64 #1 Dell Inc. PowerEdge R620/01W23F &lt;4&gt;RIP: 0010:[&lt;ffffffff81167473&gt;] [&lt;ffffffff81167473&gt;] cache_grow+0x313/0x320 &lt;4&gt;RSP: 0018:ffff881fdf96fc10 EFLAGS: 00010002 &lt;4&gt;RAX: ffff883f50d87c80 RBX: ffff881f879c1b80 RCX: 0000000000000000 &lt;4&gt;RDX: 0000000000000001 RSI: 0000000000041212 RDI: ffff881f879c1b80 &lt;4&gt;RBP: ffff881fdf96fc70 R08: 0000000000000000 R09: 000000000000dee6 &lt;4&gt;R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000041212 &lt;4&gt;R13: ffff883f50d87c40 R14: 0000000000000010 R15: 0000000000000000 &lt;4&gt;FS: 00007f7bccdf8700(0000) GS:ffff8820f0c20000(0000) knlGS:0000000000000000 &lt;4&gt;CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b &lt;4&gt;CR2: 0000000000481000 CR3: 00000040517c6000 CR4: 00000000001407e0 &lt;4&gt;DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 &lt;4&gt;DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 &lt;4&gt;Process ll_ost_io01_002 (pid: 1868, threadinfo ffff881fdf96e000, task ffff88203554e040) &lt;4&gt;Stack: &lt;4&gt; 00000000533d4823 0000000000000000 0000000000800000 0800000000000000 &lt;4&gt;&lt;d&gt; 0000000000000000 0000000000002000 000000000000127e ffff881f879c1b80 &lt;4&gt;&lt;d&gt; ffff883fd9162800 ffff883f50d87c40 0000000000000010 ffff883f50d87c60 &lt;4&gt;Call Trace: &lt;4&gt; [&lt;ffffffff81167682&gt;] cache_alloc_refill+0x202/0x240 &lt;4&gt; [&lt;ffffffff8116714e&gt;] kmem_cache_alloc_node+0x1be/0x1d0 &lt;4&gt; [&lt;ffffffffa049e8a1&gt;] cfs_mem_cache_cpt_alloc+0x41/0x50 [libcfs] &lt;4&gt; [&lt;ffffffffa09ee769&gt;] nrs_orr_res_get+0x5d9/0xba0 [ptlrpc] &lt;4&gt; [&lt;ffffffffa09e4b56&gt;] nrs_resource_get+0x56/0x110 [ptlrpc] &lt;4&gt; [&lt;ffffffffa09a5860&gt;] ? lustre_swab_niobuf_remote+0x0/0x30 [ptlrpc] &lt;4&gt; [&lt;ffffffffa09e551b&gt;] nrs_resource_get_safe+0x8b/0x100 [ptlrpc] &lt;4&gt; [&lt;ffffffffa09e7a48&gt;] ptlrpc_nrs_req_initialize+0x38/0x90 [ptlrpc] &lt;4&gt; [&lt;ffffffffa09b4e00&gt;] ptlrpc_main+0x1180/0x1700 [ptlrpc] &lt;4&gt; [&lt;ffffffffa09b3c80&gt;] ? ptlrpc_main+0x0/0x1700 [ptlrpc] &lt;4&gt; [&lt;ffffffff8100c0ca&gt;] child_rip+0xa/0x20 &lt;4&gt; [&lt;ffffffffa09b3c80&gt;] ? ptlrpc_main+0x0/0x1700 [ptlrpc] &lt;4&gt; [&lt;ffffffffa09b3c80&gt;] ? ptlrpc_main+0x0/0x1700 [ptlrpc] &lt;4&gt; [&lt;ffffffff8100c0c0&gt;] ? child_rip+0x0/0x20 &lt;4&gt;Code: 0f 1f 84 00 00 00 00 00 49 8d 54 24 30 48 c7 c0 fc ff ff ff 48 89 55 c8 e9 e1 fe ff ff 0f 0b eb fe ba 01 00 00 00 e9 2a fe ff ff &lt;0f&gt; 0b eb fe 66 0 f 1f 84 00 00 00 00 00 55 48 89 e5 41 57 41 56 &lt;1&gt;RIP [&lt;ffffffff81167473&gt;] cache_grow+0x313/0x320 &lt;4&gt; RSP &lt;ffff881fdf96fc10&gt; </pre> </div></div> <p>This time it LBUG in kernel mm/slab.c:2835</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">BUG_ON(flags &amp; GFP_SLAB_BUG_MASK) </pre> </div></div> <p>I don't see how it can happen yet, because it seems in entry of kmem_cache_alloc_node() has adjusted "flags" with "gfp_allowed_mask", and the only possible flags lustre code will set is GFP_ATOMIC, GFP_NOFS and __GFP_ZERO which are all valid.</p> <p>I will do more test to find out the cause.</p> <p>This may be a duplicate of <a href="https://jira.whamcloud.com/browse/LU-4362" title="cfs_hash_rehash_key() passed wrong parameters to cfs_hash_keycpy" class="issue-link" data-issue-key="LU-4362"><del>LU-4362</del></a>, could you verify <a href="http://review.whamcloud.com/#/c/8509/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/8509/</a> is included in your branch? if not, you may apply it and test again.</p> <p>In the latest version of master, I'm seeing this LBUG in sanityn test 77c. Is this related to <a href="https://jira.whamcloud.com/browse/LU-6558" title="replay-single: test_61c, test_90 timeout: nrs_orr_res_get() accessed NULL pointer" class="issue-link" data-issue-key="LU-6558"><del>LU-6558</del></a>?</p> <p>Here are some recent test sessions that have failed with this LBUG:<br/> review-zfs - <a href="https://testing.hpdd.intel.com/test_sets/63ca7db8-f830-11e4-a933-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/63ca7db8-f830-11e4-a933-5254006e85c2</a><br/> review-zfs - <a href="https://testing.hpdd.intel.com/test_sets/2d786c4c-f8b8-11e4-bb24-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/2d786c4c-f8b8-11e4-bb24-5254006e85c2</a></p> <p>hit again at: <a href="https://testing.hpdd.intel.com/test_logs/05f59e66-f989-11e4-939f-5254006e85c2/show_text" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_logs/05f59e66-f989-11e4-939f-5254006e85c2/show_text</a></p> <p>with the following backtrace:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00:49:01:LustreError: 12123:0:(hash.c:1256:cfs_hash_find_or_add()) ASSERTION( hlist_unhashed(hnode) ) failed: 00:49:01:LustreError: 12123:0:(hash.c:1256:cfs_hash_find_or_add()) LBUG 00:49:01:Kernel panic - not syncing: LBUG in interrupt. 00:49:01: 00:49:01:Pid: 12123, comm: ll_ost00_004 Tainted: P --------------- 2.6.32-504.16.2.el6_lustre.gd805a88.x86_64 #1 00:49:01:Call Trace: 00:49:01: [&lt;ffffffff81529fbc&gt;] ? panic+0xa7/0x16f 00:49:01: [&lt;ffffffffa0709ebd&gt;] ? lbug_with_loc+0x8d/0xb0 [libcfs] 00:49:01: [&lt;ffffffffa071df20&gt;] ? cfs_hash_findadd_unique+0x0/0x30 [libcfs] 00:49:01: [&lt;ffffffffa071df38&gt;] ? cfs_hash_findadd_unique+0x18/0x30 [libcfs] 00:49:01: [&lt;ffffffffa0aceb4b&gt;] ? nrs_orr_res_get+0x43b/0xc30 [ptlrpc] 00:49:01: [&lt;ffffffffa0ac4fb6&gt;] ? nrs_resource_get+0x56/0x110 [ptlrpc] 00:49:01: [&lt;ffffffffa0a83665&gt;] ? lustre_msg_buf+0x55/0x60 [ptlrpc] 00:49:01: [&lt;ffffffffa0ac597b&gt;] ? nrs_resource_get_safe+0x8b/0x100 [ptlrpc] 00:49:01: [&lt;ffffffffa0ac807b&gt;] ? ptlrpc_nrs_req_hp_move+0x6b/0x210 [ptlrpc] 00:49:01: [&lt;ffffffffa0aab445&gt;] ? req_capsule_client_get+0x15/0x20 [ptlrpc] 00:49:01: [&lt;ffffffffa0a633f8&gt;] ? ldlm_server_blocking_ast+0x228/0x8b0 [ptlrpc] 00:49:01: [&lt;ffffffffa0ae1ba1&gt;] ? tgt_blocking_ast+0x1b1/0x8b0 [ptlrpc] 00:49:01: [&lt;ffffffff812975c4&gt;] ? snprintf+0x34/0x40 00:49:01: [&lt;ffffffffa0a36dbd&gt;] ? ldlm_work_bl_ast_lock+0xdd/0x290 [ptlrpc] 00:49:01: [&lt;ffffffffa0a791c4&gt;] ? ptlrpc_set_wait+0x74/0x900 [ptlrpc] 00:49:01: [&lt;ffffffff81174c13&gt;] ? kmem_cache_alloc_trace+0x1b3/0x1c0 00:49:01: [&lt;ffffffff81174f6c&gt;] ? __kmalloc+0x21c/0x230 00:49:01: [&lt;ffffffffa0a758d2&gt;] ? ptlrpc_prep_set+0x112/0x2e0 [ptlrpc] 00:49:01: [&lt;ffffffffa0a36ce0&gt;] ? ldlm_work_bl_ast_lock+0x0/0x290 [ptlrpc] 00:49:01: [&lt;ffffffffa0a38f7b&gt;] ? ldlm_run_ast_work+0x1db/0x470 [ptlrpc] 00:49:01: [&lt;ffffffffa0a50685&gt;] ? ldlm_process_extent_lock+0x155/0xab0 [ptlrpc] 00:49:01: [&lt;ffffffffa0a3883b&gt;] ? ldlm_lock_enqueue+0x46b/0x9d0 [ptlrpc] 00:49:01: [&lt;ffffffffa0a6472b&gt;] ? ldlm_handle_enqueue0+0x51b/0x13f0 [ptlrpc] 00:49:01: [&lt;ffffffffa0ae56b1&gt;] ? tgt_enqueue+0x61/0x230 [ptlrpc] 00:49:01: [&lt;ffffffffa0ae61ce&gt;] ? tgt_request_handle+0x94e/0x10a0 [ptlrpc] 00:49:01: [&lt;ffffffffa0a95bf1&gt;] ? ptlrpc_main+0xe41/0x1970 [ptlrpc] 00:49:01: [&lt;ffffffffa0a94db0&gt;] ? ptlrpc_main+0x0/0x1970 [ptlrpc] 00:49:01: [&lt;ffffffff8109e71e&gt;] ? kthread+0x9e/0xc0 00:49:01: [&lt;ffffffff8100c20a&gt;] ? child_rip+0xa/0x20 00:49:01: [&lt;ffffffff8109e680&gt;] ? kthread+0x0/0xc0 00:49:01: [&lt;ffffffff8100c200&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p>It seems like a memory corruption issue.</p> <p>More instances on master branch:<br/> <a href="https://testing.hpdd.intel.com/test_sets/d086aa90-fffa-11e4-a3db-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/d086aa90-fffa-11e4-a3db-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/2d9d9918-ff2f-11e4-be81-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/2d9d9918-ff2f-11e4-be81-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/2d9d9918-ff2f-11e4-be81-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/2d9d9918-ff2f-11e4-be81-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/b8f327a0-fec9-11e4-a4ed-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/b8f327a0-fec9-11e4-a4ed-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/d505ade0-fe8e-11e4-919c-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/d505ade0-fe8e-11e4-919c-5254006e85c2</a></p> <p>another on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/3916eec4-00c5-11e5-9650-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/3916eec4-00c5-11e5-9650-5254006e85c2</a></p> <p>This is failing about twice a day on average. It has been hitting regularly since 2015-05-04, so maybe some patch that landed within the previous day or two caused this problem to fail more often?</p> <p>In hindsight, that is because the NRS testing was enabled via <a href="http://review.whamcloud.com/9286" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9286</a> "<a href="https://jira.whamcloud.com/browse/LU-3266" title="Regression tests for NRS policies" class="issue-link" data-issue-key="LU-3266"><del>LU-3266</del></a> test: regression tests for nrs policies" on 2015-05-01, so the bug itself has probably been around a long time.</p> <p>Hi,</p> <p>One more instance:<br/> <a href="https://testing.hpdd.intel.com/test_sets/320f7778-09c5-11e5-8421-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/320f7778-09c5-11e5-8421-5254006e85c2</a></p> <p>another on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/92cbbd1a-16d1-11e5-8436-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/92cbbd1a-16d1-11e5-8436-5254006e85c2</a></p> <p>Several recent failures on master:<br/> 2015-06-25 04:16:38 - <a href="https://testing.hpdd.intel.com/test_sets/3777cfde-1b55-11e5-ac09-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/3777cfde-1b55-11e5-ac09-5254006e85c2</a><br/> 2015-06-25 10:17:54 - <a href="https://testing.hpdd.intel.com/test_sets/a76ea20e-1b9b-11e5-ac09-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/a76ea20e-1b9b-11e5-ac09-5254006e85c2</a><br/> 2015-06-26 15:02:25 - <a href="https://testing.hpdd.intel.com/test_sets/474e2b02-1c76-11e5-9e33-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/474e2b02-1c76-11e5-9e33-5254006e85c2</a><br/> 2015-06-27 09:20:38 - <a href="https://testing.hpdd.intel.com/test_sets/a148e48a-1d16-11e5-9df2-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/a148e48a-1d16-11e5-9df2-5254006e85c2</a><br/> 2015-06-29 10:47:41 - <a href="https://testing.hpdd.intel.com/test_sets/923d2e80-1eae-11e5-8f20-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/923d2e80-1eae-11e5-8f20-5254006e85c2</a></p> <p><a href="https://testing.hpdd.intel.com/test_sets/1bbf8b9a-2a6b-11e5-b04d-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/1bbf8b9a-2a6b-11e5-b04d-5254006e85c2</a></p> <p>another on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/d2ef2ba8-2c15-11e5-8c67-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/d2ef2ba8-2c15-11e5-8c67-5254006e85c2</a></p> <p><a href="https://testing.hpdd.intel.com/test_sets/f8aae4b0-2d58-11e5-831c-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/f8aae4b0-2d58-11e5-831c-5254006e85c2</a></p> <p>This is not an ORR-only issue, we hit it on CRR-N too on a MDS:</p> <p> #0 <span class="error">&#91;ffff880ffaa3f4c8&#93;</span> machine_kexec at ffffffff8103b5bb<br/> #1 <span class="error">&#91;ffff880ffaa3f528&#93;</span> crash_kexec at ffffffff810c9c82<br/> #2 <span class="error">&#91;ffff880ffaa3f5f8&#93;</span> panic at ffffffff81529b1e<br/> #3 <span class="error">&#91;ffff880ffaa3f678&#93;</span> lbug_with_loc at ffffffffa038aedd <span class="error">&#91;libcfs&#93;</span><br/> #4 <span class="error">&#91;ffff880ffaa3f6f8&#93;</span> cfs_hash_findadd_unique at ffffffffa03a2db8 <span class="error">&#91;libcfs&#93;</span><br/> #5 <span class="error">&#91;ffff880ffaa3f718&#93;</span> nrs_crrn_res_get at ffffffffa06a7a93 <span class="error">&#91;ptlrpc&#93;</span><br/> #6 <span class="error">&#91;ffff880ffaa3f758&#93;</span> nrs_resource_get at ffffffffa06a1116 <span class="error">&#91;ptlrpc&#93;</span><br/> #7 <span class="error">&#91;ffff880ffaa3f7b8&#93;</span> nrs_resource_get_safe at ffffffffa06a1adb <span class="error">&#91;ptlrpc&#93;</span><br/> #8 <span class="error">&#91;ffff880ffaa3f7f8&#93;</span> ptlrpc_nrs_req_hp_move at ffffffffa06a42b8 <span class="error">&#91;ptlrpc&#93;</span><br/> #9 <span class="error">&#91;ffff880ffaa3f848&#93;</span> ldlm_server_blocking_ast at ffffffffa0642018 <span class="error">&#91;ptlrpc&#93;</span><br/> #10 <span class="error">&#91;ffff880ffaa3f898&#93;</span> ldlm_work_bl_ast_lock at ffffffffa061538d <span class="error">&#91;ptlrpc&#93;</span><br/> #11 <span class="error">&#91;ffff880ffaa3f918&#93;</span> ptlrpc_set_wait at ffffffffa065648c <span class="error">&#91;ptlrpc&#93;</span><br/> #12 <span class="error">&#91;ffff880ffaa3f9b8&#93;</span> ldlm_run_ast_work at ffffffffa061800b <span class="error">&#91;ptlrpc&#93;</span><br/> #13 <span class="error">&#91;ffff880ffaa3f9e8&#93;</span> ldlm_process_inodebits_lock at ffffffffa0646507 <span class="error">&#91;ptlrpc&#93;</span><br/> #14 <span class="error">&#91;ffff880ffaa3fa68&#93;</span> ldlm_lock_enqueue at ffffffffa06175b5 <span class="error">&#91;ptlrpc&#93;</span><br/> #15 <span class="error">&#91;ffff880ffaa3fac8&#93;</span> ldlm_cli_enqueue_local at ffffffffa0636b53 <span class="error">&#91;ptlrpc&#93;</span><br/> #16 <span class="error">&#91;ffff880ffaa3fb48&#93;</span> mdt_object_lock0 at ffffffffa0c988f6 <span class="error">&#91;mdt&#93;</span><br/> #17 <span class="error">&#91;ffff880ffaa3fbf8&#93;</span> mdt_object_lock at ffffffffa0c99334 <span class="error">&#91;mdt&#93;</span><br/> #18 <span class="error">&#91;ffff880ffaa3fc08&#93;</span> mdt_reint_unlink at ffffffffa0cb1cce <span class="error">&#91;mdt&#93;</span><br/> #19 <span class="error">&#91;ffff880ffaa3fc88&#93;</span> mdt_reint_rec at ffffffffa0cae671 <span class="error">&#91;mdt&#93;</span><br/> #20 <span class="error">&#91;ffff880ffaa3fca8&#93;</span> mdt_reint_internal at ffffffffa0c93cb3 <span class="error">&#91;mdt&#93;</span><br/> #21 <span class="error">&#91;ffff880ffaa3fce8&#93;</span> mdt_reint at ffffffffa0c93fb4 <span class="error">&#91;mdt&#93;</span><br/> #22 <span class="error">&#91;ffff880ffaa3fd08&#93;</span> mdt_handle_common at ffffffffa0c96aba <span class="error">&#91;mdt&#93;</span><br/> #23 <span class="error">&#91;ffff880ffaa3fd58&#93;</span> mds_regular_handle at ffffffffa0cd3985 <span class="error">&#91;mdt&#93;</span><br/> #24 <span class="error">&#91;ffff880ffaa3fd68&#93;</span> ptlrpc_server_handle_request at ffffffffa0670cf5 <span class="error">&#91;ptlrpc&#93;</span><br/> #25 <span class="error">&#91;ffff880ffaa3fe48&#93;</span> ptlrpc_main at ffffffffa067205d <span class="error">&#91;ptlrpc&#93;</span><br/> #26 <span class="error">&#91;ffff880ffaa3fee8&#93;</span> kthread at ffffffff8109e66e<br/> #27 <span class="error">&#91;ffff880ffaa3ff48&#93;</span> kernel_thread at ffffffff8100c20a</p> <p>Hello Henri, I think the CRR-N needs the same kind of patch that Lai had pushed for ORR!<br/> Lai, why did you have decided to finally abandon your original patch ??</p> <p>Bruno,</p> <p>That patch didn't address the problem. This looks like a memory corruption issue where mostly like a piece of freed memory was accessed and written again. Later on this piece of memory was allocated and used by NRS.</p> <p>Just to be complete, one of my auto-tests session (<a href="https://testing.hpdd.intel.com/test_sessions/01e63cdc-2fa7-11e5-97d6-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/01e63cdc-2fa7-11e5-97d6-5254006e85c2</a>) has experienced the original LBUG/problem for this ticket and it looks like only the whole struct nrs_orr_object just being (re?) allocated has been poisonned causing the LBUG :</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> Concerned struct nrs_orr_object at 0xffff88006e3c2fa0 : $4 = { oo_res = { res_parent = 0x5a5a5a5a5a5a5a5a, res_policy = 0x5a5a5a5a5a5a5a5a }, oo_hnode = { next = 0x5a5a5a5a5a5a5a5a, pprev = 0x5a5a5a5a5a5a5a5a &lt;&lt;&lt;&lt; causing the LBUG! }, oo_round = 0x5a5a5a5a5a5a5a5a, oo_sequence = 0x5a5a5a5a5a5a5a5a, oo_key = { &lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt; has just been initialized in nrs_orr_res_get() { ok_fid = { f_seq = 0x100000000, f_oid = 0x794e, f_ver = 0x0 }, ok_idx = 0x0 } }, oo_ref = 0x1, &lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt; has just been initialized in nrs_orr_res_get() oo_quantum = 0x5a5a, oo_active = 0x5a5a } Slab containing this struct : ACHE NAME OBJSIZE ALLOCATED TOTAL SLABS SSIZE ffff880079bb4500 nrs_orr_hp_0 80 2 48 1 4k SLAB MEMORY TOTAL ALLOCATED FREE ffff88006e3c2000 ffff88006e3c20f0 48 2 46 FREE / [ALLOCATED] [ffff88006e3c2fa0] PAGE PHYSICAL MAPPING INDEX CNT FLAGS ffffea000181d270 6e3c2000 0 ffff88003757ad80 1 20000000000080 Memory around in same Slab and only/also poisonned : ffff88006e3c25a0: 5a5a5a5a5a5a5a5a 5a5a5a5a5a5a5a5a ZZZZZZZZZZZZZZZZ ffff88006e3c25b0: 5a5a5a5a5a5a5a5a 5a5a5a5a5a5a5a5a ZZZZZZZZZZZZZZZZ ffff88006e3c25c0: 5a5a5a5a5a5a5a5a 5a5a5a5a5a5a5a5a ZZZZZZZZZZZZZZZZ ffff88006e3c25d0: 5a5a5a5a5a5a5a5a 5a5a5a5a5a5a5a5a ZZZZZZZZZZZZZZZZ ffff88006e3c25e0: 5a5a5a5a5a5a5a5a 5a5a5a5a5a5a5a5a ZZZZZZZZZZZZZZZZ ffff88006e3c2fa0: 5a5a5a5a5a5a5a5a 5a5a5a5a5a5a5a5a ZZZZZZZZZZZZZZZZ ffff88006e3c2fb0: 5a5a5a5a5a5a5a5a 5a5a5a5a5a5a5a5a ZZZZZZZZZZZZZZZZ ffff88006e3c2fc0: 5a5a5a5a5a5a5a5a 5a5a5a5a5a5a5a5a ZZZZZZZZZZZZZZZZ ffff88006e3c2fe0: 0000000000000001 5a5a5a5a5a5a5a5a ........ZZZZZZZZ Where: CACHE NAME OBJSIZE ALLOCATED TOTAL SLABS SSIZE ffff880079bb4500 nrs_orr_hp_0 80 2 48 1 4k SLAB MEMORY TOTAL ALLOCATED FREE ffff88006e3c2000 ffff88006e3c20f0 48 2 46 FREE / [ALLOCATED] ffff88006e3c25a0 (cpu 0 cache) PAGE PHYSICAL MAPPING INDEX CNT FLAGS ffffea000181d270 6e3c2000 0 ffff88003757ad80 1 20000000000080 So only an other free nrs_orr_object in same Slab and the one just allocated causing the LBUG !! </pre> </div></div> <p>Hello Jinshan, I agree that this is confusing and looking as a corruption, but if you have a look at my crash-dump extracts, it should have been a very precise one!! By the way even the end of the Slab, right after the concerned nrs_orr_object has been preserved :</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>ffff88006e3c2f60: 0000000000000000 0000000000000000 ................ ffff88006e3c2f70: 0000000000000000 0000000000000000 ................ ffff88006e3c2f80: 0000000000000000 0000000000000000 ................ ffff88006e3c2f90: 0000000000000000 0000000000000000 ................ ffff88006e3c2fa0: 5a5a5a5a5a5a5a5a 5a5a5a5a5a5a5a5a ZZZZZZZZZZZZZZZZ ffff88006e3c2fb0: 5a5a5a5a5a5a5a5a 5a5a5a5a5a5a5a5a ZZZZZZZZZZZZZZZZ ffff88006e3c2fc0: 5a5a5a5a5a5a5a5a 5a5a5a5a5a5a5a5a ZZZZZZZZZZZZZZZZ ffff88006e3c2fd0: 0000000100000000 000000000000794e ........Ny...... ffff88006e3c2fe0: 0000000000000001 5a5a5a5a5a5a5a5a ........ZZZZZZZZ ffff88006e3c2ff0: 0000000000000000 0000000001a87067 ........gp...... </pre> </div></div> <p>Another case:<br/> 2015-07-22 13:58:00 - <a href="https://testing.hpdd.intel.com/test_sets/e15f2588-30bd-11e5-ae23-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/e15f2588-30bd-11e5-ae23-5254006e85c2</a></p> <p>another on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/ff5a31b8-308b-11e5-aa87-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/ff5a31b8-308b-11e5-aa87-5254006e85c2</a></p> <p>More from master review-zfs-part-1:<br/> 2015-07-28 20:20:41 - <a href="https://testing.hpdd.intel.com/test_sets/7b300648-35a3-11e5-b949-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/7b300648-35a3-11e5-b949-5254006e85c2</a><br/> 2015-07-28 22:07:39 - <a href="https://testing.hpdd.intel.com/test_sets/8927e85c-35ac-11e5-bbc3-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/8927e85c-35ac-11e5-bbc3-5254006e85c2</a><br/> 2015-07-29 04:24:48 - <a href="https://testing.hpdd.intel.com/test_sets/c3dfd67e-35e5-11e5-8c30-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/c3dfd67e-35e5-11e5-8c30-5254006e85c2</a></p> <p>another on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/5bfd264e-3886-11e5-9969-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/5bfd264e-3886-11e5-9969-5254006e85c2</a></p> <p>another on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/b9aec8d4-38ee-11e5-8dec-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/b9aec8d4-38ee-11e5-8dec-5254006e85c2</a></p> <p>hit again<br/> <a href="https://testing.hpdd.intel.com/test_sets/c6cd44f2-3b53-11e5-95fa-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/c6cd44f2-3b53-11e5-95fa-5254006e85c2</a></p> <p>I made a patch at: <a href="http://review.whamcloud.com/15670" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/15670</a> to reproduce it with MALLOC debug flag set.</p> <p>I'm working on this issue.</p> <p>Jinshan Xiong (jinshan.xiong@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/15943" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/15943</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-4499" title="NRS ORR cfs_hash_find_or_add() LBUG" class="issue-link" data-issue-key="LU-4499"><del>LU-4499</del></a> nrs: adjust the order of REQ NRS initilization<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 263a722ce77cac0fe55f316c100e83b92b451064</p> <p>We hit this bug again:<br/> 2015-08-21 10:39:47 - <a href="https://testing.hpdd.intel.com/test_sets/3147f89a-4829-11e5-8db5-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/3147f89a-4829-11e5-8db5-5254006e85c2</a></p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/15943/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/15943/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-4499" title="NRS ORR cfs_hash_find_or_add() LBUG" class="issue-link" data-issue-key="LU-4499"><del>LU-4499</del></a> nrs: adjust the order of REQ NRS initilization<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: f904a6617b57eb8b4b90f5bc198bdec758133922</p> <p>Landed for 2.8.</p> Duplicate LU-6633 Related LU-7084 LU-6688 Development Rank 1|hzwcyv: Rank (Obsolete) 12304 Severity