https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7084 Lustre <p>This issue was created by maloo for Bob Glossman &lt;bob.glossman@intel.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.hpdd.intel.com/test_sets/6bace048-50dc-11e5-95a9-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/6bace048-50dc-11e5-95a9-5254006e85c2</a>.</p> <p>The sub-test test_77c failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>test failed to respond and timed out </pre> </div></div> <p>this test fail looks like <a href="https://jira.whamcloud.com/browse/LU-4499" title="NRS ORR cfs_hash_find_or_add() LBUG" class="issue-link" data-issue-key="LU-4499"><del>LU-4499</del></a>, but that has been marked Fixed. As far as I can tell the fix landed before the mod in this build.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>16:23:54:LustreError: 24876:0:(hash.c:1253:cfs_hash_find_or_add()) ASSERTION( hlist_unhashed(hnode) ) failed: 16:23:54:LustreError: 24876:0:(hash.c:1253:cfs_hash_find_or_add()) LBUG 16:23:54:Kernel panic - not syncing: LBUG in interrupt. 16:23:54: 16:23:54:Pid: 24876, comm: ll_ost00_006 Tainted: P -- ------------ 2.6.32-573.3.1.el6_lustre.g4276203.x86_64 #1 16:23:54:Call Trace: 16:23:54: [&lt;ffffffff815384e4&gt;] ? panic+0xa7/0x16f 16:23:54: [&lt;ffffffffa0713ebd&gt;] ? lbug_with_loc+0x8d/0xb0 [libcfs] 16:23:54: [&lt;ffffffffa0727d10&gt;] ? cfs_hash_findadd_unique+0x0/0x30 [libcfs] 16:23:54: [&lt;ffffffffa0727d28&gt;] ? cfs_hash_findadd_unique+0x18/0x30 [libcfs] 16:23:54: [&lt;ffffffffa0adf390&gt;] ? nrs_orr_res_get+0x430/0xb70 [ptlrpc] 16:23:54: [&lt;ffffffffa0ad4f86&gt;] ? nrs_resource_get+0x56/0x110 [ptlrpc] 16:23:54: [&lt;ffffffffa0a91935&gt;] ? lustre_msg_buf+0x55/0x60 [ptlrpc] 16:23:54: [&lt;ffffffffa0ad5d0b&gt;] ? nrs_resource_get_safe+0x8b/0x100 [ptlrpc] 16:23:54: [&lt;ffffffffa0ad99eb&gt;] ? ptlrpc_nrs_req_hp_move+0x6b/0x210 [ptlrpc] 16:23:54: [&lt;ffffffffa0abb165&gt;] ? req_capsule_client_get+0x15/0x20 [ptlrpc] 16:23:54: [&lt;ffffffffa0a6fab8&gt;] ? ldlm_server_blocking_ast+0x238/0x8c0 [ptlrpc] 16:23:54: [&lt;ffffffffa0af7329&gt;] ? tgt_blocking_ast+0x1b9/0x8c0 [ptlrpc] 16:23:54: [&lt;ffffffff8129bc34&gt;] ? snprintf+0x34/0x40 16:23:54: [&lt;ffffffffa0a4139e&gt;] ? ldlm_work_bl_ast_lock+0xde/0x290 [ptlrpc] 16:23:54: [&lt;ffffffffa0a87064&gt;] ? ptlrpc_set_wait+0x74/0xa20 [ptlrpc] 16:23:54: [&lt;ffffffff8117900d&gt;] ? kmem_cache_alloc_node_trace+0x1cd/0x200 16:23:54: [&lt;ffffffffa0a7e23e&gt;] ? ptlrpc_prep_set+0xbe/0x270 [ptlrpc] 16:23:54: [&lt;ffffffffa0a412c0&gt;] ? ldlm_work_bl_ast_lock+0x0/0x290 [ptlrpc] 16:23:54: [&lt;ffffffffa0a3dfcf&gt;] ? ldlm_run_ast_work+0xcf/0x4a0 [ptlrpc] 16:23:54: [&lt;ffffffffa0a5ca45&gt;] ? ldlm_process_extent_lock+0x155/0xab0 [ptlrpc] 16:23:54: [&lt;ffffffffa0a445be&gt;] ? ldlm_lock_enqueue+0x47e/0x8e0 [ptlrpc] 16:23:54: [&lt;ffffffffa0a710d7&gt;] ? ldlm_handle_enqueue0+0x807/0x15b0 [ptlrpc] 16:23:54: [&lt;ffffffffa0afbea1&gt;] ? tgt_enqueue+0x61/0x230 [ptlrpc] 16:23:54: [&lt;ffffffffa0afcaec&gt;] ? tgt_request_handle+0x8bc/0x12e0 [ptlrpc] 16:23:54: [&lt;ffffffffa0aa4731&gt;] ? ptlrpc_main+0xe41/0x1910 [ptlrpc] 16:23:54: [&lt;ffffffffa0aa38f0&gt;] ? ptlrpc_main+0x0/0x1910 [ptlrpc] 16:23:54: [&lt;ffffffff810a101e&gt;] ? kthread+0x9e/0xc0 16:23:54: [&lt;ffffffff8100c28a&gt;] ? child_rip+0xa/0x20 16:23:54: [&lt;ffffffff810a0f80&gt;] ? kthread+0x0/0xc0 </pre> </div></div> <p>Info required for matching: sanityn 77c</p> LU-7084

sanityn test_77c: cfs_hash_find_or_add()) ASSERTION( hlist_unhashed(hnode) ) failed

Bug Major Resolved Fixed Mikhail Pershin Maloo Tue, 1 Sep 2015 21:03:41 +0000 Fri, 26 Aug 2016 06:58:46 +0000 Wed, 23 Mar 2016 15:31:02 +0000 Lustre 2.8.0 Lustre 2.9.0 0 16 <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>12:14:15:Lustre: DEBUG MARKER: lctl set_param ost.OSS.*.nrs_orr_offset_type=logical 12:14:15:LustreError: 23366:0:(hash.c:1253:cfs_hash_find_or_add()) ASSERTION( hlist_unhashed(hnode) ) failed: 12:14:15:LustreError: 23366:0:(hash.c:1253:cfs_hash_find_or_add()) LBUG 12:14:15:Kernel panic - not syncing: LBUG in interrupt. 12:14:15: 12:14:15:Pid: 23366, comm: ll_ost00_017 Tainted: P --------------- 2.6.32-504.30.3.el6_lustre.gc67434c.x86_64 #1 12:14:15:Call Trace: 12:14:15: [&lt;ffffffff81529c9c&gt;] ? panic+0xa7/0x16f 12:14:15: [&lt;ffffffffa0709ebd&gt;] ? lbug_with_loc+0x8d/0xb0 [libcfs] 12:14:15: [&lt;ffffffffa071dd60&gt;] ? cfs_hash_findadd_unique+0x0/0x30 [libcfs] 12:14:15: [&lt;ffffffffa071dd78&gt;] ? cfs_hash_findadd_unique+0x18/0x30 [libcfs] 12:14:15: [&lt;ffffffffa0ad9070&gt;] ? nrs_orr_res_get+0x430/0xb70 [ptlrpc] 12:14:15: [&lt;ffffffffa0acec46&gt;] ? nrs_resource_get+0x56/0x110 [ptlrpc] 12:14:15: [&lt;ffffffffa0a8b7b5&gt;] ? lustre_msg_buf+0x55/0x60 [ptlrpc] 12:14:15: [&lt;ffffffffa0acf9eb&gt;] ? nrs_resource_get_safe+0x8b/0x100 [ptlrpc] 12:14:15: [&lt;ffffffffa0ad36cb&gt;] ? ptlrpc_nrs_req_hp_move+0x6b/0x210 [ptlrpc] 12:14:15: [&lt;ffffffffa0ab4eb5&gt;] ? req_capsule_client_get+0x15/0x20 [ptlrpc] 12:14:15: [&lt;ffffffffa0a69988&gt;] ? ldlm_server_blocking_ast+0x238/0x8c0 [ptlrpc] 12:14:15: [&lt;ffffffffa0af0ff9&gt;] ? tgt_blocking_ast+0x1b9/0x8c0 [ptlrpc] 12:14:15: [&lt;ffffffffa0a3b39e&gt;] ? ldlm_work_bl_ast_lock+0xde/0x290 [ptlrpc] 12:14:15: [&lt;ffffffffa0a80ee4&gt;] ? ptlrpc_set_wait+0x74/0xa20 [ptlrpc] 12:14:15: [&lt;ffffffff8117591d&gt;] ? kmem_cache_alloc_node_trace+0x1cd/0x200 12:14:15: [&lt;ffffffffa0a780ae&gt;] ? ptlrpc_prep_set+0xbe/0x270 [ptlrpc] 12:14:15: [&lt;ffffffffa0a3b2c0&gt;] ? ldlm_work_bl_ast_lock+0x0/0x290 [ptlrpc] 12:14:15: [&lt;ffffffffa0a37fcf&gt;] ? ldlm_run_ast_work+0xcf/0x4a0 [ptlrpc] 12:14:15: [&lt;ffffffffa0a56915&gt;] ? ldlm_process_extent_lock+0x155/0xab0 [ptlrpc] 12:14:15: [&lt;ffffffffa0a3e5be&gt;] ? ldlm_lock_enqueue+0x47e/0x8e0 [ptlrpc] 12:14:15: [&lt;ffffffffa0a6afa7&gt;] ? ldlm_handle_enqueue0+0x807/0x15b0 [ptlrpc] 12:14:15: [&lt;ffffffffa0af5b71&gt;] ? tgt_enqueue+0x61/0x230 [ptlrpc] 12:14:15: [&lt;ffffffffa0af694c&gt;] ? tgt_request_handle+0xa4c/0x1290 [ptlrpc] 12:14:15: [&lt;ffffffffa0a9e5b1&gt;] ? ptlrpc_main+0xe41/0x1910 [ptlrpc] 12:14:15: [&lt;ffffffff8109e78e&gt;] ? kthread+0x9e/0xc0 </pre> </div></div> <p>Another instance: <a href="https://testing.hpdd.intel.com/test_sets/64c3d51a-5c13-11e5-9dac-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/64c3d51a-5c13-11e5-9dac-5254006e85c2</a></p> <p>Here is a failure on master within the past week:<br/> <a href="https://testing.hpdd.intel.com/test_sessions/19297758-627c-11e5-a45a-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/19297758-627c-11e5-a45a-5254006e85c2</a></p> <p>another on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/f7e06f60-7064-11e5-a914-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/f7e06f60-7064-11e5-a914-5254006e85c2</a></p> <p>+1 at <a href="https://testing.hpdd.intel.com/test_sets/b442abee-72dd-11e5-b8fe-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/b442abee-72dd-11e5-b8fe-5254006e85c2</a></p> <p>another on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/691d1330-78e0-11e5-9aa5-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/691d1330-78e0-11e5-9aa5-5254006e85c2</a></p> <p>another one:<br/> <a href="https://testing.hpdd.intel.com/test_sets/ce9c606e-7c12-11e5-9851-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/ce9c606e-7c12-11e5-9851-5254006e85c2</a></p> <p>Mike Pershin (mike.pershin@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/16952" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16952</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7084" title="sanityn test_77c: cfs_hash_find_or_add()) ASSERTION( hlist_unhashed(hnode) ) failed" class="issue-link" data-issue-key="LU-7084"><del>LU-7084</del></a> ptlrpc: add request to the HP list properly<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 43b4181507f1e3e2c84188386959d9ee71b69968</p> <p>More failures at:<br/> 2015-10-26 22:44:58 - <a href="https://testing.hpdd.intel.com/test_sets/142ebe04-7c5b-11e5-9ca1-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/142ebe04-7c5b-11e5-9ca1-5254006e85c2</a><br/> 2015-10-26 20:29:26 - <a href="https://testing.hpdd.intel.com/test_sets/1cd1f35e-7c52-11e5-9851-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/1cd1f35e-7c52-11e5-9851-5254006e85c2</a></p> <p>I was wrong with that patch, ignore it.</p> <p>In fact this ticket is just duplicate of <a href="https://jira.whamcloud.com/browse/LU-4499" title="NRS ORR cfs_hash_find_or_add() LBUG" class="issue-link" data-issue-key="LU-4499"><del>LU-4499</del></a> which wasn't fixed it seems. Though patch for <a href="https://jira.whamcloud.com/browse/LU-4499" title="NRS ORR cfs_hash_find_or_add() LBUG" class="issue-link" data-issue-key="LU-4499"><del>LU-4499</del></a> makes sense it doesn't fix the issue.</p> <p>Looking at Bruno analysis in <a href="https://jira.whamcloud.com/browse/LU-4499" title="NRS ORR cfs_hash_find_or_add() LBUG" class="issue-link" data-issue-key="LU-4499"><del>LU-4499</del></a> it is clear that hnode is not initialized in ORR object right after allocation, meanwhile cfs_hash_findadd_unique() checks it should be uninitialized by comparing it with NULL which is not true for 0x5a5a5a5a of course. I don't get how that might be, if oo_key and oo_ref are set both then it happened right after allocation, meanwhile our allocations use __GFP_ZERO so there can't be 0x5a pattern. This can't be also use-after-free case because in that case all fields would be poisoned and oo_ref, oo_key can't be re-initialized of orro structure was found via hash lookup. It looks like __GFP_ZERO is not working in some cases, can that be true?</p> <p>Another failure on master: <a href="https://testing.hpdd.intel.com/test_sets/12eb8de2-7e8b-11e5-aa3e-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/12eb8de2-7e8b-11e5-aa3e-5254006e85c2</a><br/> 2015-10-30 15:25:57 - <a href="https://testing.hpdd.intel.com/test_sets/bb3f4fc0-7f57-11e5-8550-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/bb3f4fc0-7f57-11e5-8550-5254006e85c2</a><br/> 2015-11-01 06:53:15 - <a href="https://testing.hpdd.intel.com/test_sets/6e62e3ae-808a-11e5-9307-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/6e62e3ae-808a-11e5-9307-5254006e85c2</a><br/> 2015-11-02 05:22:46 - <a href="https://testing.hpdd.intel.com/test_sets/4c42df08-814e-11e5-a535-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/4c42df08-814e-11e5-a535-5254006e85c2</a><br/> 2015-11-03 06:53:04 - <a href="https://testing.hpdd.intel.com/test_sets/58ca9030-821c-11e5-a6c5-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/58ca9030-821c-11e5-a6c5-5254006e85c2</a><br/> 2015-11-04 18:45:04 - <a href="https://testing.hpdd.intel.com/test_sets/3a5c8c64-8372-11e5-aa25-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/3a5c8c64-8372-11e5-aa25-5254006e85c2</a><br/> 2015-11-04 21:44:38 - <a href="https://testing.hpdd.intel.com/test_sets/90791a1e-8367-11e5-93d9-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/90791a1e-8367-11e5-93d9-5254006e85c2</a><br/> 2015-11-05 22:17:18 - <a href="https://testing.hpdd.intel.com/test_sets/665edf50-843b-11e5-a1b5-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/665edf50-843b-11e5-a1b5-5254006e85c2</a><br/> 2015-11-05 20:37:50 - <a href="https://testing.hpdd.intel.com/test_sets/8799f51e-842d-11e5-8df7-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/8799f51e-842d-11e5-8df7-5254006e85c2</a><br/> 2015-11-05 19:49:18 - <a href="https://testing.hpdd.intel.com/test_sets/4af05cea-8426-11e5-a390-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/4af05cea-8426-11e5-a390-5254006e85c2</a><br/> 2015-11-12 03:29:09 - <a href="https://testing.hpdd.intel.com/test_sets/ccbdcab2-891e-11e5-8174-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/ccbdcab2-891e-11e5-8174-5254006e85c2</a><br/> 2015-11-12 05:31:24 - <a href="https://testing.hpdd.intel.com/test_sets/84231ad6-8938-11e5-9de3-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/84231ad6-8938-11e5-9de3-5254006e85c2</a><br/> 2015-11-12 12:16:13 - <a href="https://testing.hpdd.intel.com/test_sets/5a230980-8974-11e5-8174-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/5a230980-8974-11e5-8174-5254006e85c2</a><br/> 2015-11-14 20:25:49 - <a href="https://testing.hpdd.intel.com/test_sets/2c8bdca6-8b3e-11e5-8e06-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/2c8bdca6-8b3e-11e5-8e06-5254006e85c2</a><br/> 2015-11-15 16:03:32 - <a href="https://testing.hpdd.intel.com/test_sets/9b6b3fa2-8bdb-11e5-81ec-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/9b6b3fa2-8bdb-11e5-81ec-5254006e85c2</a></p> <p>Another failure instance on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/fc899e1c-8147-11e5-9c69-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/fc899e1c-8147-11e5-9c69-5254006e85c2</a></p> <p>Mike Pershin (mike.pershin@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/17051" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17051</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7084" title="sanityn test_77c: cfs_hash_find_or_add()) ASSERTION( hlist_unhashed(hnode) ) failed" class="issue-link" data-issue-key="LU-7084"><del>LU-7084</del></a> debug: add debug to get more data<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 5028c7255d886fd6fc7dc6fa95f0c00ce99bb84a</p> <p><a href="https://testing.hpdd.intel.com/test_sets/d90368e4-9991-11e5-b944-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/d90368e4-9991-11e5-b944-5254006e85c2</a></p> <p>another on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/6a00beaa-9929-11e5-9236-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/6a00beaa-9929-11e5-9236-5254006e85c2</a></p> <p>More failures on master in review-zfs-part-1:<br/> 2015-12-03 17:50:06 - <a href="https://testing.hpdd.intel.com/test_sets/4a09f2c6-9a26-11e5-a313-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/4a09f2c6-9a26-11e5-a313-5254006e85c2</a><br/> 2015-12-03 21:57:16 - <a href="https://testing.hpdd.intel.com/test_sets/c78bf85c-9a53-11e5-8b28-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/c78bf85c-9a53-11e5-8b28-5254006e85c2</a><br/> 2015-12-04 00:29:27 - <a href="https://testing.hpdd.intel.com/test_sets/dceb8258-9a49-11e5-8b28-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/dceb8258-9a49-11e5-8b28-5254006e85c2</a><br/> 2015-12-08 23:29:35 - <a href="https://testing.hpdd.intel.com/test_sets/4f246306-9e34-11e5-98a4-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/4f246306-9e34-11e5-98a4-5254006e85c2</a><br/> 2015-12-08 05:08:11 - <a href="https://testing.hpdd.intel.com/test_sets/e06968d2-9daf-11e5-91b0-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/e06968d2-9daf-11e5-91b0-5254006e85c2</a><br/> 2015-12-08 03:10:54 - <a href="https://testing.hpdd.intel.com/test_sets/be6523f2-9daa-11e5-a24e-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/be6523f2-9daa-11e5-a24e-5254006e85c2</a></p> <p>2015-12-11 06:50:30 <a href="https://testing.hpdd.intel.com/sub_tests/140fdcdc-a00f-11e5-8d69-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/140fdcdc-a00f-11e5-8d69-5254006e85c2</a><br/> 2015-12-11 14:09:05 <a href="https://testing.hpdd.intel.com/test_sets/72f3afbe-a04e-11e5-90cc-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/72f3afbe-a04e-11e5-90cc-5254006e85c2</a></p> <p>Again: <a href="https://testing.hpdd.intel.com/test_sets/4f246306-9e34-11e5-98a4-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/4f246306-9e34-11e5-98a4-5254006e85c2</a></p> <p>More failures on master:<br/> 2015-12-11 03:41:17 - <a href="https://testing.hpdd.intel.com/test_sets/9b9e814c-9fe4-11e5-ae0a-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/9b9e814c-9fe4-11e5-ae0a-5254006e85c2</a><br/> 2015-12-11 06:36:27 - <a href="https://testing.hpdd.intel.com/test_sets/8760168c-a00c-11e5-a33d-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/8760168c-a00c-11e5-a33d-5254006e85c2</a><br/> 2015-12-13 20:01:38 - <a href="https://testing.hpdd.intel.com/test_sets/504e086c-a1ff-11e5-8c8b-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/504e086c-a1ff-11e5-8c8b-5254006e85c2</a><br/> 2015-12-15 00:54:27 - <a href="https://testing.hpdd.intel.com/test_sets/b9fff734-a2f4-11e5-9b3d-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/b9fff734-a2f4-11e5-9b3d-5254006e85c2</a><br/> 2015-12-15 08:06:47 - <a href="https://testing.hpdd.intel.com/test_sets/5d8fb048-a34e-11e5-867e-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/5d8fb048-a34e-11e5-867e-5254006e85c2</a><br/> 2015-12-16 14:25:59 - <a href="https://testing.hpdd.intel.com/test_sets/6d6be14a-a438-11e5-806a-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/6d6be14a-a438-11e5-806a-5254006e85c2</a><br/> 2015-12-17 03:07:23 - <a href="https://testing.hpdd.intel.com/test_sets/b0c8dfbc-a49f-11e5-947b-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/b0c8dfbc-a49f-11e5-947b-5254006e85c2</a><br/> 2015-12-17 04:22:29 - <a href="https://testing.hpdd.intel.com/test_sets/59e6dc42-a4bf-11e5-947b-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/59e6dc42-a4bf-11e5-947b-5254006e85c2</a><br/> 2015-12-17 20:57:45 - <a href="https://testing.hpdd.intel.com/test_sets/d31c4362-a529-11e5-947b-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/d31c4362-a529-11e5-947b-5254006e85c2</a><br/> 2015-12-18 06:12:50 - <a href="https://testing.hpdd.intel.com/test_sets/20560a20-a5c3-11e5-a028-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/20560a20-a5c3-11e5-a028-5254006e85c2</a><br/> 2015-12-18 16:05:03 - <a href="https://testing.hpdd.intel.com/test_sets/19f29384-a5e5-11e5-9f01-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/19f29384-a5e5-11e5-9f01-5254006e85c2</a><br/> 2015-12-19 11:23:27 - <a href="https://testing.hpdd.intel.com/test_sets/1675bfe0-a678-11e5-860e-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/1675bfe0-a678-11e5-860e-5254006e85c2</a><br/> 2015-12-19 18:05:15 - <a href="https://testing.hpdd.intel.com/test_sets/7c3a5770-a6c1-11e5-b558-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/7c3a5770-a6c1-11e5-b558-5254006e85c2</a><br/> 2015-12-20 10:53:10 - <a href="https://testing.hpdd.intel.com/test_sets/a4955ac0-a73a-11e5-ab33-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/a4955ac0-a73a-11e5-ab33-5254006e85c2</a><br/> 2015-12-22 10:45:13 - <a href="https://testing.hpdd.intel.com/test_sets/71d6f384-a8d2-11e5-99ca-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/71d6f384-a8d2-11e5-99ca-5254006e85c2</a><br/> 2015-12-24 18:42:12 - <a href="https://testing.hpdd.intel.com/test_sets/00d2fc82-aa94-11e5-a386-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/00d2fc82-aa94-11e5-a386-5254006e85c2</a><br/> 2015-12-25 03:49:35 - <a href="https://testing.hpdd.intel.com/test_sets/a0a3dc3a-aae0-11e5-bed4-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/a0a3dc3a-aae0-11e5-bed4-5254006e85c2</a><br/> 2015-12-28 10:39:50 - <a href="https://testing.hpdd.intel.com/test_sets/82d03ae2-ad7d-11e5-8114-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/82d03ae2-ad7d-11e5-8114-5254006e85c2</a><br/> 2016-01-02 10:10:20 - <a href="https://testing.hpdd.intel.com/test_sets/2922f26a-b164-11e5-8114-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/2922f26a-b164-11e5-8114-5254006e85c2</a></p> <p>Another failure: <a href="https://testing.hpdd.intel.com/test_sets/d658c48e-a3fc-11e5-8701-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/d658c48e-a3fc-11e5-8701-5254006e85c2</a></p> <p>Yang Sheng (yang.sheng@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/17673" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17673</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7084" title="sanityn test_77c: cfs_hash_find_or_add()) ASSERTION( hlist_unhashed(hnode) ) failed" class="issue-link" data-issue-key="LU-7084"><del>LU-7084</del></a> mem: cleanup &amp; debug patch<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 587aa2138882a6f2e95a666a66b6dda47223e960</p> <p>This is due to a lack of protective parens in the definitions of the <tt>OBD_SLAB_ALLOC_GFP_*()</tt> macros. If moving_req is true (as it is in the path for the stack traces seen here) then <tt>__GFP_ZERO</tt> is not used for the allocation:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-keyword">static</span> <span class="code-object">int</span> nrs_orr_res_get(struct ptlrpc_nrs_policy *policy, struct ptlrpc_nrs_request *nrq, <span class="code-keyword">const</span> struct ptlrpc_nrs_resource *parent, struct ptlrpc_nrs_resource **resp, bool moving_req) { ... OBD_SLAB_CPT_ALLOC_PTR_GFP(orro, orrd-&gt;od_cache, nrs_pol2cptab(policy), nrs_pol2cptid(policy), moving_req ? GFP_ATOMIC : GFP_NOFS); ... } #define OBD_SLAB_CPT_ALLOC_PTR_GFP(ptr, slab, cptab, cpt, flags) \ OBD_SLAB_CPT_ALLOC_GFP(ptr, slab, cptab, cpt, sizeof *(ptr), flags) #define OBD_SLAB_CPT_ALLOC_GFP(ptr, slab, cptab, cpt, size, flags) \ __OBD_SLAB_ALLOC_VERBOSE(ptr, slab, cptab, cpt, size, flags) #define __OBD_SLAB_ALLOC_VERBOSE(ptr, slab, cptab, cpt, size, type) \ <span class="code-keyword">do</span> { \ LASSERT(ergo((type) != GFP_ATOMIC, !in_interrupt())); \ (ptr) = (cptab) == NULL ? \ kmem_cache_alloc(slab, type | __GFP_ZERO) : \ cfs_mem_cache_cpt_alloc(slab, cptab, cpt, type | __GFP_ZERO); \ <span class="code-keyword">if</span> (likely((ptr))) \ OBD_ALLOC_POST(ptr, size, <span class="code-quote">"slab-alloced"</span>); \ } <span class="code-keyword">while</span>(0) </pre> </div></div> <p>John L. Hammond (john.hammond@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/17755" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17755</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7084" title="sanityn test_77c: cfs_hash_find_or_add()) ASSERTION( hlist_unhashed(hnode) ) failed" class="issue-link" data-issue-key="LU-7084"><del>LU-7084</del></a> obd: correct some OBD allocator macro defines<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: b8b6a98d7b37d1a4f196d2f7b17c42688ec88eda</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/17755/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17755/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7084" title="sanityn test_77c: cfs_hash_find_or_add()) ASSERTION( hlist_unhashed(hnode) ) failed" class="issue-link" data-issue-key="LU-7084"><del>LU-7084</del></a> obd: correct some OBD allocator macro defines<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 08fcdbb95cd7ab3fc1246f03c3ef27c0b8a0d218</p> <p>It looks like we experienced this error on a master patch that contains <a href="http://review.whamcloud.com/17755/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17755/</a>. Logs are at:<br/> <a href="https://testing.hpdd.intel.com/test_sets/90f9959a-b459-11e5-9134-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/90f9959a-b459-11e5-9134-5254006e85c2</a></p> <p>&gt; It looks like we experienced this error on a master patch that contains <a href="http://review.whamcloud.com/17755/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17755/</a>. Logs are at:<br/> <a href="https://testing.hpdd.intel.com/test_sets/90f9959a-b459-11e5-9134-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/90f9959a-b459-11e5-9134-5254006e85c2</a></p> <p>The revision that failed was 9d1cf5779235716d9801148aee4d06597ceaab6f. This is patch set 2 of <a href="http://review.whamcloud.com/#/c/17633/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/17633/</a> which is based on eb6cd4804d65dda1b6ea4a1289cc01647d03a47a (<a href="https://jira.whamcloud.com/browse/LU-7223" title="Failover - mmp: MMP was not enabled on /dev/lvm-Role_OSS/P1 on ost1" class="issue-link" data-issue-key="LU-7223"><del>LU-7223</del></a> tests: print more information when mmp.sh failed) and does not include 17755.</p> <p>patch was landed</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/17673/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17673/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7084" title="sanityn test_77c: cfs_hash_find_or_add()) ASSERTION( hlist_unhashed(hnode) ) failed" class="issue-link" data-issue-key="LU-7084"><del>LU-7084</del></a> mem: code cleanup for memory allocation<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: e533c7c6d1eb2bbdc4818f688f76fab1d03bc2c8</p> <p>Landed for 2.9.0</p> Related LU-4499 Development Rank 1|hzxm1z: Rank (Obsolete) 9223372036854775807 Severity