Whamcloud Community JIRA

Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us

9.4.14

940014

05-12-2023

[LU-464] Test failure on test suite obdfilter-survey https://jira.whamcloud.com/browse/LU-464 Lustre <p>This issue was created by maloo for sarah <sarah@whamcloud.com></p> <p>This issue relates to the following test suite run: <a href="https://maloo.whamcloud.com/test_sets/261d7db2-a032-11e0-aee5-52540025f9af" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/261d7db2-a032-11e0-aee5-52540025f9af</a>.</p> LU-464

Test failure on test suite obdfilter-survey

Bug Blocker Resolved Fixed Jinshan Xiong Maloo Mon, 27 Jun 2011 02:16:06 +0000 Sat, 20 Jan 2018 02:24:40 +0000 Fri, 15 Jul 2011 11:08:58 +0000 Lustre 2.1.0 0 4 <p>18:56:19:Lustre: DEBUG MARKER: == obdfilter-survey test 2a: Stripe F/S over the Network ============================================= 11:56:18 (1309114578)<br/> 18:56:20:Lustre: 6472:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import lustre-OST0000_osc->host_0_UUID netid 20000: select flavor null<br/> 18:56:20:Lustre: 6472:0:(sec.c:1474:sptlrpc_import_sec_adapt()) Skipped 7 previous similar messages<br/> 19:27:35:LustreError: 7354:0:(ldlm_lib.c:499:client_disconnect_export()) disconnecting disconnected device (lustre-OST0000_osc)<br/> 19:27:35:LustreError: 7354:0:(genops.c:1565:obd_zombie_import_add()) ASSERTION(imp->imp_sec == NULL) failed<br/> 19:27:35:LustreError: 7354:0:(genops.c:1565:obd_zombie_import_add()) LBUG<br/> 19:27:35:Pid: 7354, comm: lctl<br/> 19:27:35:<br/> 19:27:35:Call Trace:<br/> 19:27:35: <span class="error">[<00000000f8e095b0>]</span> libcfs_debug_dumpstack+0x50/0x70 <span class="error">[libcfs]</span><br/> 19:27:35: <span class="error">[<00000000f8e09d4d>]</span> lbug_with_loc+0x6d/0xd0 <span class="error">[libcfs]</span><br/> 19:27:35: <span class="error">[<00000000f8e199b6>]</span> libcfs_assertion_failed+0x66/0x70 <span class="error">[libcfs]</span><br/> 19:27:36: <span class="error">[<00000000f9afcf61>]</span> class_import_put+0x541/0x550 <span class="error">[obdclass]</span><br/> 19:27:36: <span class="error">[<00000000f9b1c39a>]</span> class_handle_unhash+0x2a/0x40 <span class="error">[obdclass]</span><br/> 19:27:36: <span class="error">[<00000000f9b00c93>]</span> class_destroy_import+0x23/0x80 <span class="error">[obdclass]</span><br/> 19:27:36: <span class="error">[<00000000f955e992>]</span> osc_precleanup+0x402/0x560 <span class="error">[osc]</span><br/> 19:27:36: <span class="error">[<00000000f9b5d7b9>]</span> lu_context_fini+0x19/0x40 <span class="error">[obdclass]</span><br/> 19:27:36: <span class="error">[<00000000f9b26d2c>]</span> class_cleanup+0x2cc/0x1c70 <span class="error">[obdclass]</span><br/> 19:27:36: <span class="error">[<00000000c045e126>]</span> get_page_from_freelist+0x96/0x370<br/> 19:27:36: <span class="error">[<00000000c045e34f>]</span> get_page_from_freelist+0x2bf/0x370<br/> 19:27:36: <span class="error">[<00000000f9b2df49>]</span> class_process_config+0x25e9/0x5ed0 <span class="error">[obdclass]</span><br/> 19:27:36: <span class="error">[<00000000c045fa6c>]</span> blockable_page_cache_readahead+0x46/0x99<br/> 19:27:36: <span class="error">[<00000000c0429f21>]</span> current_fs_time+0x4a/0x55<br/> 19:27:36: <span class="error">[<00000000c04f2d45>]</span> copy_from_user+0x31/0x5d<br/> 19:27:36: <span class="error">[<00000000f9acf2ec>]</span> obd_ioctl_getdata+0x2dc/0x22c0 <span class="error">[obdclass]</span><br/> 19:27:36: <span class="error">[<00000000c045a6dc>]</span> do_generic_mapping_read+0x378/0x380<br/> 19:27:36: <span class="error">[<00000000f9af8ece>]</span> class_handle_ioctl+0x43ce/0x4770 <span class="error">[obdclass]</span><br/> 19:27:36: <span class="error">[<00000000c045e126>]</span> get_page_from_freelist+0x96/0x370<br/> 19:27:36: <span class="error">[<00000000c045e34f>]</span> get_page_from_freelist+0x2bf/0x370<br/> 19:27:37: <span class="error">[<00000000f9ace70e>]</span> obd_class_ioctl+0xae/0x470 <span class="error">[obdclass]</span><br/> 19:27:37: <span class="error">[<00000000c046604f>]</span> __handle_mm_fault+0x690/0xaac<br/> 19:27:37: <span class="error">[<00000000f9ace660>]</span> obd_class_ioctl+0x0/0x470 <span class="error">[obdclass]</span><br/> 19:27:37: <span class="error">[<00000000c0487625>]</span> do_ioctl+0x1c/0x5d<br/> 19:27:37: <span class="error">[<00000000c0487bb9>]</span> vfs_ioctl+0x47b/0x4d3<br/> 19:27:37: <span class="error">[<00000000c044cb5b>]</span> audit_syscall_entry+0x18f/0x1b9<br/> 19:27:37: <span class="error">[<00000000c0487c59>]</span> sys_ioctl+0x48/0x5f<br/> 19:27:37: <span class="error">[<00000000c0404f4b>]</span> syscall_call+0x7/0xb<br/> 19:27:37: <IRQ> <br/> 19:27:37:Kernel panic - not syncing: LBUG</p> <p>Lai will look at this one</p> <p>19:27:35:LustreError: 7354:0:(ldlm_lib.c:499:client_disconnect_export()) disconnecting disconnected device (lustre-OST0000_osc)<br/> This message looks a bit suspicious, and I guess there may be a race in echo client setup/cleanup, will do some test to find the place tomorrow.</p> <p>The race looks like this:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>1. `lctl --device $echo_client_dev detach` -> class_detach() puts last refcount of export -> obd_zombie_add_export(), and wake up zombie thread to cull this export. 2. zombie thread culls this export: class_destroy_export() put last refcount of echo_client_obd -> obd_cleanup() -> echo_device_free() -> echo_client_cleanup() -> osc_disconnect() -> client_disconnect_export() will clear cli->cl_import 3. `lctl --device $osc_dev cleanup` -> osc_precleanup() found cl_import is not NULL (not NULL means this import is not connected), it will call -> class_destroy_import() -> class_import_put() -> LASSERT(imp->imp_sec == NULL) </pre> </div></div> <p>It's expected that step 2 is done before step 3, but if not (this can't be guaranteed currently), the ASSERT will be triggered because cli->cl_import is not cleared (see description in step 3).</p> <p>I think this bug is introduced in CLIO for obdecho.</p> <p>Review is on <a href="http://review.whamcloud.com/#change,1093" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,1093</a></p> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el5,ib_stack=ofa/211/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el5,ofa #211</a><br/> <a href="https://jira.whamcloud.com/browse/LU-464" title="Test failure on test suite obdfilter-survey" class="issue-link" data-issue-key="LU-464"><del>LU-464</del></a> obdfilter-survey test 2a ASSERT(imp->imp_sec == NULL)</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=32ff1453bab3b1771d46b717df1ec479d156e5f8" class="external-link" target="_blank" rel="nofollow noopener">32ff1453bab3b1771d46b717df1ec479d156e5f8</a><br/> Files : </p> <ul> <li>lustre/osc/osc_request.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://newbuild.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://newbuild.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el5,ib_stack=ofa/211/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el5,ofa #211</a><br/> <a href="https://jira.whamcloud.com/browse/LU-464" title="Test failure on test suite obdfilter-survey" class="issue-link" data-issue-key="LU-464"><del>LU-464</del></a> obdfilter-survey test 2a ASSERT(imp->imp_sec == NULL)</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=32ff1453bab3b1771d46b717df1ec479d156e5f8" class="external-link" target="_blank" rel="nofollow noopener">32ff1453bab3b1771d46b717df1ec479d156e5f8</a><br/> Files : </p> <ul> <li>lustre/osc/osc_request.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el6,ib_stack=inkernel/211/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el6,inkernel #211</a><br/> <a href="https://jira.whamcloud.com/browse/LU-464" title="Test failure on test suite obdfilter-survey" class="issue-link" data-issue-key="LU-464"><del>LU-464</del></a> obdfilter-survey test 2a ASSERT(imp->imp_sec == NULL)</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=32ff1453bab3b1771d46b717df1ec479d156e5f8" class="external-link" target="_blank" rel="nofollow noopener">32ff1453bab3b1771d46b717df1ec479d156e5f8</a><br/> Files : </p> <ul> <li>lustre/osc/osc_request.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el6,ib_stack=inkernel/211/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el6,inkernel #211</a><br/> <a href="https://jira.whamcloud.com/browse/LU-464" title="Test failure on test suite obdfilter-survey" class="issue-link" data-issue-key="LU-464"><del>LU-464</del></a> obdfilter-survey test 2a ASSERT(imp->imp_sec == NULL)</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=32ff1453bab3b1771d46b717df1ec479d156e5f8" class="external-link" target="_blank" rel="nofollow noopener">32ff1453bab3b1771d46b717df1ec479d156e5f8</a><br/> Files : </p> <ul> <li>lustre/osc/osc_request.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el5,ib_stack=inkernel/211/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el5,inkernel #211</a><br/> <a href="https://jira.whamcloud.com/browse/LU-464" title="Test failure on test suite obdfilter-survey" class="issue-link" data-issue-key="LU-464"><del>LU-464</del></a> obdfilter-survey test 2a ASSERT(imp->imp_sec == NULL)</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=32ff1453bab3b1771d46b717df1ec479d156e5f8" class="external-link" target="_blank" rel="nofollow noopener">32ff1453bab3b1771d46b717df1ec479d156e5f8</a><br/> Files : </p> <ul> <li>lustre/osc/osc_request.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el5,ib_stack=inkernel/211/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el5,inkernel #211</a><br/> <a href="https://jira.whamcloud.com/browse/LU-464" title="Test failure on test suite obdfilter-survey" class="issue-link" data-issue-key="LU-464"><del>LU-464</del></a> obdfilter-survey test 2a ASSERT(imp->imp_sec == NULL)</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=32ff1453bab3b1771d46b717df1ec479d156e5f8" class="external-link" target="_blank" rel="nofollow noopener">32ff1453bab3b1771d46b717df1ec479d156e5f8</a><br/> Files : </p> <ul> <li>lustre/osc/osc_request.c</li> </ul> <p>Landed for 2.1</p> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=sles11,ib_stack=inkernel/211/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,sles11,inkernel #211</a><br/> <a href="https://jira.whamcloud.com/browse/LU-464" title="Test failure on test suite obdfilter-survey" class="issue-link" data-issue-key="LU-464"><del>LU-464</del></a> obdfilter-survey test 2a ASSERT(imp->imp_sec == NULL)</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=32ff1453bab3b1771d46b717df1ec479d156e5f8" class="external-link" target="_blank" rel="nofollow noopener">32ff1453bab3b1771d46b717df1ec479d156e5f8</a><br/> Files : </p> <ul> <li>lustre/osc/osc_request.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel/211/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el6,inkernel #211</a><br/> <a href="https://jira.whamcloud.com/browse/LU-464" title="Test failure on test suite obdfilter-survey" class="issue-link" data-issue-key="LU-464"><del>LU-464</del></a> obdfilter-survey test 2a ASSERT(imp->imp_sec == NULL)</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=32ff1453bab3b1771d46b717df1ec479d156e5f8" class="external-link" target="_blank" rel="nofollow noopener">32ff1453bab3b1771d46b717df1ec479d156e5f8</a><br/> Files : </p> <ul> <li>lustre/osc/osc_request.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el6,ib_stack=inkernel/211/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el6,inkernel #211</a><br/> <a href="https://jira.whamcloud.com/browse/LU-464" title="Test failure on test suite obdfilter-survey" class="issue-link" data-issue-key="LU-464"><del>LU-464</del></a> obdfilter-survey test 2a ASSERT(imp->imp_sec == NULL)</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=32ff1453bab3b1771d46b717df1ec479d156e5f8" class="external-link" target="_blank" rel="nofollow noopener">32ff1453bab3b1771d46b717df1ec479d156e5f8</a><br/> Files : </p> <ul> <li>lustre/osc/osc_request.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=ubuntu1004,ib_stack=inkernel/211/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,ubuntu1004,inkernel #211</a><br/> <a href="https://jira.whamcloud.com/browse/LU-464" title="Test failure on test suite obdfilter-survey" class="issue-link" data-issue-key="LU-464"><del>LU-464</del></a> obdfilter-survey test 2a ASSERT(imp->imp_sec == NULL)</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=32ff1453bab3b1771d46b717df1ec479d156e5f8" class="external-link" target="_blank" rel="nofollow noopener">32ff1453bab3b1771d46b717df1ec479d156e5f8</a><br/> Files : </p> <ul> <li>lustre/osc/osc_request.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el5,ib_stack=ofa/211/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el5,ofa #211</a><br/> <a href="https://jira.whamcloud.com/browse/LU-464" title="Test failure on test suite obdfilter-survey" class="issue-link" data-issue-key="LU-464"><del>LU-464</del></a> obdfilter-survey test 2a ASSERT(imp->imp_sec == NULL)</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=32ff1453bab3b1771d46b717df1ec479d156e5f8" class="external-link" target="_blank" rel="nofollow noopener">32ff1453bab3b1771d46b717df1ec479d156e5f8</a><br/> Files : </p> <ul> <li>lustre/osc/osc_request.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el5,ib_stack=inkernel/211/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el5,inkernel #211</a><br/> <a href="https://jira.whamcloud.com/browse/LU-464" title="Test failure on test suite obdfilter-survey" class="issue-link" data-issue-key="LU-464"><del>LU-464</del></a> obdfilter-survey test 2a ASSERT(imp->imp_sec == NULL)</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=32ff1453bab3b1771d46b717df1ec479d156e5f8" class="external-link" target="_blank" rel="nofollow noopener">32ff1453bab3b1771d46b717df1ec479d156e5f8</a><br/> Files : </p> <ul> <li>lustre/osc/osc_request.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el5,ib_stack=inkernel/211/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el5,inkernel #211</a><br/> <a href="https://jira.whamcloud.com/browse/LU-464" title="Test failure on test suite obdfilter-survey" class="issue-link" data-issue-key="LU-464"><del>LU-464</del></a> obdfilter-survey test 2a ASSERT(imp->imp_sec == NULL)</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=32ff1453bab3b1771d46b717df1ec479d156e5f8" class="external-link" target="_blank" rel="nofollow noopener">32ff1453bab3b1771d46b717df1ec479d156e5f8</a><br/> Files : </p> <ul> <li>lustre/osc/osc_request.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el5,ib_stack=ofa/211/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el5,ofa #211</a><br/> <a href="https://jira.whamcloud.com/browse/LU-464" title="Test failure on test suite obdfilter-survey" class="issue-link" data-issue-key="LU-464"><del>LU-464</del></a> obdfilter-survey test 2a ASSERT(imp->imp_sec == NULL)</p> <p>Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=32ff1453bab3b1771d46b717df1ec479d156e5f8" class="external-link" target="_blank" rel="nofollow noopener">32ff1453bab3b1771d46b717df1ec479d156e5f8</a><br/> Files : </p> <ul> <li>lustre/osc/osc_request.c</li> </ul> Development Rank 1|hzv8rb: Rank (Obsolete) 4953 Severity