Whamcloud Community JIRA

Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us

9.4.14

940014

05-12-2023

[LU-1017] MDS oops when running racer test https://jira.whamcloud.com/browse/LU-1017 Lustre <p>Each time running the racer test the MDS eventually oops when running with the newest lustre code from master.</p> LU-1017

MDS oops when running racer test

Bug Blocker Resolved Fixed Niu Yawei James A Simmons Fri, 20 Jan 2012 12:52:46 +0000 Fri, 4 May 2012 09:04:21 +0000 Mon, 13 Feb 2012 12:49:22 +0000 Lustre 2.2.0 Lustre 2.2.0 Lustre 2.1.2 0 5 <p>Here is the oops</p> <p>barry-mds1: kernel: Oops 0000 <span class="error">[1]</span> SMP<br/> last sysfs file: /devices/pci0000:00/0000:00:00.0/irq<br/> CPU 7<br/> Modules linked in: ipmi_si cmm(U) osd_ldiskfs(U) mdt(U) mdd(U) mds(U) fsfilt_ldiskfs(U) exportfs mgc(U) ldiskfs(U) jbd2 crc16 lustre(U) lquota(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) libcfs(U) mpt2sas mptctl dell_rbu autofs4 ib_srp(U) ipmi_devintf ipmi_msghandler ipt_REJECT xt_tcpudp xt_state ip_conntrack nfnetlink iptable_filter ip_tables x_tables be2iscsi iscsi_tcp bnx2i cnic uio libiscsi_tcp libiscsi2 scsi_transport_iscsi2 scsi_transport_iscsi rdma_ucm(U) ib_sdp(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U) ipoib_helper(U) ib_cm(U) ib_sa(U) ipv6 xfrm_nalgo crypto_api ib_uverbs(U) ib_umad(U) iw_nes(U) iw_cxgb3(U) cxgb3(U) ib_qib(U) dca mlx4_ib(U) mlx4_en(U) mlx4_core(U) dm_mirror dm_log dm_multipath scsi_dh dm_mod raid0 raid1 video backlight sbs power_meter hwmon i2c_ec i2c_core dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport sd_mod sr_mod cdrom sg joydev mptsas ib_mthca(U) ata_piix mptscsih i50<br/> 00_edac pcspkr libata mptbase ib_mad(U) shpchp edac_mc scsi_transport_sas ib_core(U) scsi_mod tpm_tis uhci_hcd tpm ehci_hcd serio_raw tpm_bios nfs nfs_acl lockd fscache sunrpc bnx2<br/> Pid: 8111, comm: mdt_372 Tainted: G 2.6.18-238.19.1.el5.head #1<br/> RIP: 0010:<span class="error">[<ffffffff889b8ce9>]</span> <span class="error">[<ffffffff889b8ce9>]</span> :obdclass:lu_object_find_at+0x139/0x450<br/> RSP: 0018:ffff8103eb4d9630 EFLAGS: 00010286<br/> RAX: fffffffffffffff5 RBX: ffff8103de825080 RCX: 000000000000000e<br/> RDX: 0000000000000007 RSI: 0000000000000004 RDI: ffff8103d9415b40<br/> RBP: ffff8103eb4d9710 R08: ffffc20036cb8170 R09: ffff8103eb4d96c0<br/> R10: 00000000000000fe R11: 0000000200000430 R12: ffff8103de825a80<br/> R13: fffffffffffffff5 R14: ffffc20036cb8170 R15: ffff8103dbd87000<br/> FS: 00002ac79ed186e0(0000) GS:ffff81043fc59740(0000) knlGS:0000000000000000<br/> CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b<br/> CR2: fffffffffffffff5 CR3: 00000004171d6000 CR4: 00000000000006e0<br/> Process mdt_372 (pid: 8111, threadinfo ffff8103eb4d8000, task ffff8103ebcf5040)<br/> Stack: ffff8103eb4d9640 ffff8103eb4d9680 000004e5eb4d97b0 ffffffff88eec336<br/> ffffffff88ef0988 0000000200000000 0000000000000000 0000000000000000<br/> ffff8103eb4d9ed0 ffff8103dcbf43a4 0000000000000000 ffff8103ebcf5040<br/> Call Trace:<br/> <span class="error">[<ffffffff80093cea>]</span> default_wake_function+0x0/0xf<br/> <span class="error">[<ffffffff88a88970>]</span> :ptlrpc:lustre_msg_buf+0x0/0x80<br/> <span class="error">[<ffffffff889b9011>]</span> :obdclass:lu_object_find+0x11/0x20<br/> <span class="error">[<ffffffff88eb2d80>]</span> :mdt:mdt_object_find+0xf0/0x180<br/> <span class="error">[<ffffffff88eb4d1a>]</span> :mdt:mdt_object_find_lock+0x3a/0x140<br/> <span class="error">[<ffffffff88a87eb3>]</span> :ptlrpc:lustre_msg_get_flags+0x23/0x80<br/> <span class="error">[<ffffffff88edf2a0>]</span> :mdt:mdt_reint_open+0x1b80/0x3250<br/> <span class="error">[<ffffffff88e82702>]</span> :mdd:md_ucred+0x42/0x50<br/> <span class="error">[<ffffffff88e82702>]</span> :mdd:md_ucred+0x42/0x50<br/> <span class="error">[<ffffffff88ecaf53>]</span> :mdt:mdt_reint_rec+0x83/0x100<br/> <span class="error">[<ffffffff88eb13a5>]</span> :mdt:mdt_reint_internal+0x8e5/0x990<br/> <span class="error">[<ffffffff88ebef55>]</span> :mdt:mdt_intent_reint+0x1f5/0x5e0<br/> <span class="error">[<ffffffff88ebf7a7>]</span> :mdt:mdt_intent_policy+0x467/0x650<br/> <span class="error">[<ffffffff88a46299>]</span> :ptlrpc:ldlm_lock_enqueue+0x179/0xa00<br/> <span class="error">[<ffffffff88a62880>]</span> :ptlrpc:ldlm_export_lock_get+0x10/0x20<br/> <span class="error">[<ffffffff888d8511>]</span> :libcfs:cfs_hash_bd_add_locked+0x71/0x80<br/> <span class="error">[<ffffffff888dbecd>]</span> :libcfs:cfs_hash_add+0x17d/0x190<br/> <span class="error">[<ffffffff88a64d4b>]</span> :ptlrpc:ldlm_handle_enqueue0+0x9eb/0xfe0<br/> <span class="error">[<ffffffff88eb33ee>]</span> :mdt:mdt_unpack_req_pack_rep+0x5de/0x6a0<br/> <span class="error">[<ffffffff88ebec02>]</span> :mdt:mdt_enqueue+0x72/0x100<br/> <span class="error">[<ffffffff88eb4696>]</span> :mdt:mdt_handle_common+0x11e6/0x1750<br/> <span class="error">[<ffffffff88eb4cd0>]</span> :mdt:mdt_regular_handle+0x10/0x20<br/> <span class="error">[<ffffffff88a943a7>]</span> :ptlrpc:ptlrpc_server_handle_request+0x897/0xe90<br/> <span class="error">[<ffffffff8004abb1>]</span> mod_timer+0x2b/0x2d<br/> <span class="error">[<ffffffff888cb099>]</span> :libcfs:cfs_timer_arm+0x9/0x10<br/> <span class="error">[<ffffffff888d6fb3>]</span> :libcfs:lc_watchdog_touch+0xc3/0x120<br/> <span class="error">[<ffffffff88a971d8>]</span> :ptlrpc:ptlrpc_wait_event+0x258/0x360<br/> <span class="error">[<ffffffff8002f7b2>]</span> __wake_up+0x43/0x50<br/> <span class="error">[<ffffffff88a98699>]</span> :ptlrpc:ptlrpc_main+0x13b9/0x15a0<br/> <span class="error">[<ffffffff88a972e0>]</span> :ptlrpc:ptlrpc_main+0x0/0x15a0<br/> <span class="error">[<ffffffff80061fb1>]</span> child_rip+0xa/0x11<br/> <span class="error">[<ffffffff88a972e0>]</span> :ptlrpc:ptlrpc_main+0x0/0x15a0<br/> <span class="error">[<ffffffff88a972e0>]</span> :ptlrpc:ptlrpc_main+0x0/0x15a0<br/> <span class="error">[<ffffffff80061fa7>]</span> child_rip+0x0/0x11</p> <p> Code: 48 8b 10 48 83 c2 38 48 8b 0a 48 39 d1 0f 84 d4 02 00 00 48<br/> RIP <span class="error">[<ffffffff889b8ce9>]</span> :obdclass:lu_object_find_at+0x139/0x450<br/> RSP <ffff8103eb4d9630><br/> CR2: fffffffffffffff5<br/> <0>Kernel panic - not syncing: Fatal exception</p> <p>Cliff</p> <p>Does this appear to be a duplicate of the issue that you ran into on Hyperion?</p> <p>Peter</p> <p>This does not appear to be the same, we have hit LBUG/Assert, not an OOPS, I will see if I can replicate on hyperion.</p> <p>Finished git bisecting to track down the commit that is causing it. The racer problem shows up after commit 22464d1230ed58461f51d881f512d5e16644a735 which is the patch for <a href="https://jira.whamcloud.com/browse/LU-909" title="changes to OSD API" class="issue-link" data-issue-key="LU-909"><del>LU-909</del></a>. Cliff can you test the branch with that patch reverted to see if you still see your LBUG?</p> <p>I am traveling this week, so it may be difficult. Will see what I can do, good that you identified the commit. </p> <p>I uploaded our kernel module at an enginners request. Its on your ftp site in directory uploads/<a href="https://jira.whamcloud.com/browse/LU-1017" title="MDS oops when running racer test" class="issue-link" data-issue-key="LU-1017"><del>LU-1017</del></a>/obdclass.ko</p> <p>000000000004fccc <lu_object_find_at+0x12c> callq 000000000004f4f0 <htable_lookup><br/> 000000000004fcd1 <lu_object_find_at+0x131> test %rax,%rax<br/> 000000000004fcd4 <lu_object_find_at+0x134> mov %rax,%r13<br/> 000000000004fcd7 <lu_object_find_at+0x137> je 000000000004fd03 <lu_object_find_at+0x163><br/> 000000000004fcd9 <lu_object_find_at+0x139> mov (%rax),%rdx</p> <p>RAX: fffffffffffffff5 = -11</p> <p> shadow = htable_lookup(s, &bd, f, waiter, &version);<br/> if (likely(shadow == NULL)) {</p> <p>in turn...</p> <p>htable_lookup() can return -EAGAIN:</p> <p> /*</p> <ul> <li>Lookup found an object being destroyed this object cannot be</li> <li>returned (to assure that references to dying objects are eventually</li> <li>drained), and moreover, lookup has to wait until object is freed.<br/> */<br/> cfs_atomic_dec(&h->loh_ref);</li> </ul> <p> cfs_waitlink_init(waiter);<br/> cfs_waitq_add(&bkt->lsb_marche_funebre, waiter);<br/> cfs_set_current_state(CFS_TASK_UNINT);<br/> lprocfs_counter_incr(s->ls_stats, LU_SS_CACHE_DEATH_RACE);<br/> return ERR_PTR(-EAGAIN);</p> <p>Niu</p> <p>Could you please create a patch for this?</p> <p>Thanks</p> <p>Peter</p> <p>Hi, Alex</p> <p>Do you mean that the oops is triggered when htable_lookup() returns -EAGAIN? I don't see the reason from the code, could you explain it more? Thanks.</p> <p>htable_lookup() found an object being destroyed and returned -EAGAIN</p> <p><a href="http://review.whamcloud.com/2066" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/2066</a></p> <p>No more oops.</p> <p>Spoke to fast. The test complete but once I ran llmountcleanup.sh I get the following oops</p> <p>Jan 31 14:13:43 spoon02 kernel: Lustre: DEBUG MARKER: == racer racer.sh test complete, duration 659 sec ==================================================== 14:1<br/> 3:43 (1328037223)<br/> Jan 31 14:29:31 spoon02 kernel: Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request<br/> Jan 31 14:29:31 spoon02 kernel: LustreError: 25339:0:(file.c:157:ll_close_inode_openhandle()) inode 144115205255725063 mdc close failed: rc = -4<br/> Jan 31 14:29:31 spoon02 kernel: Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request<br/> Jan 31 14:29:31 spoon02 kernel: LustreError: 25321:0:(file.c:157:ll_close_inode_openhandle()) inode 144115205255725059 mdc close failed: rc = -108<br/> Jan 31 14:29:31 spoon02 kernel: LustreError: 25321:0:(file.c:157:ll_close_inode_openhandle()) Skipped 1 previous similar message<br/> Jan 31 14:29:31 spoon02 kernel: LustreError: 25321:0:(cl_lock.c:2082:cl_locks_prune()) ASSERTION(lock->cll_users == 0) failed<br/> Jan 31 14:29:31 spoon02 kernel: LustreError: 25321:0:(cl_lock.c:2082:cl_locks_prune()) LBUG<br/> Jan 31 14:29:31 spoon02 kernel: Pid: 25321, comm: dd<br/> Jan 31 14:29:31 spoon02 kernel: <br/> Jan 31 14:29:31 spoon02 kernel: Call Trace:<br/> Jan 31 14:29:31 spoon02 kernel: <span class="error">[<ffffffff8882372f>]</span> libcfs_debug_dumpstack+0x5f/0x80 <span class="error">[libcfs]</span><br/> Jan 31 14:29:31 spoon02 kernel: <span class="error">[<ffffffff88823c5f>]</span> lbug_with_loc+0x7f/0xd0 <span class="error">[libcfs]</span><br/> Jan 31 14:29:31 spoon02 kernel: <span class="error">[<ffffffff8882f191>]</span> libcfs_assertion_failed+0x61/0x70 <span class="error">[libcfs]</span><br/> Jan 31 14:29:31 spoon02 kernel: <span class="error">[<ffffffff88954e7c>]</span> cl_locks_prune+0x14c/0x210 <span class="error">[obdclass]</span><br/> Jan 31 14:29:31 spoon02 kernel: <span class="error">[<ffffffff88831aec>]</span> cfs_hash_bd_from_key+0x3c/0xc0 <span class="error">[libcfs]</span><br/> Jan 31 14:29:31 spoon02 kernel: <span class="error">[<ffffffff8894a992>]</span> cl_object_kill+0x82/0x90 <span class="error">[obdclass]</span><br/> Jan 31 14:29:31 spoon02 kernel: <span class="error">[<ffffffff88bba781>]</span> lov_delete_raid0+0x141/0x300 <span class="error">[lov]</span><br/> Jan 31 14:29:31 spoon02 kernel: <span class="error">[<ffffffff88943fa1>]</span> lu_obj_hop_hash+0x131/0x240 <span class="error">[obdclass]</span><br/> Jan 31 14:29:31 spoon02 kernel: <span class="error">[<ffffffff88bbb7df>]</span> lov_object_delete+0xcf/0x150 <span class="error">[lov]</span><br/> Jan 31 14:29:31 spoon02 kernel: <span class="error">[<ffffffff88942709>]</span> lu_object_free+0x89/0x190 <span class="error">[obdclass]</span><br/> Jan 31 14:29:31 spoon02 kernel: <span class="error">[<ffffffff88833c8d>]</span> cfs_hash_hd_hnode_del+0xd/0x50 <span class="error">[libcfs]</span><br/> Jan 31 14:29:31 spoon02 kernel: <span class="error">[<ffffffff88943613>]</span> lu_object_put+0x1c3/0x1e0 <span class="error">[obdclass]</span><br/> Jan 31 14:29:31 spoon02 kernel: <span class="error">[<ffffffff88949629>]</span> cl_object_put+0x9/0x10 <span class="error">[obdclass]</span><br/> Jan 31 14:29:31 spoon02 kernel: <span class="error">[<ffffffff88c8b133>]</span> cl_inode_fini+0x1d3/0x240 <span class="error">[lustre]</span><br/> Jan 31 14:29:31 spoon02 kernel: <span class="error">[<ffffffff88cf30cf>]</span> lmv_change_cbdata+0x55f/0x580 <span class="error">[lmv]</span><br/> Jan 31 14:29:31 spoon02 kernel: <span class="error">[<ffffffff88c53a60>]</span> null_if_equal+0x0/0x50 <span class="error">[lustre]</span><br/> Jan 31 14:29:31 spoon02 kernel: <span class="error">[<ffffffff88c63248>]</span> ll_clear_inode+0x838/0xc70 <span class="error">[lustre]</span><br/> Jan 31 14:29:31 spoon02 kernel: <span class="error">[<ffffffff88c56c10>]</span> ll_delete_inode+0x0/0x600 <span class="error">[lustre]</span><br/> Jan 31 14:29:31 spoon02 kernel: <span class="error">[<ffffffff80023be7>]</span> clear_inode+0xda/0x12d<br/> Jan 31 14:29:31 spoon02 kernel: <span class="error">[<ffffffff88c5718a>]</span> ll_delete_inode+0x57a/0x600 <span class="error">[lustre]</span><br/> Jan 31 14:29:31 spoon02 kernel: <span class="error">[<ffffffff88c56c10>]</span> ll_delete_inode+0x0/0x600 <span class="error">[lustre]</span><br/> Jan 31 14:29:31 spoon02 kernel: <span class="error">[<ffffffff800309d8>]</span> generic_delete_inode+0xc9/0x147<br/> Jan 31 14:29:31 spoon02 kernel: <span class="error">[<ffffffff8003b777>]</span> generic_drop_inode+0x15/0x15f<br/> Jan 31 14:29:31 spoon02 kernel: <span class="error">[<ffffffff8002c25c>]</span> iput+0x85/0x8a<br/> Jan 31 14:29:31 spoon02 kernel: <span class="error">[<ffffffff88c289ab>]</span> ll_d_iput+0x4b/0x60 <span class="error">[lustre]</span><br/> Jan 31 14:29:31 spoon02 kernel: <span class="error">[<ffffffff80036aa8>]</span> dentry_iput+0x8c/0xae<br/> Jan 31 14:29:31 spoon02 kernel: <span class="error">[<ffffffff8000d890>]</span> dput+0xf7/0x115<br/> Jan 31 14:29:31 spoon02 kernel: <span class="error">[<ffffffff800131c5>]</span> __fput+0x19c/0x1bd<br/> Jan 31 14:29:31 spoon02 kernel: <span class="error">[<ffffffff8002e001>]</span> fput+0x14/0x16<br/> Jan 31 14:29:31 spoon02 kernel: <span class="error">[<ffffffff800249d3>]</span> filp_close+0x65/0x70<br/> Jan 31 14:29:31 spoon02 kernel: <span class="error">[<ffffffff8003aa87>]</span> put_files_struct+0x6b/0xb3<br/> Jan 31 14:29:31 spoon02 kernel: <span class="error">[<ffffffff80015fb2>]</span> do_exit+0x38c/0x9c0<br/> Jan 31 14:29:31 spoon02 kernel: <span class="error">[<ffffffff8004b974>]</span> cpuset_exit+0x0/0x8f<br/> Jan 31 14:29:31 spoon02 kernel: <span class="error">[<ffffffff8002c6d6>]</span> get_signal_to_deliver+0x475/0x4a7<br/> Jan 31 14:29:31 spoon02 kernel: <span class="error">[<ffffffff8006119f>]</span> sysret_signal+0x1c/0x27<br/> Jan 31 14:29:31 spoon02 kernel: <span class="error">[<ffffffff8005e0d0>]</span> do_notify_resume+0xa4/0x84e<br/> Jan 31 14:29:31 spoon02 kernel: <span class="error">[<ffffffff8016685c>]</span> list_add+0xc/0xe<br/> Jan 31 14:29:31 spoon02 kernel: <span class="error">[<ffffffff88948d68>]</span> cl_env_put+0x288/0x2c0 <span class="error">[obdclass]</span><br/> Jan 31 14:29:31 spoon02 kernel: <span class="error">[<ffffffff80017267>]</span> vfs_write+0xcf/0x175<br/> Jan 31 14:29:31 spoon02 kernel: <span class="error">[<ffffffff8006119f>]</span> sysret_signal+0x1c/0x27<br/> Jan 31 14:29:31 spoon02 kernel: <span class="error">[<ffffffff80061427>]</span> ptregscall_common+0x67/0xac<br/> Jan 31 14:29:31 spoon02 kernel: </p> <p>This should be a separate defect, it looks like the io process exit with sigkill, and we missed lock unuse somewhere in this code path, but I didn't figure it out from the code yet. I've asked Jingshan to look into it too.</p> <p>The recent significant change on that part is fanyong's <a href="https://jira.whamcloud.com/browse/LU-925" title="land Async Glimpse Lock (AGL) to master to pre-fetch size/block attributes from OSTs for statahead" class="issue-link" data-issue-key="LU-925"><del>LU-925</del></a> with commit number 6f5813d36102a19f314c9aab409972e8a9f1112b</p> <p>James, can you please share me some parameters to run racer as I want to reproduce it in our lab?</p> <p>> Jan 31 14:29:31 spoon02 kernel: LustreError: 25321:0:(file.c:157:ll_close_inode_openhandle()) Skipped 1 previous similar message<br/> > Jan 31 14:29:31 spoon02 kernel: LustreError: 25321:0:(cl_lock.c:2082:cl_locks_prune()) ASSERTION(lock->cll_users == 0) failed<br/> > Jan 31 14:29:31 spoon02 kernel: LustreError: 25321:0:(cl_lock.c:2082:cl_locks_prune()) LBUG</p> <p>It maybe caused by AGL (async glimpse lock), which holds an user count for AGL RPC reply processing later. Since it is not the same as original issue, I will fix it in a new ticket <a href="https://jira.whamcloud.com/browse/LU-1061" title="LBUG on cl_locks_prune()" class="issue-link" data-issue-key="LU-1061"><del>LU-1061</del></a>.</p> <p>it probably caused by AGL (async glimpse lock) which doesn't grab inode, I'll open a separate ticket for it.</p> <p>Here is the cfg file I use to run racer in the test suite.</p> <p>Landed for 2.2</p> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el5,ib_stack=ofa/468/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el5,ofa #468</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1017" title="MDS oops when running racer test" class="issue-link" data-issue-key="LU-1017"><del>LU-1017</del></a> handle -EAGAIN properly in lu_object_find_try() (Revision 4ce426ab33196ff04cc7016a2dc717e53d1b4a75)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=4ce426ab33196ff04cc7016a2dc717e53d1b4a75" class="external-link" target="_blank" rel="nofollow noopener">4ce426ab33196ff04cc7016a2dc717e53d1b4a75</a><br/> Files : </p> <ul> <li>lustre/obdclass/lu_object.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el5,ib_stack=inkernel/468/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el5,inkernel #468</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1017" title="MDS oops when running racer test" class="issue-link" data-issue-key="LU-1017"><del>LU-1017</del></a> handle -EAGAIN properly in lu_object_find_try() (Revision 4ce426ab33196ff04cc7016a2dc717e53d1b4a75)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=4ce426ab33196ff04cc7016a2dc717e53d1b4a75" class="external-link" target="_blank" rel="nofollow noopener">4ce426ab33196ff04cc7016a2dc717e53d1b4a75</a><br/> Files : </p> <ul> <li>lustre/obdclass/lu_object.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=ubuntu1004,ib_stack=inkernel/468/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,ubuntu1004,inkernel #468</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1017" title="MDS oops when running racer test" class="issue-link" data-issue-key="LU-1017"><del>LU-1017</del></a> handle -EAGAIN properly in lu_object_find_try() (Revision 4ce426ab33196ff04cc7016a2dc717e53d1b4a75)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=4ce426ab33196ff04cc7016a2dc717e53d1b4a75" class="external-link" target="_blank" rel="nofollow noopener">4ce426ab33196ff04cc7016a2dc717e53d1b4a75</a><br/> Files : </p> <ul> <li>lustre/obdclass/lu_object.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=sles11,ib_stack=inkernel/468/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,sles11,inkernel #468</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1017" title="MDS oops when running racer test" class="issue-link" data-issue-key="LU-1017"><del>LU-1017</del></a> handle -EAGAIN properly in lu_object_find_try() (Revision 4ce426ab33196ff04cc7016a2dc717e53d1b4a75)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=4ce426ab33196ff04cc7016a2dc717e53d1b4a75" class="external-link" target="_blank" rel="nofollow noopener">4ce426ab33196ff04cc7016a2dc717e53d1b4a75</a><br/> Files : </p> <ul> <li>lustre/obdclass/lu_object.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el5,ib_stack=inkernel/468/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el5,inkernel #468</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1017" title="MDS oops when running racer test" class="issue-link" data-issue-key="LU-1017"><del>LU-1017</del></a> handle -EAGAIN properly in lu_object_find_try() (Revision 4ce426ab33196ff04cc7016a2dc717e53d1b4a75)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=4ce426ab33196ff04cc7016a2dc717e53d1b4a75" class="external-link" target="_blank" rel="nofollow noopener">4ce426ab33196ff04cc7016a2dc717e53d1b4a75</a><br/> Files : </p> <ul> <li>lustre/obdclass/lu_object.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel/468/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el6,inkernel #468</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1017" title="MDS oops when running racer test" class="issue-link" data-issue-key="LU-1017"><del>LU-1017</del></a> handle -EAGAIN properly in lu_object_find_try() (Revision 4ce426ab33196ff04cc7016a2dc717e53d1b4a75)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=4ce426ab33196ff04cc7016a2dc717e53d1b4a75" class="external-link" target="_blank" rel="nofollow noopener">4ce426ab33196ff04cc7016a2dc717e53d1b4a75</a><br/> Files : </p> <ul> <li>lustre/obdclass/lu_object.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el5,ib_stack=ofa/468/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el5,ofa #468</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1017" title="MDS oops when running racer test" class="issue-link" data-issue-key="LU-1017"><del>LU-1017</del></a> handle -EAGAIN properly in lu_object_find_try() (Revision 4ce426ab33196ff04cc7016a2dc717e53d1b4a75)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=4ce426ab33196ff04cc7016a2dc717e53d1b4a75" class="external-link" target="_blank" rel="nofollow noopener">4ce426ab33196ff04cc7016a2dc717e53d1b4a75</a><br/> Files : </p> <ul> <li>lustre/obdclass/lu_object.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el6,ib_stack=inkernel/468/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el6,inkernel #468</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1017" title="MDS oops when running racer test" class="issue-link" data-issue-key="LU-1017"><del>LU-1017</del></a> handle -EAGAIN properly in lu_object_find_try() (Revision 4ce426ab33196ff04cc7016a2dc717e53d1b4a75)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=4ce426ab33196ff04cc7016a2dc717e53d1b4a75" class="external-link" target="_blank" rel="nofollow noopener">4ce426ab33196ff04cc7016a2dc717e53d1b4a75</a><br/> Files : </p> <ul> <li>lustre/obdclass/lu_object.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el6,ib_stack=inkernel/468/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el6,inkernel #468</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1017" title="MDS oops when running racer test" class="issue-link" data-issue-key="LU-1017"><del>LU-1017</del></a> handle -EAGAIN properly in lu_object_find_try() (Revision 4ce426ab33196ff04cc7016a2dc717e53d1b4a75)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=4ce426ab33196ff04cc7016a2dc717e53d1b4a75" class="external-link" target="_blank" rel="nofollow noopener">4ce426ab33196ff04cc7016a2dc717e53d1b4a75</a><br/> Files : </p> <ul> <li>lustre/obdclass/lu_object.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el6,ib_stack=inkernel/468/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el6,inkernel #468</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1017" title="MDS oops when running racer test" class="issue-link" data-issue-key="LU-1017"><del>LU-1017</del></a> handle -EAGAIN properly in lu_object_find_try() (Revision 4ce426ab33196ff04cc7016a2dc717e53d1b4a75)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=4ce426ab33196ff04cc7016a2dc717e53d1b4a75" class="external-link" target="_blank" rel="nofollow noopener">4ce426ab33196ff04cc7016a2dc717e53d1b4a75</a><br/> Files : </p> <ul> <li>lustre/obdclass/lu_object.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el5,ib_stack=inkernel/468/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el5,inkernel #468</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1017" title="MDS oops when running racer test" class="issue-link" data-issue-key="LU-1017"><del>LU-1017</del></a> handle -EAGAIN properly in lu_object_find_try() (Revision 4ce426ab33196ff04cc7016a2dc717e53d1b4a75)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=4ce426ab33196ff04cc7016a2dc717e53d1b4a75" class="external-link" target="_blank" rel="nofollow noopener">4ce426ab33196ff04cc7016a2dc717e53d1b4a75</a><br/> Files : </p> <ul> <li>lustre/obdclass/lu_object.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el5,ib_stack=inkernel/468/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el5,inkernel #468</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1017" title="MDS oops when running racer test" class="issue-link" data-issue-key="LU-1017"><del>LU-1017</del></a> handle -EAGAIN properly in lu_object_find_try() (Revision 4ce426ab33196ff04cc7016a2dc717e53d1b4a75)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=4ce426ab33196ff04cc7016a2dc717e53d1b4a75" class="external-link" target="_blank" rel="nofollow noopener">4ce426ab33196ff04cc7016a2dc717e53d1b4a75</a><br/> Files : </p> <ul> <li>lustre/obdclass/lu_object.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=server,distro=el5,ib_stack=ofa/468/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,server,el5,ofa #468</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1017" title="MDS oops when running racer test" class="issue-link" data-issue-key="LU-1017"><del>LU-1017</del></a> handle -EAGAIN properly in lu_object_find_try() (Revision 4ce426ab33196ff04cc7016a2dc717e53d1b4a75)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=4ce426ab33196ff04cc7016a2dc717e53d1b4a75" class="external-link" target="_blank" rel="nofollow noopener">4ce426ab33196ff04cc7016a2dc717e53d1b4a75</a><br/> Files : </p> <ul> <li>lustre/obdclass/lu_object.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el5,ib_stack=ofa/468/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el5,ofa #468</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1017" title="MDS oops when running racer test" class="issue-link" data-issue-key="LU-1017"><del>LU-1017</del></a> handle -EAGAIN properly in lu_object_find_try() (Revision 4ce426ab33196ff04cc7016a2dc717e53d1b4a75)</p> <p> Result = SUCCESS<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=4ce426ab33196ff04cc7016a2dc717e53d1b4a75" class="external-link" target="_blank" rel="nofollow noopener">4ce426ab33196ff04cc7016a2dc717e53d1b4a75</a><br/> Files : </p> <ul> <li>lustre/obdclass/lu_object.c</li> </ul> <p>this patch overlaps with previous commit b9ccecd1453c5c76fe135048c39f149c241650c6 <a href="https://jira.whamcloud.com/browse/LU-1013" title="recovery-mds lu_object.c:116:lu_object_put()) ASSERTION(cfs_list_empty(&top->loh_lru)) failed" class="issue-link" data-issue-key="LU-1013"><del>LU-1013</del></a> obdclass: lu_object_find miss to unlink object from LRU.</p> <p>Fan Yong was trying to solve the same issue it seems but cover only single case and also didn't handle -EAGAIN case. This is his change:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">@@ -627,12 +627,14 @@ <span class="code-keyword">static</span> struct lu_object *lu_object_find_try(<span class="code-keyword">const</span> struct lu_env *env, bkt->lsb_busy++; cfs_hash_bd_unlock(hs, &bd, 1); <span class="code-keyword">return</span> o; + } <span class="code-keyword">else</span> { + <span class="code-keyword">if</span> (!cfs_list_empty(&shadow->lo_header->loh_lru)) + cfs_list_del_init(&shadow->lo_header->loh_lru); + lprocfs_counter_incr(s->ls_stats, LU_SS_CACHE_RACE); + cfs_hash_bd_unlock(hs, &bd, 1); + lu_object_free(env, o); + <span class="code-keyword">return</span> shadow; } - - lprocfs_counter_incr(s->ls_stats, LU_SS_CACHE_RACE); - cfs_hash_bd_unlock(hs, &bd, 1); - lu_object_free(env, o); - <span class="code-keyword">return</span> shadow; } </pre> </div></div> <p>Now we have other code using result of htable_lookup() without checking for -EAGAIN. Meanwhile this commit is not needed at all after <a href="https://jira.whamcloud.com/browse/LU-1017" title="MDS oops when running racer test" class="issue-link" data-issue-key="LU-1017"><del>LU-1017</del></a> fix, because the last does list_del_init inside of htable_lookup, so we can just revert b9ccecd1453c5c76fe135048c39f149c241650c6 to solve this issue.</p> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/red.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=server,distro=el6,ib_stack=ofa/480/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,server,el6,ofa #480</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1017" title="MDS oops when running racer test" class="issue-link" data-issue-key="LU-1017"><del>LU-1017</del></a> handle -EAGAIN properly in lu_object_find_try() (Revision 4ce426ab33196ff04cc7016a2dc717e53d1b4a75)</p> <p> Result = FAILURE<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=4ce426ab33196ff04cc7016a2dc717e53d1b4a75" class="external-link" target="_blank" rel="nofollow noopener">4ce426ab33196ff04cc7016a2dc717e53d1b4a75</a><br/> Files : </p> <ul> <li>lustre/obdclass/lu_object.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/red.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=x86_64,build_type=client,distro=el6,ib_stack=ofa/480/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » x86_64,client,el6,ofa #480</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1017" title="MDS oops when running racer test" class="issue-link" data-issue-key="LU-1017"><del>LU-1017</del></a> handle -EAGAIN properly in lu_object_find_try() (Revision 4ce426ab33196ff04cc7016a2dc717e53d1b4a75)</p> <p> Result = FAILURE<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=4ce426ab33196ff04cc7016a2dc717e53d1b4a75" class="external-link" target="_blank" rel="nofollow noopener">4ce426ab33196ff04cc7016a2dc717e53d1b4a75</a><br/> Files : </p> <ul> <li>lustre/obdclass/lu_object.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://build.whamcloud.com/images/16x16/grey.png" style="border: 0px solid black" /></span> <a href="http://build.whamcloud.com/job/lustre-master/./arch=i686,build_type=client,distro=el6,ib_stack=ofa/480/" class="external-link" target="_blank" rel="nofollow noopener">lustre-master » i686,client,el6,ofa #480</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1017" title="MDS oops when running racer test" class="issue-link" data-issue-key="LU-1017"><del>LU-1017</del></a> handle -EAGAIN properly in lu_object_find_try() (Revision 4ce426ab33196ff04cc7016a2dc717e53d1b4a75)</p> <p> Result = ABORTED<br/> Oleg Drokin : <a href="http://git.whamcloud.com/gitweb?p=fs/lustre-release.git;a=shortlog;h=refs/heads/master&a=commit&h=4ce426ab33196ff04cc7016a2dc717e53d1b4a75" class="external-link" target="_blank" rel="nofollow noopener">4ce426ab33196ff04cc7016a2dc717e53d1b4a75</a><br/> Files : </p> <ul> <li>lustre/obdclass/lu_object.c</li> </ul> <p><a href="http://review.whamcloud.com/#change,2629" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,2629</a><br/> back port to b2_1</p> Development Rank 1|hzv7fr: Rank (Obsolete) 4739 Severity