https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1243 Lustre <p>During I/O testing we hit a client stack overrun on a Lustre 2.1.1 client. We have the <a href="https://jira.whamcloud.com/browse/LU-969" title="2.1 client stack overruns" class="issue-link" data-issue-key="LU-969"><del>LU-969</del></a> stack reduction patch in our local branch. The stacktrace looks very similar to the one reported for <a href="https://jira.whamcloud.com/browse/LU-969" title="2.1 client stack overruns" class="issue-link" data-issue-key="LU-969"><del>LU-969</del></a>.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2012-03-18 06:05:23 BUG: unable to handle kernel paging request at 000000032cbc7960 2012-03-18 06:05:23 IP: [&lt;ffffffff81052814&gt;] update_curr+0x144/0x1f0 2012-03-18 06:05:23 PGD 417be5067 PUD 0 2012-03-18 06:05:23 Thread overran stack, or stack corrupted 2012-03-18 06:05:23 Oops: 0000 [#1] SMP 2012-03-18 06:05:23 last sysfs file: /sys/devices/system/cpu/cpu31/online 2012-03-18 06:05:23 CPU 6 2012-03-18 06:05:23 Modules linked in: xt_owner nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack lmv(U) mgc(U) lustre(U) lov(U) osc(U) lquota(U) mdc(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) acpi_cpufreq freq_table mperf ko2iblnd(U) lnet(U) libcfs(U) ipt_LOG xt_multiport iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ib_sa dm_mirror dm_region_hash dm_log dm_mod vhost_net macvtap macvlan tun kvm uinput sg sd_mod crc_t10dif isci libsas scsi_transport_sas ahci microcode sb_edac edac_core iTCO_wdt iTCO_vendor_support i2c_i801 i2c_core ib_qib(U) ib_mad ib_core wmi ioatdma ipv6 nfs lockd fscache nfs_acl auth_rpcgss sunrpc igb dca [last unloaded: cpufreq_ondemand] 2012-03-18 06:05:23 2012-03-18 06:05:23 Pid: 68872, comm: lmp Not tainted 2.6.32-220.7.1.3chaos.ch5.x86_64 #1 appro appro-512x/S2600JF 2012-03-18 06:05:23 RIP: 0010:[&lt;ffffffff81052814&gt;] [&lt;ffffffff81052814&gt;] update_curr+0x144/0x1f0 2012-03-18 06:05:23 RSP: 0018:ffff8800366c3db8 EFLAGS: 00010086 2012-03-18 06:05:23 RAX: ffff8808334c9580 RBX: 00000000755fa0c8 RCX: ffff880437f111c0 2012-03-18 06:05:23 RDX: 0000000000018b48 RSI: 0000000000000000 RDI: ffff8808334c95b8 2012-03-18 06:05:23 RBP: ffff8800366c3de8 R08: ffffffff8160b665 R09: 0000000000000000 2012-03-18 06:05:23 R10: 0000000000000010 R11: 0000000000000000 R12: ffff8800366d5fe8 2012-03-18 06:05:23 R13: 00000000000f3c4d R14: 000081fe1182f6c5 R15: ffff8808334c9580 2012-03-18 06:05:23 FS: 00002aaaacfd0800(0000) GS:ffff8800366c0000(0000) knlGS:0000000000000000 2012-03-18 06:05:23 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b 2012-03-18 06:05:23 CR2: 000000032cbc7960 CR3: 0000000326499000 CR4: 00000000000406e0 2012-03-18 06:05:23 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 2012-03-18 06:05:23 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 2012-03-18 06:05:23 Process lmp (pid: 68872, threadinfo ffff8807755fa000, task ffff8808334c9580) 2012-03-18 06:05:23 Stack: 2012-03-18 06:05:23 ffff8800366c3dc8 ffffffff81013783 ffff8808334c95b8 ffff8800366d5fe8 2012-03-18 06:05:23 &lt;0&gt; 0000000000000000 0000000000000000 ffff8800366c3e18 ffffffff81052e2b 2012-03-18 06:05:23 &lt;0&gt; ffff8800366d5f80 0000000000000006 0000000000015f80 0000000000000006 2012-03-18 06:05:23 Call Trace: 2012-03-18 06:05:23 &lt;IRQ&gt; 2012-03-18 06:05:23 [&lt;ffffffff81013783&gt;] ? native_sched_clock+0x13/0x80 2012-03-18 06:05:23 [&lt;ffffffff81052e2b&gt;] task_tick_fair+0xdb/0x160 2012-03-18 06:05:23 [&lt;ffffffff81056891&gt;] scheduler_tick+0xc1/0x260 2012-03-18 06:05:23 [&lt;ffffffff810a0e70&gt;] ? tick_sched_timer+0x0/0xc0 2012-03-18 06:05:23 [&lt;ffffffff8107c512&gt;] update_process_times+0x52/0x70 2012-03-18 06:05:23 [&lt;ffffffff810a0ed6&gt;] tick_sched_timer+0x66/0xc0 2012-03-18 06:05:23 [&lt;ffffffff8109555e&gt;] __run_hrtimer+0x8e/0x1a0 2012-03-18 06:05:23 [&lt;ffffffff81012b59&gt;] ? read_tsc+0x9/0x20 2012-03-18 06:05:23 [&lt;ffffffff81095906&gt;] hrtimer_interrupt+0xe6/0x250 2012-03-18 06:05:23 [&lt;ffffffff814f6f0b&gt;] smp_apic_timer_interrupt+0x6b/0x9b 2012-03-18 06:05:23 [&lt;ffffffff8100bc13&gt;] apic_timer_interrupt+0x13/0x20 2012-03-18 06:05:23 &lt;EOI&gt; 2012-03-18 06:05:23 [&lt;ffffffff8112adbe&gt;] ? shrink_inactive_list+0x2de/0x740 2012-03-18 06:05:23 [&lt;ffffffff8112ad72&gt;] ? shrink_inactive_list+0x292/0x740 2012-03-18 06:05:23 [&lt;ffffffff8112baef&gt;] shrink_zone+0x38f/0x520 2012-03-18 06:05:23 [&lt;ffffffff8112c894&gt;] zone_reclaim+0x354/0x410 2012-03-18 06:05:23 [&lt;ffffffff8112d4e0&gt;] ? isolate_pages_global+0x0/0x350 2012-03-18 06:05:23 [&lt;ffffffff81122d94&gt;] get_page_from_freelist+0x694/0x820 2012-03-18 06:05:23 [&lt;ffffffffa0475b6f&gt;] ? cfs_hash_bd_from_key+0x3f/0xc0 [libcfs] 2012-03-18 06:05:23 [&lt;ffffffffa05d4232&gt;] ? lu_object_put+0x92/0x200 [obdclass] 2012-03-18 06:05:23 [&lt;ffffffff811337c9&gt;] ? zone_statistics+0x99/0xc0 2012-03-18 06:05:23 [&lt;ffffffff81124011&gt;] __alloc_pages_nodemask+0x111/0x940 2012-03-18 06:05:23 [&lt;ffffffff8115e592&gt;] kmem_getpages+0x62/0x170 2012-03-18 06:05:23 [&lt;ffffffff8115ebff&gt;] cache_grow+0x2cf/0x320 2012-03-18 06:05:23 [&lt;ffffffff8115ee52&gt;] cache_alloc_refill+0x202/0x240 2012-03-18 06:05:23 [&lt;ffffffffa0469863&gt;] ? cfs_alloc+0x63/0x90 [libcfs] 2012-03-18 06:05:23 [&lt;ffffffff8115fb79&gt;] __kmalloc+0x1a9/0x220 2012-03-18 06:05:23 [&lt;ffffffffa0469863&gt;] cfs_alloc+0x63/0x90 [libcfs] 2012-03-18 06:05:23 [&lt;ffffffffa0702237&gt;] ptlrpc_request_alloc_internal+0x167/0x360 [ptlrpc] 2012-03-18 06:05:23 [&lt;ffffffffa070243e&gt;] ptlrpc_request_alloc_pool+0xe/0x10 [ptlrpc] 2012-03-18 06:05:23 [&lt;ffffffffa08d684b&gt;] osc_brw_prep_request+0x1ab/0xcc0 [osc] 2012-03-18 06:05:23 [&lt;ffffffffa08e7fbb&gt;] ? osc_req_attr_set+0xfb/0x250 [osc] 2012-03-18 06:05:23 [&lt;ffffffffa09f7638&gt;] ? ccc_req_attr_set+0x78/0x150 [lustre] 2012-03-18 06:05:23 [&lt;ffffffffa05e4e81&gt;] ? cl_req_attr_set+0xd1/0x1a0 [obdclass] 2012-03-18 06:05:23 [&lt;ffffffffa05e476c&gt;] ? cl_req_prep+0x8c/0x130 [obdclass] 2012-03-18 06:05:23 [&lt;ffffffffa08d821d&gt;] osc_send_oap_rpc+0xebd/0x1690 [osc] 2012-03-18 06:05:23 [&lt;ffffffffa08d17a6&gt;] ? loi_list_maint+0xa6/0x130 [osc] 2012-03-18 06:05:23 [&lt;ffffffffa08d8c9e&gt;] osc_check_rpcs+0x2ae/0x3b0 [osc] 2012-03-18 06:05:23 [&lt;ffffffffa08e86df&gt;] osc_io_submit+0x1df/0x4a0 [osc] 2012-03-18 06:05:23 [&lt;ffffffffa05e4938&gt;] cl_io_submit_rw+0x78/0x130 [obdclass] 2012-03-18 06:05:23 [&lt;ffffffffa094160f&gt;] lov_io_submit+0x2bf/0x9d0 [lov] 2012-03-18 06:05:23 [&lt;ffffffffa05e4938&gt;] cl_io_submit_rw+0x78/0x130 [obdclass] 2012-03-18 06:05:23 [&lt;ffffffffa09ddf15&gt;] ll_writepage+0x205/0x3b0 [lustre] 2012-03-18 06:05:23 [&lt;ffffffff8112a34b&gt;] pageout.clone.1+0x12b/0x300 2012-03-18 06:05:23 [&lt;ffffffff81129a01&gt;] ? __remove_mapping+0x61/0x160 2012-03-18 06:05:23 [&lt;ffffffff8112a8d2&gt;] shrink_page_list.clone.0+0x3b2/0x5c0 2012-03-18 06:05:23 [&lt;ffffffff811280bc&gt;] ? release_pages+0x6c/0x250 2012-03-18 06:05:23 [&lt;ffffffff8116804d&gt;] ? mem_cgroup_del_lru_list+0x2d/0xc0 2012-03-18 06:05:23 [&lt;ffffffff81168169&gt;] ? mem_cgroup_del_lru+0x39/0x40 2012-03-18 06:05:23 [&lt;ffffffff811689ae&gt;] ? mem_cgroup_isolate_pages+0xee/0x1d0 2012-03-18 06:05:23 [&lt;ffffffff8112addb&gt;] shrink_inactive_list+0x2fb/0x740 2012-03-18 06:05:23 [&lt;ffffffff8112baef&gt;] shrink_zone+0x38f/0x520 2012-03-18 06:05:23 [&lt;ffffffff811a0e4d&gt;] ? bdi_queue_work+0x7d/0x110 2012-03-18 06:05:23 [&lt;ffffffff8112bd7e&gt;] do_try_to_free_pages+0xfe/0x520 2012-03-18 06:05:23 [&lt;ffffffff8112c2ec&gt;] try_to_free_mem_cgroup_pages+0x8c/0x90 2012-03-18 06:05:23 [&lt;ffffffff811688c0&gt;] ? mem_cgroup_isolate_pages+0x0/0x1d0 2012-03-18 06:05:23 [&lt;ffffffff8116a450&gt;] mem_cgroup_hierarchical_reclaim+0x2f0/0x460 2012-03-18 06:05:23 [&lt;ffffffff8116b992&gt;] __mem_cgroup_try_charge+0x202/0x460 2012-03-18 06:05:23 [&lt;ffffffff811241e7&gt;] ? __alloc_pages_nodemask+0x2e7/0x940 2012-03-18 06:05:23 [&lt;ffffffff8116c4e7&gt;] mem_cgroup_charge_common+0x87/0xd0 2012-03-18 06:05:23 [&lt;ffffffff8116c658&gt;] mem_cgroup_cache_charge+0x128/0x140 2012-03-18 06:05:23 [&lt;ffffffff8111161a&gt;] add_to_page_cache_locked+0x4a/0x140 2012-03-18 06:05:23 [&lt;ffffffff8111173c&gt;] add_to_page_cache_lru+0x2c/0x80 2012-03-18 06:05:23 [&lt;ffffffff811122f9&gt;] grab_cache_page_write_begin+0x99/0xc0 2012-03-18 06:05:23 [&lt;ffffffffa09efa98&gt;] ll_write_begin+0x58/0x170 [lustre] 2012-03-18 06:05:23 [&lt;ffffffff81111b7e&gt;] generic_file_buffered_write+0x10e/0x2a0 2012-03-18 06:05:23 [&lt;ffffffff81070637&gt;] ? current_fs_time+0x27/0x30 2012-03-18 06:05:23 [&lt;ffffffff811134d0&gt;] __generic_file_aio_write+0x250/0x480 2012-03-18 06:05:23 [&lt;ffffffffa05d8d95&gt;] ? cl_env_info+0x15/0x20 [obdclass] 2012-03-18 06:05:23 [&lt;ffffffff8111376f&gt;] generic_file_aio_write+0x6f/0xe0 2012-03-18 06:05:23 [&lt;ffffffffa09fdf9e&gt;] vvp_io_write_start+0x9e/0x1e0 [lustre] 2012-03-18 06:05:23 [&lt;ffffffffa05e4a5a&gt;] cl_io_start+0x6a/0x100 [obdclass] 2012-03-18 06:05:23 [&lt;ffffffffa05e8094&gt;] cl_io_loop+0xd4/0x160 [obdclass] 2012-03-18 06:05:23 [&lt;ffffffffa09ba43b&gt;] ll_file_io_generic+0x3bb/0x4b0 [lustre] 2012-03-18 06:05:23 [&lt;ffffffffa09bf4ec&gt;] ll_file_aio_write+0x13c/0x280 [lustre] 2012-03-18 06:05:23 [&lt;ffffffffa09bf79c&gt;] ll_file_write+0x16c/0x290 [lustre] 2012-03-18 06:05:23 [&lt;ffffffff81177ae8&gt;] vfs_write+0xb8/0x1a0 2012-03-18 06:05:23 [&lt;ffffffff811784f1&gt;] sys_write+0x51/0x90 2012-03-18 06:05:23 [&lt;ffffffff8100b0f2&gt;] system_call_fastpath+0x16/0x1b 2012-03-18 06:05:23 Code: 00 8b 15 cc 36 a4 00 85 d2 74 34 48 8b 50 08 8b 5a 18 48 8b 90 10 09 00 00 48 8b 4a 50 48 85 c9 74 1d 48 63 db 66 90 48 8b 51 20 &lt;48&gt; 03 14 dd 20 73 bf 81 4c 01 2a 48 8b 49 78 48 85 c9 75 e8 48 2012-03-18 06:05:23 RIP [&lt;ffffffff81052814&gt;] update_curr+0x144/0x1f0 2012-03-18 06:05:23 RSP &lt;ffff8800366c3db8&gt; 2012-03-18 06:05:23 CR2: 000000032cbc7960 </pre> </div></div> <a href="https://github.com/chaos/lustre/commits/2.1.1-llnl">https://github.com/chaos/lustre/commits/2.1.1-llnl</a> LU-1243

Lustre 2.1.1 client stack overrun

Bug Minor Resolved Cannot Reproduce Hongchao Zhang Ned Bass llnl Tue, 20 Mar 2012 13:33:20 +0000 Mon, 29 May 2017 03:36:21 +0000 Mon, 29 May 2017 03:36:20 +0000 Lustre 2.1.1 0 4 <p>Hongchao</p> <p>Could you please comment on this one?</p> <p>Thanks</p> <p>Peter</p> <p>it's similar with <a href="https://jira.whamcloud.com/browse/LU-969" title="2.1 client stack overruns" class="issue-link" data-issue-key="LU-969"><del>LU-969</del></a>, except this one goes a little further, we should use less stack space in our client IO layers.<br/> there are some places which can reduce the stack by ~100 bytes, will attach a patch soon, but the better solution should be<br/> moving datas in OSC layer into lu_env, e.g.</p> <p>static int osc_brw_prep_request(int cmd, struct client_obd *cli,struct obdo *oa,<br/> struct lov_stripe_md *lsm, obd_count page_count,<br/> struct brw_page **pga,<br/> struct ptlrpc_request **reqp,<br/> struct obd_capa *ocapa, int reserve,<br/> int resend)<br/> {<br/> struct ptlrpc_request *req;<br/> struct ptlrpc_bulk_desc *desc;<br/> struct ost_body *body;<br/> struct obd_ioobj *ioobj;<br/> struct niobuf_remote *niobuf;<br/> int niocount, i, requested_nob, opc, rc;<br/> struct osc_brw_async_args *aa;<br/> struct req_capsule *pill;<br/> struct brw_page *pg_prev;</p> <p> ENTRY;<br/> ...<br/> }</p> <p>the patch to move some data in OSC to lu_env is under creation&amp;test, and will attach it ASAP</p> <p>status update:<br/> moving variables into lu_env is of little usage to reduce stack, and a new patch with a separated patch to split the IO<br/> in OSC layer(mainly call osc_check_rpcs) is under test, and it encounters several issues during testing, then it will<br/> need a little more time to debug it before attaching it.</p> <p>the patch is tracked at <a href="http://review.whamcloud.com/#change,2418" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,2418</a></p> <p>by moving the variables into lu_env, the stack reduction is not obvious, the attached patch is using an separated thread<br/> to call "osc_check_rpcs" to flush the pending IO of OSCs, and using this specific thread only if the stack usage of current<br/> thread is too much (more than 3 *THREAD_SIZE / 4), then it should not affect the whole IO performance comparing to previous <br/> case with many threads doing IO</p> <p>We already have threads that send RPCs - that is ptlrpcd. Adding more threads in the very common IO path just to reduce stack usage will increase complexity and hurt performance due to extra context switches. We already have to do work to improve the NUMA CPU page locality for multiple ptlrpcds, and we would need to repeat this work for this new OSC thread to keep the page handling local to the CPU on which the pages were previously handled.</p> <p>It looks like the main problem is that the write path is going into kernel memory allocation/reclamation in ptlrpc_request_alloc_pool(), but this thread is already in the memory reclaim path from the top-level write:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>ll_file_write -&gt;generic_file_buffered_write -&gt;ll_write_begin -&gt;grab_cache_page_write_begin -&gt;do_try_to_free_pages -&gt;shrink_zone -&gt;shrink_page_list -&gt;ll_writepage -&gt;osc_send_oap_rpc -&gt;ptlrpc_request_alloc_pool -&gt;cfs_alloc -&gt;get_page_from_freelist -&gt;zone_reclaim -&gt;shrink_zone </pre> </div></div> <p>so adding another thread will not help very much because there will still be memory pressure, but it will slow down EVERY IO operation. A much better approach is to figure out why the ptlrpc request pool is not being used for this write request. It is entirely possible that the changes in 2.x for CLIO broke the initialization or usage of the ptlrpc request pool. This was added specifically to avoid memory allocations in the write path, to avoid memory allocation under pressure.</p> <p>Integrated in <span class="image-wrap" style=""><img src="http://iu-build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://iu-build.whamcloud.com/job/lustre-reviews/./arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel/4589/" class="external-link" target="_blank" rel="nofollow noopener">lustre-reviews » x86_64,client,el6,inkernel #4589</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1243" title="Lustre 2.1.1 client stack overrun" class="issue-link" data-issue-key="LU-1243"><del>LU-1243</del></a> osc: use separated thread for IO to reduce stack (Revision 4ce80fb0405bc88a69456a3e2a33d21b6fed3977)</p> <p> Result = SUCCESS<br/> hongchao.zhang : <a href="http://git.whamcloud.com/gitweb/?p=fs/lustre-release.git&amp;a=commit&amp;h=4ce80fb0405bc88a69456a3e2a33d21b6fed3977" class="external-link" target="_blank" rel="nofollow noopener">4ce80fb0405bc88a69456a3e2a33d21b6fed3977</a><br/> Files : </p> <ul> <li>lustre/obdclass/lprocfs_status.c</li> <li>lustre/osc/osc_request.c</li> <li>lustre/lov/lov_obd.c</li> <li>lustre/include/obd.h</li> <li>lustre/include/obd_class.h</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://iu-build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://iu-build.whamcloud.com/job/lustre-reviews/./arch=i686,build_type=client,distro=el6,ib_stack=inkernel/4589/" class="external-link" target="_blank" rel="nofollow noopener">lustre-reviews » i686,client,el6,inkernel #4589</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1243" title="Lustre 2.1.1 client stack overrun" class="issue-link" data-issue-key="LU-1243"><del>LU-1243</del></a> osc: use separated thread for IO to reduce stack (Revision 4ce80fb0405bc88a69456a3e2a33d21b6fed3977)</p> <p> Result = SUCCESS<br/> hongchao.zhang : <a href="http://git.whamcloud.com/gitweb/?p=fs/lustre-release.git&amp;a=commit&amp;h=4ce80fb0405bc88a69456a3e2a33d21b6fed3977" class="external-link" target="_blank" rel="nofollow noopener">4ce80fb0405bc88a69456a3e2a33d21b6fed3977</a><br/> Files : </p> <ul> <li>lustre/lov/lov_obd.c</li> <li>lustre/include/obd.h</li> <li>lustre/osc/osc_request.c</li> <li>lustre/include/obd_class.h</li> <li>lustre/obdclass/lprocfs_status.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://iu-build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://iu-build.whamcloud.com/job/lustre-reviews/./arch=x86_64,build_type=client,distro=el5,ib_stack=inkernel/4589/" class="external-link" target="_blank" rel="nofollow noopener">lustre-reviews » x86_64,client,el5,inkernel #4589</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1243" title="Lustre 2.1.1 client stack overrun" class="issue-link" data-issue-key="LU-1243"><del>LU-1243</del></a> osc: use separated thread for IO to reduce stack (Revision 4ce80fb0405bc88a69456a3e2a33d21b6fed3977)</p> <p> Result = SUCCESS<br/> hongchao.zhang : <a href="http://git.whamcloud.com/gitweb/?p=fs/lustre-release.git&amp;a=commit&amp;h=4ce80fb0405bc88a69456a3e2a33d21b6fed3977" class="external-link" target="_blank" rel="nofollow noopener">4ce80fb0405bc88a69456a3e2a33d21b6fed3977</a><br/> Files : </p> <ul> <li>lustre/osc/osc_request.c</li> <li>lustre/include/obd_class.h</li> <li>lustre/obdclass/lprocfs_status.c</li> <li>lustre/include/obd.h</li> <li>lustre/lov/lov_obd.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://iu-build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://iu-build.whamcloud.com/job/lustre-reviews/./arch=i686,build_type=server,distro=el5,ib_stack=inkernel/4589/" class="external-link" target="_blank" rel="nofollow noopener">lustre-reviews » i686,server,el5,inkernel #4589</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1243" title="Lustre 2.1.1 client stack overrun" class="issue-link" data-issue-key="LU-1243"><del>LU-1243</del></a> osc: use separated thread for IO to reduce stack (Revision 4ce80fb0405bc88a69456a3e2a33d21b6fed3977)</p> <p> Result = SUCCESS<br/> hongchao.zhang : <a href="http://git.whamcloud.com/gitweb/?p=fs/lustre-release.git&amp;a=commit&amp;h=4ce80fb0405bc88a69456a3e2a33d21b6fed3977" class="external-link" target="_blank" rel="nofollow noopener">4ce80fb0405bc88a69456a3e2a33d21b6fed3977</a><br/> Files : </p> <ul> <li>lustre/obdclass/lprocfs_status.c</li> <li>lustre/include/obd_class.h</li> <li>lustre/lov/lov_obd.c</li> <li>lustre/include/obd.h</li> <li>lustre/osc/osc_request.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://iu-build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://iu-build.whamcloud.com/job/lustre-reviews/./arch=x86_64,build_type=server,distro=el6,ib_stack=inkernel/4589/" class="external-link" target="_blank" rel="nofollow noopener">lustre-reviews » x86_64,server,el6,inkernel #4589</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1243" title="Lustre 2.1.1 client stack overrun" class="issue-link" data-issue-key="LU-1243"><del>LU-1243</del></a> osc: use separated thread for IO to reduce stack (Revision 4ce80fb0405bc88a69456a3e2a33d21b6fed3977)</p> <p> Result = SUCCESS<br/> hongchao.zhang : <a href="http://git.whamcloud.com/gitweb/?p=fs/lustre-release.git&amp;a=commit&amp;h=4ce80fb0405bc88a69456a3e2a33d21b6fed3977" class="external-link" target="_blank" rel="nofollow noopener">4ce80fb0405bc88a69456a3e2a33d21b6fed3977</a><br/> Files : </p> <ul> <li>lustre/obdclass/lprocfs_status.c</li> <li>lustre/osc/osc_request.c</li> <li>lustre/include/obd.h</li> <li>lustre/lov/lov_obd.c</li> <li>lustre/include/obd_class.h</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://iu-build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://iu-build.whamcloud.com/job/lustre-reviews/./arch=i686,build_type=server,distro=el6,ib_stack=inkernel/4589/" class="external-link" target="_blank" rel="nofollow noopener">lustre-reviews » i686,server,el6,inkernel #4589</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1243" title="Lustre 2.1.1 client stack overrun" class="issue-link" data-issue-key="LU-1243"><del>LU-1243</del></a> osc: use separated thread for IO to reduce stack (Revision 4ce80fb0405bc88a69456a3e2a33d21b6fed3977)</p> <p> Result = SUCCESS<br/> hongchao.zhang : <a href="http://git.whamcloud.com/gitweb/?p=fs/lustre-release.git&amp;a=commit&amp;h=4ce80fb0405bc88a69456a3e2a33d21b6fed3977" class="external-link" target="_blank" rel="nofollow noopener">4ce80fb0405bc88a69456a3e2a33d21b6fed3977</a><br/> Files : </p> <ul> <li>lustre/osc/osc_request.c</li> <li>lustre/lov/lov_obd.c</li> <li>lustre/include/obd_class.h</li> <li>lustre/include/obd.h</li> <li>lustre/obdclass/lprocfs_status.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://iu-build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://iu-build.whamcloud.com/job/lustre-reviews/./arch=x86_64,build_type=server,distro=el5,ib_stack=inkernel/4589/" class="external-link" target="_blank" rel="nofollow noopener">lustre-reviews » x86_64,server,el5,inkernel #4589</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1243" title="Lustre 2.1.1 client stack overrun" class="issue-link" data-issue-key="LU-1243"><del>LU-1243</del></a> osc: use separated thread for IO to reduce stack (Revision 4ce80fb0405bc88a69456a3e2a33d21b6fed3977)</p> <p> Result = SUCCESS<br/> hongchao.zhang : <a href="http://git.whamcloud.com/gitweb/?p=fs/lustre-release.git&amp;a=commit&amp;h=4ce80fb0405bc88a69456a3e2a33d21b6fed3977" class="external-link" target="_blank" rel="nofollow noopener">4ce80fb0405bc88a69456a3e2a33d21b6fed3977</a><br/> Files : </p> <ul> <li>lustre/osc/osc_request.c</li> <li>lustre/include/obd_class.h</li> <li>lustre/obdclass/lprocfs_status.c</li> <li>lustre/include/obd.h</li> <li>lustre/lov/lov_obd.c</li> </ul> <p>Integrated in <span class="image-wrap" style=""><img src="http://iu-build.whamcloud.com/images/16x16/blue.png" style="border: 0px solid black" /></span> <a href="http://iu-build.whamcloud.com/job/lustre-reviews/./arch=i686,build_type=client,distro=el5,ib_stack=inkernel/4589/" class="external-link" target="_blank" rel="nofollow noopener">lustre-reviews » i686,client,el5,inkernel #4589</a><br/> <a href="https://jira.whamcloud.com/browse/LU-1243" title="Lustre 2.1.1 client stack overrun" class="issue-link" data-issue-key="LU-1243"><del>LU-1243</del></a> osc: use separated thread for IO to reduce stack (Revision 4ce80fb0405bc88a69456a3e2a33d21b6fed3977)</p> <p> Result = SUCCESS<br/> hongchao.zhang : <a href="http://git.whamcloud.com/gitweb/?p=fs/lustre-release.git&amp;a=commit&amp;h=4ce80fb0405bc88a69456a3e2a33d21b6fed3977" class="external-link" target="_blank" rel="nofollow noopener">4ce80fb0405bc88a69456a3e2a33d21b6fed3977</a><br/> Files : </p> <ul> <li>lustre/osc/osc_request.c</li> <li>lustre/lov/lov_obd.c</li> <li>lustre/include/obd_class.h</li> <li>lustre/obdclass/lprocfs_status.c</li> <li>lustre/include/obd.h</li> </ul> <p>1, the stack used by osc_check_rpcs0/osc_send_oap_rpc/osc_build_req/osc_brw_prep_request is ~350bytes<br/> (only account local variables, not include arguments).</p> <p>2, the ptlrpc_request is called by osc_check_rpcs, which can't guarantee it must to be write request, and for read request,<br/> it will don't use the pool</p> <p> cfs_alloc+0x63/0x90 <span class="error">&#91;libcfs&#93;</span><br/> ptlrpc_request_alloc_internal+0x167/0x360 <span class="error">&#91;ptlrpc&#93;</span><br/> ptlrpc_request_alloc_pool+0xe/0x10 <span class="error">&#91;ptlrpc&#93;</span><br/> osc_brw_prep_request+0x1ab/0xcc0 <span class="error">&#91;osc&#93;</span><br/> ? osc_req_attr_set+0xfb/0x250 <span class="error">&#91;osc&#93;</span><br/> ? ccc_req_attr_set+0x78/0x150 <span class="error">&#91;lustre&#93;</span><br/> ? cl_req_attr_set+0xd1/0x1a0 <span class="error">&#91;obdclass&#93;</span><br/> ? cl_req_prep+0x8c/0x130 <span class="error">&#91;obdclass&#93;</span><br/> osc_send_oap_rpc+0xebd/0x1690 <span class="error">&#91;osc&#93;</span><br/> ? loi_list_maint+0xa6/0x130 <span class="error">&#91;osc&#93;</span><br/> osc_check_rpcs+0x2ae/0x3b0 <span class="error">&#91;osc&#93;</span><br/> osc_io_submit+0x1df/0x4a0 <span class="error">&#91;osc&#93;</span><br/> cl_io_submit_rw+0x78/0x130 <span class="error">&#91;obdclass&#93;</span><br/> lov_io_submit+0x2bf/0x9d0 <span class="error">&#91;lov&#93;</span><br/> cl_io_submit_rw+0x78/0x130 <span class="error">&#91;obdclass&#93;</span><br/> ll_writepage+0x205/0x3b0 <span class="error">&#91;lustre&#93;</span></p> <p>static int osc_brw_prep_request(int cmd, struct client_obd *cli,struct obdo *oa, ...<br/> {<br/> ...<br/> if ((cmd &amp; OBD_BRW_WRITE) != 0) </p> { opc = OST_WRITE; req = ptlrpc_request_alloc_pool(cli-&gt;cl_import, cli-&gt;cl_import-&gt;imp_rq_pool, &amp;RQF_OST_BRW_WRITE); } <p> else </p> { opc = OST_READ; req = ptlrpc_request_alloc(cli-&gt;cl_import, &amp;RQF_OST_BRW_READ); } <p> ...<br/> }</p> <p>struct ptlrpc_request *ptlrpc_request_alloc(struct obd_import *imp,<br/> const struct req_format *format)<br/> { <br/> return ptlrpc_request_alloc_internal(imp, NULL, format);<br/> }</p> <p>3, the ptlrpcd is for async io, the new introduced thread will block the calling thread until it finish, which is better?</p> <p>4, the separated thread will only be used if the calling thread is under low stack space, and will it also affect IO much?</p> <p>Even in the case of read operations at the top level, if the free memory is low and the VM is trying to flush dirty data from cache, it will be a write operation at the bottom of the stack that will use the RPC request pool.</p> <p>Getting the RPC request pool working will not only (hopefully) fix this stack usage issue, but also has the benefit of avoiding memory allocation deadlocks when the client and server are running on the same machine, and avoids making large RPC request allocations frequenty (which is why it was added originally). So there is a benefit in getting this code working properly for several reasons, not just the stack usage. Also, it is best to avoid making the code more complex (i.e. adding another thread) if this is not necessary.</p> <p>I'd like to suggest to issue BRW RPC asynchronously so that we can reduce stack usage dramatically.</p> <p>I have added the functionality to issue async RPC at commit 2064c2a7e616b172f72462884b23d899bfc040ff so I think we can port it to b2_1 and fire writeback request in osc_io_submit() too.</p> <p>To best honest, obd interfaces are almost obsoleted in client IO so any attempt to add new obd ops is not good.</p> <p>the updated patch based on the above async PTLRPC is under creation&amp;testing.</p> <p>the updated patch is under testing and will attach it soon.</p> <p><a href="http://review.whamcloud.com/#change,2418" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,2418</a></p> <p>after rebasing the patch, two new problems are found, one is related to IO requests advancing, the IO performance is very<br/> slow during testing sanity, another is a deadlock issue during testing sanity sub-test 42e</p> <p>the updated test is under testing, hope to attach it tomorrow.</p> <p>there are much change in OSC module recently, and the patch needed to be updated against it.</p> <p>Close old ticket.</p> Development End date Fri, 27 Jun 2014 13:33:20 +0000 Rank 1|hzvyav: Rank (Obsolete) 9741 Severity Start date Tue, 20 Mar 2012 13:33:20 +0000