https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3239 Lustre <p>Lustre: DEBUG MARKER: == parallel-scale test simul: simul == 22:08:33 (1367125713)<br/> LustreError: 19190:0:(ofd_internal.h:518:ofd_info_init()) ASSERTION( info ) failed: <br/> LustreError: 19182:0:(ofd_internal.h:518:ofd_info_init()) ASSERTION( info ) failed: <br/> LustreError: 19182:0:(ofd_internal.h:518:ofd_info_init()) LBUG<br/> Pid: 19182, comm: ll_ost_io00_001</p> <p>Call Trace:<br/> <span class="error">&#91;&lt;ffffffffa044e895&gt;&#93;</span> libcfs_debug_dumpstack+0x55/0x80 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa044ee97&gt;&#93;</span> lbug_with_loc+0x47/0xb0 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0e03e62&gt;&#93;</span> ofd_info_init+0x92/0x130 <span class="error">&#91;ofd&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0e05835&gt;&#93;</span> ofd_get_info+0x2e5/0xa90 <span class="error">&#91;ofd&#93;</span><br/> <span class="error">&#91;&lt;ffffffff812805cd&gt;&#93;</span> ? pointer+0x8d/0x830<br/> <span class="error">&#91;&lt;ffffffffa029f7e5&gt;&#93;</span> ? lprocfs_counter_add+0x125/0x182 <span class="error">&#91;lvfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa078528a&gt;&#93;</span> nrs_orr_range_fill_physical+0x18a/0x540 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0762dd6&gt;&#93;</span> ? __req_capsule_get+0x166/0x700 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa073e630&gt;&#93;</span> ? lustre_swab_ost_body+0x0/0x10 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa07871d7&gt;&#93;</span> nrs_orr_res_get+0x817/0xb80 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa077d306&gt;&#93;</span> nrs_resource_get+0x56/0x110 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa077dccb&gt;&#93;</span> nrs_resource_get_safe+0x8b/0x100 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0780248&gt;&#93;</span> ptlrpc_nrs_req_initialize+0x38/0x90 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa074cff0&gt;&#93;</span> ptlrpc_main+0x1170/0x16f0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa074be80&gt;&#93;</span> ? ptlrpc_main+0x0/0x16f0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff8100c0ca&gt;&#93;</span> child_rip+0xa/0x20<br/> <span class="error">&#91;&lt;ffffffffa074be80&gt;&#93;</span> ? ptlrpc_main+0x0/0x16f0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa074be80&gt;&#93;</span> ? ptlrpc_main+0x0/0x16f0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff8100c0c0&gt;&#93;</span> ? child_rip+0x0/0x20</p> <p>LustreError: dumping log to /tmp/lustre-log.1367125716.19182<br/> LustreError: 19190:0:(ofd_internal.h:518:ofd_info_init()) LBUG<br/> Pid: 19190, comm: ll_ost_io03_000</p> <p>Call Trace:<br/> <span class="error">&#91;&lt;ffffffffa044e895&gt;&#93;</span> libcfs_debug_dumpstack+0x55/0x80 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa044ee97&gt;&#93;</span> lbug_with_loc+0x47/0xb0 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0e03e62&gt;&#93;</span> ofd_info_init+0x92/0x130 <span class="error">&#91;ofd&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0e05835&gt;&#93;</span> ofd_get_info+0x2e5/0xa90 <span class="error">&#91;ofd&#93;</span><br/> <span class="error">&#91;&lt;ffffffff812805cd&gt;&#93;</span> ? pointer+0x8d/0x830<br/> <span class="error">&#91;&lt;ffffffffa029f7e5&gt;&#93;</span> ? lprocfs_counter_add+0x125/0x182 <span class="error">&#91;lvfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa078528a&gt;&#93;</span> nrs_orr_range_fill_physical+0x18a/0x540 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0762dd6&gt;&#93;</span> ? __req_capsule_get+0x166/0x700 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa073e630&gt;&#93;</span> ? lustre_swab_ost_body+0x0/0x10 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa07871d7&gt;&#93;</span> nrs_orr_res_get+0x817/0xb80 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa077d306&gt;&#93;</span> nrs_resource_get+0x56/0x110 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa077dccb&gt;&#93;</span> nrs_resource_get_safe+0x8b/0x100 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0780248&gt;&#93;</span> ptlrpc_nrs_req_initialize+0x38/0x90 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa074cff0&gt;&#93;</span> ptlrpc_main+0x1170/0x16f0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa074be80&gt;&#93;</span> ? ptlrpc_main+0x0/0x16f0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff8100c0ca&gt;&#93;</span> child_rip+0xa/0x20<br/> <span class="error">&#91;&lt;ffffffffa074be80&gt;&#93;</span> ? ptlrpc_main+0x0/0x16f0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa074be80&gt;&#93;</span> ? ptlrpc_main+0x0/0x16f0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff8100c0c0&gt;&#93;</span> ? child_rip+0x0/0x20</p> <p>LustreError: dumping log to /tmp/lustre-log.1367125719.19190<br/> LNet: Service thread pid 19182 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:<br/> Pid: 19182, comm: ll_ost_io00_001</p> <p>I set nrs on mds to crrn, oss02 to orr, oss03 to trr and run parallel-scale</p> <a href="https://maloo.whamcloud.com/test_sets/0e21f9ba-afec-11e2-b8a3-52540035b04c">https://maloo.whamcloud.com/test_sets/0e21f9ba-afec-11e2-b8a3-52540035b04c</a> LU-3239

ofd_internal.h:518:ofd_info_init()) ASSERTION( info )

Bug Blocker Resolved Fixed Di Wang Minh Diep LB Sun, 28 Apr 2013 10:16:52 +0000 Thu, 2 May 2013 10:39:17 +0000 Thu, 2 May 2013 10:39:17 +0000 Lustre 2.4.0 Lustre 2.4.0 0 9 <p>Possibly related similar stack trace but from next ASSERTION line (sanityn test_71):</p> <p><a href="https://maloo.whamcloud.com/test_sets/4c4286e0-b0de-11e2-bece-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/4c4286e0-b0de-11e2-bece-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/f78b8b88-b100-11e2-b2c4-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/f78b8b88-b100-11e2-b2c4-52540035b04c</a></p> <p>It seems to me, NRS makes get_info step into OFD before it is initialized. </p> <p>Alex, should the OFD environment already be initialized by this point? Is there something that can easily be done to avoid this problem?</p> <p>IMHO, we can simply add lu_env_refill in ofd_get_info. I will cook a patch now.</p> <p><a href="http://review.whamcloud.com/6204" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/6204</a> </p> <p>Wang Di,</p> <p>I hit this after running your patch</p> <p>general protection fault: 0000 <a href="#1" target="_blank" rel="noopener">1</a> SMP ^M<br/> last sysfs file: /sys/devices/pci0000:80/0000:80:05.0/0000:84:00.0/local_cpus^M<br/> CPU 0 ^M<br/> Modules linked in: osp(U) ofd(U) ost(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) lquota(U) jbd2 mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) sha512_generic sha256_generic libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 microcode serio_raw mlx4_ib ib_sa ib_mad ib_core mlx4_en mlx4_core i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma i7core_edac edac_core ses enclosure sg igb dca ptp pps_core ext3 jbd mbcache sr_mod cdrom sd_mod crc_t10dif pata_acpi ata_generic ata_piix mpt2sas scsi_transport_sas raid_class dm_mirror dm_region_hash dm_log dm_mod <span class="error">&#91;last unloaded: scsi_wait_scan&#93;</span>^M<br/> ^M<br/> Pid: 13927, comm: ll_ost_io00_003 Not tainted 2.6.32-358.6.1.el6_lustre.gce942d0.x86_64 #1 Supermicro X8DTH-i/6/iF/6F/X8DTH^M<br/> RIP: 0010:<span class="error">&#91;&lt;ffffffffa05aa678&gt;&#93;</span> <span class="error">&#91;&lt;ffffffffa05aa678&gt;&#93;</span> keys_fill+0x48/0x190 <span class="error">&#91;obdclass&#93;</span>^M<br/> RSP: 0018:ffff8806a9a4d9c0 EFLAGS: 00010246^M<br/> RAX: 5a5a5a5a5a5a5a5a RBX: 0000000000000000 RCX: 0000000000000000^M<br/> RDX: 00000000000000f8 RSI: ffff8806a9a4dbd6 RDI: ffff88069d88af40^M<br/> RBP: ffff8806a9a4da00 R08: 0000000000000000 R09: ffff8806a9a4dcd0^M<br/> R10: 000000000000000f R11: 0000000000000001 R12: ffff88069d88af40^M<br/> R13: ffffffffa0613a00 R14: ffff8806a9a4dbd0 R15: ffff880fb6667800^M<br/> FS: 00007f4b3d486700(0000) GS:ffff880028200000(0000) knlGS:0000000000000000^M<br/> CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b^M<br/> CR2: 00000000006d4420 CR3: 00000010335b5000 CR4: 00000000000007f0^M<br/> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000^M<br/> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400^M<br/> Process ll_ost_io00_003 (pid: 13927, threadinfo ffff8806a9a4c000, task ffff8806b04deaa0)^M<br/> Stack:^M<br/> ffffffffffffffff 0000000000000000 0000000000000001 ffff8806ae9a06c0^M<br/> &lt;d&gt; ffff880fccc03000 0000000000000000 ffff8806a9a4dbd0 ffff880fb6667800^M<br/> &lt;d&gt; ffff8806a9a4da10 ffffffffa05aa7dd ffff8806a9a4da30 ffffffffa05aa818^M<br/> Call Trace:^M<br/> <span class="error">&#91;&lt;ffffffffa05aa7dd&gt;&#93;</span> lu_context_refill+0x1d/0x30 <span class="error">&#91;obdclass&#93;</span>^M<br/> <span class="error">&#91;&lt;ffffffffa05aa818&gt;&#93;</span> lu_env_refill+0x28/0x30 <span class="error">&#91;obdclass&#93;</span>^M<br/> <span class="error">&#91;&lt;ffffffffa0e06852&gt;&#93;</span> ofd_get_info+0x302/0xba0 <span class="error">&#91;ofd&#93;</span>^M<br/> <span class="error">&#91;&lt;ffffffff812805fd&gt;&#93;</span> ? pointer+0x8d/0x830^M<br/> <span class="error">&#91;&lt;ffffffffa029f7e5&gt;&#93;</span> ? lprocfs_counter_add+0x125/0x182 <span class="error">&#91;lvfs&#93;</span>^M<br/> <span class="error">&#91;&lt;ffffffffa078528a&gt;&#93;</span> nrs_orr_range_fill_physical+0x18a/0x540 <span class="error">&#91;ptlrpc&#93;</span>^M<br/> <span class="error">&#91;&lt;ffffffffa0762dd6&gt;&#93;</span> ? __req_capsule_get+0x166/0x700 <span class="error">&#91;ptlrpc&#93;</span>^M<br/> <span class="error">&#91;&lt;ffffffffa073e630&gt;&#93;</span> ? lustre_swab_ost_body+0x0/0x10 <span class="error">&#91;ptlrpc&#93;</span>^M<br/> <span class="error">&#91;&lt;ffffffffa07871c6&gt;&#93;</span> nrs_orr_res_get+0x806/0xb90 <span class="error">&#91;ptlrpc&#93;</span>^M<br/> <span class="error">&#91;&lt;ffffffffa077d306&gt;&#93;</span> nrs_resource_get+0x56/0x110 <span class="error">&#91;ptlrpc&#93;</span>^M<br/> <span class="error">&#91;&lt;ffffffffa077dccb&gt;&#93;</span> nrs_resource_get_safe+0x8b/0x100 <span class="error">&#91;ptlrpc&#93;</span>^M<br/> <span class="error">&#91;&lt;ffffffffa0780248&gt;&#93;</span> ptlrpc_nrs_req_initialize+0x38/0x90 <span class="error">&#91;ptlrpc&#93;</span>^M<br/> <span class="error">&#91;&lt;ffffffffa074cff0&gt;&#93;</span> ptlrpc_main+0x1170/0x16f0 <span class="error">&#91;ptlrpc&#93;</span>^M<br/> <span class="error">&#91;&lt;ffffffffa074be80&gt;&#93;</span> ? ptlrpc_main+0x0/0x16f0 <span class="error">&#91;ptlrpc&#93;</span>^M<br/> <span class="error">&#91;&lt;ffffffff8100c0ca&gt;&#93;</span> child_rip+0xa/0x20^M<br/> <span class="error">&#91;&lt;ffffffffa074be80&gt;&#93;</span> ? ptlrpc_main+0x0/0x16f0 <span class="error">&#91;ptlrpc&#93;</span>^M<br/> <span class="error">&#91;&lt;ffffffffa074be80&gt;&#93;</span> ? ptlrpc_main+0x0/0x16f0 <span class="error">&#91;ptlrpc&#93;</span>^M<br/> <span class="error">&#91;&lt;ffffffff8100c0c0&gt;&#93;</span> ? child_rip+0x0/0x20^M</p> <p>This is not about missed keys but non-initialized context at all because obd_get_info() is called not from ptlrpc_server_handle_request(). We need all lu_context setup things in ptlrpc_server_handle_req_in like that is done in ptlrpc_server_handle_request() since ptlrpc_server_handle_req_in() can cause calls to device stack now.</p> <p>yes, I forgot this is in handle_req_in, instead of handle_request.</p> <p>I just updated the patch. To avoid the hassle in handle_req_in, ofd_get_info(KEY_FIEMAP) will initialize the env itself, since ofd_get_info(KEY_FIEMAP) is the only use case from handle_req_in, adding env initialization in handle_req_in might not worth for now. </p> <p>I tend to agree, if this is the only single case.</p> <p>we do already have env in ptlrpc_main()..</p> <p>well, those envs might be initialized too early before ofd stack setup. that is why we do env_refill in ost_handle? </p> <p>env_refill() is fine, but recreation env from scratch on potentially every read/write is not, i think.</p> <p>hmm, only for ofd_get_info, because get_info is the only use case from handle_request_in, where env(le_ses) is not initialized correctly. so I figured initializing env(le_ses) for all requests in req_in might not worth. Only do this env_init in ofd_get_info should be enough. Besides we can not use env_refill here, because le_ses might be in some uninitialized state, you can see from Minh's test. Hmm, I did not see ofd_get_info is called in the process of read/write. Do I miss sth? </p> <p>get_info(KEY_FIEMAP) is used to order requests with regard physical offset, AFAIK.</p> <p>hmm, yes, probably env_refill should be better. I change it back to env_refill here in the update patch.</p> <p>Please do not close this bug until a test that runs at least basic NRS policy functionality.</p> <p>Landed for 2.4</p> Blocker LU-398 Development Rank 1|hzvp7r: Rank (Obsolete) 7954 Severity