Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 [LU-2652] lov_io.c:222:lov_sub_get()) ASSERTION( stripe < lio->lis_stripe_count ) failed https://jira.whamcloud.com/browse/LU-2652 Lustre <p>HAd this crash running sanityn test 12:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[45366.990436] Lustre: DEBUG MARKER: == sanityn test 12: test lock ordering (link, stat, unlink) ============= 10:01:13 (1358607673) [45367.146837] Lustre: DEBUG MARKER: start dir: /mnt/lustre/lockdir=144115205255725085 file: /mnt/lustre/lockdir/lockfile=144115205255725083 [45445.727378] LustreError: 20512:0:(lov_io.c:222:lov_sub_get()) ASSERTION( stripe &lt; lio-&gt;lis_stripe_count ) failed: [45445.728276] LustreError: 20512:0:(lov_io.c:222:lov_sub_get()) LBUG [45445.728751] Pid: 20512, comm: statmany [45445.729089] [45445.729090] Call Trace: [45445.729696] [&lt;ffffffffa0a7a915&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] [45445.730184] [&lt;ffffffffa0a7af17&gt;] lbug_with_loc+0x47/0xb0 [libcfs] [45445.730673] [&lt;ffffffffa14041ba&gt;] lov_sub_get+0x4aa/0x690 [lov] [45445.731129] [&lt;ffffffffa1400132&gt;] lov_sublock_env_get+0xd2/0x140 [lov] [45445.731612] [&lt;ffffffffa1401841&gt;] lov_sublock_alloc+0xf1/0x450 [lov] [45445.732086] [&lt;ffffffffa1402a2c&gt;] lov_lock_init_raid0+0x3ec/0xe50 [lov] [45445.732632] [&lt;ffffffffa13fa2ae&gt;] lov_lock_init+0x1e/0x60 [lov] [45445.733135] [&lt;ffffffffa0bec52c&gt;] cl_lock_hold_mutex+0x34c/0x660 [obdclass] [45445.733652] [&lt;ffffffffa0bec9a2&gt;] cl_lock_request+0x62/0x270 [obdclass] [45445.734143] [&lt;ffffffffa090b1f9&gt;] cl_glimpse_lock+0x179/0x480 [lustre] [45445.734635] [&lt;ffffffffa090ba65&gt;] cl_glimpse_size0+0x1a5/0x1d0 [lustre] [45445.735142] [&lt;ffffffffa08c5298&gt;] ll_inode_revalidate_it+0x198/0x1c0 [lustre] [45445.735748] [&lt;ffffffffa08c5309&gt;] ll_getattr_it+0x49/0x170 [lustre] [45445.736230] [&lt;ffffffffa08c5467&gt;] ll_getattr+0x37/0x40 [lustre] [45445.736719] [&lt;ffffffff81214fa3&gt;] ? security_inode_getattr+0x23/0x30 [45445.737183] [&lt;ffffffff81180891&gt;] vfs_getattr+0x51/0x80 [45445.737613] [&lt;ffffffff810385d8&gt;] ? pvclock_clocksource_read+0x58/0xd0 [45445.738070] [&lt;ffffffff81180920&gt;] vfs_fstatat+0x60/0x80 [45445.738508] [&lt;ffffffff81180a6b&gt;] vfs_stat+0x1b/0x20 [45445.738919] [&lt;ffffffff81180a94&gt;] sys_newstat+0x24/0x50 [45445.739332] [&lt;ffffffff8109b05a&gt;] ? do_gettimeofday+0x1a/0x50 [45445.740967] [&lt;ffffffff8100c5b5&gt;] ? math_state_restore+0x45/0x60 [45445.741415] [&lt;ffffffff814fb72e&gt;] ? do_device_not_available+0xe/0x10 [45445.741875] [&lt;ffffffff8100befb&gt;] ? device_not_available+0x1b/0x20 [45445.742317] [&lt;ffffffff8100b0f2&gt;] system_call_fastpath+0x16/0x1b [45445.742763] [45445.744348] Kernel panic - not syncing: LBUG </pre> </div></div> <p>Crashdump is in /exports/crashdumps/192.168.10.217-2013-01-19-10\:02\:36</p> LU-2652 lov_io.c:222:lov_sub_get()) ASSERTION( stripe < lio->lis_stripe_count ) failed Bug Blocker Resolved Duplicate Jinshan Xiong Oleg Drokin Sat, 19 Jan 2013 14:44:44 +0000 Fri, 5 Jul 2013 14:12:05 +0000 Sun, 30 Jun 2013 18:59:06 +0000 Lustre 2.4.0 Lustre 2.4.0 0 5 <p>Sometimes instead of the crash I am getting this sort of a locked up thread:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>.111182] Lustre: DEBUG MARKER: == sanityn test 12: test lock ordering (link, sta .378607] Lustre: DEBUG MARKER: start dir: /mnt/lustre/lockdir=144115205255725085 .812526] LNet: Service thread pid 12451 was inactive for 40.00s. The thread migh .813279] Pid: 12451, comm: mdt00_001 .813563] .813564] Call Trace: .813979] [&lt;ffffffff814fb054&gt;] ? _spin_lock_irqsave+0x24/0x30 .814234] [&lt;ffffffff814f8ad1&gt;] schedule_timeout+0x191/0x2e0 .814507] [&lt;ffffffff8107bcd0&gt;] ? process_timeout+0x0/0x10 .814762] [&lt;ffffffffa0e00771&gt;] cfs_waitq_timedwait+0x11/0x20 [libcfs] .815052] [&lt;ffffffffa053184d&gt;] ldlm_completion_ast+0x4dd/0x950 [ptlrpc] .815338] [&lt;ffffffffa052d360&gt;] ? ldlm_expired_completion_wait+0x0/0x260 [ptlrpc] .815791] [&lt;ffffffff81057d60&gt;] ? default_wake_function+0x0/0x20 .816062] [&lt;ffffffffa0530f98&gt;] ldlm_cli_enqueue_local+0x1f8/0x5d0 [ptlrpc] .816351] [&lt;ffffffffa0531370&gt;] ? ldlm_completion_ast+0x0/0x950 [ptlrpc] .816692] [&lt;ffffffffa09dc830&gt;] ? mdt_blocking_ast+0x0/0x2a0 [mdt] .816980] [&lt;ffffffffa09dedc0&gt;] mdt_object_lock0+0x360/0xb10 [mdt] .817278] [&lt;ffffffffa09dc830&gt;] ? mdt_blocking_ast+0x0/0x2a0 [mdt] .817584] [&lt;ffffffffa0531370&gt;] ? ldlm_completion_ast+0x0/0x950 [ptlrpc] .817878] [&lt;ffffffffa09df614&gt;] mdt_object_lock+0x14/0x20 [mdt] .818158] [&lt;ffffffffa09fef3a&gt;] mdt_reint_link+0x3fa/0x910 [mdt] .818459] [&lt;ffffffffa0559c5e&gt;] ? lustre_pack_reply_flags+0xae/0x1f0 [ptlrpc] .818918] [&lt;ffffffffa09faa91&gt;] mdt_reint_rec+0x41/0xe0 [mdt] .819194] [&lt;ffffffffa09f4133&gt;] mdt_reint_internal+0x4f3/0x7e0 [mdt] .819495] [&lt;ffffffffa09f4464&gt;] mdt_reint+0x44/0xe0 [mdt] .819764] [&lt;ffffffffa09e4782&gt;] mdt_handle_common+0x8e2/0x1680 [mdt] .820052] [&lt;ffffffffa0a19ed5&gt;] mds_regular_handle+0x15/0x20 [mdt] .820352] [&lt;ffffffffa0568483&gt;] ptlrpc_server_handle_request+0x463/0xe60 [ptlrpc] .820872] [&lt;ffffffffa0e0065e&gt;] ? cfs_timer_arm+0xe/0x10 [libcfs] .821172] [&lt;ffffffffa0561221&gt;] ? ptlrpc_wait_event+0xb1/0x2a0 [ptlrpc] .821474] [&lt;ffffffff81051f73&gt;] ? __wake_up+0x53/0x70 .821758] [&lt;ffffffffa056af8d&gt;] ptlrpc_main+0xb3d/0x18c0 [ptlrpc] .822055] [&lt;ffffffffa056a450&gt;] ? ptlrpc_main+0x0/0x18c0 [ptlrpc] .822333] [&lt;ffffffff8100c14a&gt;] child_rip+0xa/0x20 .822608] [&lt;ffffffffa056a450&gt;] ? ptlrpc_main+0x0/0x18c0 [ptlrpc] .822904] [&lt;ffffffffa056a450&gt;] ? ptlrpc_main+0x0/0x18c0 [ptlrpc] .823181] [&lt;ffffffff8100c140&gt;] ? child_rip+0x0/0x20 .823438] .. (this thread actually finished after 100 seconds) ... .864250] INFO: task statmany:14503 blocked for more than 120 seconds. .865404] "echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this messag .867239] statmany D 0000000000000000 2608 14503 14484 0x00000000 .868449] ffff880055affb10 0000000000000086 ffff880055affad8 ffff880055affad4 .869752] ffff880055affaf8 ffff8800bcc23080 ffff8800063167c0 0000000000000000 .870198] ffff8800aa03e8b8 ffff880055afffd8 000000000000fba8 ffff8800aa03e8b8 .870647] Call Trace: .870842] [&lt;ffffffff814fabbd&gt;] rwsem_down_failed_common+0x8d/0x1d0 .871102] [&lt;ffffffff81054aaa&gt;] ? enqueue_task_fair+0x14a/0x4e0 .871356] [&lt;ffffffff81096d6a&gt;] ? sched_clock_cpu+0x6a/0x110 .871610] [&lt;ffffffff814fad56&gt;] rwsem_down_read_failed+0x26/0x30 .871867] [&lt;ffffffff8127c104&gt;] call_rwsem_down_read_failed+0x14/0x30 .872156] [&lt;ffffffff814f9ec7&gt;] ? down_read+0x37/0x40 .872474] [&lt;ffffffffa0717cba&gt;] lov_lsm_addref+0x2a/0x130 [lov] .872796] [&lt;ffffffffa0717dea&gt;] lov_lsm_get+0x2a/0x40 [lov] .873063] [&lt;ffffffffa0b28c95&gt;] ccc_inode_lsm_get+0x15/0x20 [lustre] .873332] [&lt;ffffffffa0addb83&gt;] ll_merge_lvb+0x53/0x5e0 [lustre] .873600] [&lt;ffffffff814f919e&gt;] ? mutex_unlock+0xe/0x10 .873863] [&lt;ffffffffa0f6dd0a&gt;] ? cl_lock_mutex_put+0x7a/0x90 [obdclass] .874149] [&lt;ffffffffa0b283a8&gt;] cl_glimpse_lock+0x328/0x480 [lustre] .874445] [&lt;ffffffffa0b28a65&gt;] cl_glimpse_size0+0x1a5/0x1d0 [lustre] .874735] [&lt;ffffffffa0ae2298&gt;] ll_inode_revalidate_it+0x198/0x1c0 [lustre] .875036] [&lt;ffffffffa0ae2309&gt;] ll_getattr_it+0x49/0x170 [lustre] .875322] [&lt;ffffffffa0ae2467&gt;] ll_getattr+0x37/0x40 [lustre] .875598] [&lt;ffffffff81214fa3&gt;] ? security_inode_getattr+0x23/0x30 .875875] [&lt;ffffffff81180891&gt;] vfs_getattr+0x51/0x80 .876154] [&lt;ffffffff810385d8&gt;] ? pvclock_clocksource_read+0x58/0xd0 .876439] [&lt;ffffffff81180920&gt;] vfs_fstatat+0x60/0x80 .876696] [&lt;ffffffff81180a6b&gt;] vfs_stat+0x1b/0x20 .876947] [&lt;ffffffff81180a94&gt;] sys_newstat+0x24/0x50 .877203] [&lt;ffffffff8109b05a&gt;] ? do_gettimeofday+0x1a/0x50 .877473] [&lt;ffffffff8100c5b5&gt;] ? math_state_restore+0x45/0x60 .877751] [&lt;ffffffff814fb72e&gt;] ? do_device_not_available+0xe/0x10 .878001] [&lt;ffffffff8100befb&gt;] ? device_not_available+0x1b/0x20 .878277] [&lt;ffffffff8100b0f2&gt;] system_call_fastpath+0x16/0x1b .878552] INFO: task ll_imp_inval:14515 blocked for more than 120 seconds. .878838] "echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this messag .879297] ll_imp_inval D 0000000000000003 4640 14515 2 0x00000000 .879607] ffff8800b5f31980 0000000000000046 0000000000000000 ffffffff810385d8 .880107] ffff8800ffffffff 00075e5906b4fe40 ffff8800ffffffff ffff8800b5f31980 .880590] ffff8800ac106af8 ffff8800b5f31fd8 000000000000fba8 ffff8800ac106af8 .881064] Call Trace: .881268] [&lt;ffffffff810385d8&gt;] ? pvclock_clocksource_read+0x58/0xd0 .881548] [&lt;ffffffff814fabbd&gt;] rwsem_down_failed_common+0x8d/0x1d0 .881807] [&lt;ffffffff814fad23&gt;] rwsem_down_write_failed+0x23/0x30 .882054] [&lt;ffffffff8127c133&gt;] call_rwsem_down_write_failed+0x13/0x20 .882339] [&lt;ffffffff814f9e85&gt;] ? down_write+0x45/0x50 .882605] [&lt;ffffffffa0716e44&gt;] lov_conf_lock+0x34/0xc0 [lov] .882877] [&lt;ffffffffa0717268&gt;] lov_conf_set+0x48/0x6a0 [lov] .883163] [&lt;ffffffffa0f65aad&gt;] ? cl_env_nested_get+0x5d/0xc0 [obdclass] .883465] [&lt;ffffffffa0e189eb&gt;] ? cfs_hash_add_unique+0x1b/0x40 [libcfs] .883771] [&lt;ffffffffa0f63d08&gt;] cl_conf_set+0x58/0x100 [obdclass] .884076] [&lt;ffffffffa0adb1c8&gt;] ll_layout_conf+0x78/0x250 [lustre] .884365] [&lt;ffffffffa0b0e947&gt;] ll_md_blocking_ast+0x3e7/0x720 [lustre] .884655] [&lt;ffffffff81051f73&gt;] ? __wake_up+0x53/0x70 .884936] [&lt;ffffffffa051106c&gt;] ldlm_cancel_callback+0x6c/0x1a0 [ptlrpc] .885243] [&lt;ffffffffa052afda&gt;] ldlm_cli_cancel_local+0x8a/0x470 [ptlrpc] .885561] [&lt;ffffffffa052f8cb&gt;] ldlm_cli_cancel+0x5b/0x360 [ptlrpc] .885848] [&lt;ffffffffa0519c1d&gt;] cleanup_resource+0x12d/0x300 [ptlrpc] .886128] [&lt;ffffffffa0519e1f&gt;] ldlm_resource_clean+0x2f/0x60 [ptlrpc] .886993] [&lt;ffffffffa0e167af&gt;] cfs_hash_for_each_relax+0x17f/0x360 [libcfs] .887434] [&lt;ffffffffa0519df0&gt;] ? ldlm_resource_clean+0x0/0x60 [ptlrpc] .887717] [&lt;ffffffffa0519df0&gt;] ? ldlm_resource_clean+0x0/0x60 [ptlrpc] .887997] [&lt;ffffffffa0e1817f&gt;] cfs_hash_for_each_nolock+0x7f/0x1c0 [libcfs] .888463] [&lt;ffffffffa051685e&gt;] ldlm_namespace_cleanup+0x2e/0xc0 [ptlrpc] .888800] [&lt;ffffffffa04bbc81&gt;] mdc_import_event+0x1c1/0xa30 [mdc] .889143] [&lt;ffffffffa057993f&gt;] ptlrpc_invalidate_import+0x29f/0x8c0 [ptlrpc] .889596] [&lt;ffffffffa0e10401&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] .889870] [&lt;ffffffffa0579f60&gt;] ? ptlrpc_invalidate_import_thread+0x0/0x2e0 [ptlr .890341] [&lt;ffffffffa0579faf&gt;] ptlrpc_invalidate_import_thread+0x4f/0x2e0 [ptlrp .890807] [&lt;ffffffff8100c14a&gt;] child_rip+0xa/0x20 .891077] [&lt;ffffffffa0579f60&gt;] ? ptlrpc_invalidate_import_thread+0x0/0x2e0 [ptlr .892918] [&lt;ffffffffa0579f60&gt;] ? ptlrpc_invalidate_import_thread+0x0/0x2e0 [ptlr .893398] [&lt;ffffffff8100c140&gt;] ? child_rip+0x0/0x20 (these two threads never complete) </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 756.853670] LustreError: 25456:0:(lov_object.c:595:lov_layout_change()) ASSERTION( list_empty(&amp;hdr-&gt;coh_locks) ) failed: [ 756.854210] LustreError: 25456:0:(lov_object.c:595:lov_layout_change()) LBUG [ 756.854498] Pid: 25456, comm: ll_sa_25403 [ 756.854724] [ 756.854725] Call Trace: [ 756.855090] [&lt;ffffffffa0449915&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] [ 756.855355] [&lt;ffffffffa0449f17&gt;] lbug_with_loc+0x47/0xb0 [libcfs] [ 756.855617] [&lt;ffffffffa0a711af&gt;] lov_layout_change+0x2df/0x350 [lov] [ 756.855874] [&lt;ffffffffa0a714b7&gt;] lov_conf_set+0x297/0x6a0 [lov] [ 756.856124] [&lt;ffffffffa045a401&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] [ 756.856384] [&lt;ffffffffa04629eb&gt;] ? cfs_hash_add_unique+0x1b/0x40 [libcfs] [ 756.856807] [&lt;ffffffffa061ad08&gt;] cl_conf_set+0x58/0x100 [obdclass] [ 756.857073] [&lt;ffffffffa0dd71c8&gt;] ll_layout_conf+0x78/0x250 [lustre] [ 756.857334] [&lt;ffffffffa0de62ea&gt;] ll_layout_lock_set+0x40a/0xd30 [lustre] [ 756.857599] [&lt;ffffffffa0947320&gt;] ? lmv_set_lock_data+0x100/0x480 [lmv] [ 756.857862] [&lt;ffffffffa0de73ef&gt;] ll_layout_refresh+0x7df/0xd80 [lustre] [ 756.858128] [&lt;ffffffffa0e0a560&gt;] ? ll_md_blocking_ast+0x0/0x720 [lustre] [ 756.858411] [&lt;ffffffffa0765370&gt;] ? ldlm_completion_ast+0x0/0x950 [ptlrpc] [ 756.858685] [&lt;ffffffffa0e2d2ed&gt;] vvp_io_fini+0xfd/0x1b0 [lustre] [ 756.858939] [&lt;ffffffffa0a7b7ad&gt;] ? lov_io_fini+0x16d/0x3d0 [lov] [ 756.859202] [&lt;ffffffffa062c807&gt;] cl_io_fini+0x77/0x260 [obdclass] [ 756.859460] [&lt;ffffffffa0e24985&gt;] cl_glimpse_size0+0xc5/0x1d0 [lustre] [ 756.859725] [&lt;ffffffffa0e1cde2&gt;] ll_agl_trigger+0xd2/0x350 [lustre] [ 756.859983] [&lt;ffffffffa0e233a7&gt;] ll_statahead_thread+0x2d7/0xf40 [lustre] [ 756.860245] [&lt;ffffffff81057d60&gt;] ? default_wake_function+0x0/0x20 [ 756.860501] [&lt;ffffffffa0e230d0&gt;] ? ll_statahead_thread+0x0/0xf40 [lustre] [ 756.860776] [&lt;ffffffff8100c14a&gt;] child_rip+0xa/0x20 [ 756.861012] [&lt;ffffffffa0e230d0&gt;] ? ll_statahead_thread+0x0/0xf40 [lustre] [ 756.861279] [&lt;ffffffffa0e230d0&gt;] ? ll_statahead_thread+0x0/0xf40 [lustre] [ 756.861537] [&lt;ffffffff8100c140&gt;] ? child_rip+0x0/0x20 [ 756.861769] [ 756.885413] Kernel panic - not syncing: LBUG </pre> </div></div> <p>I've known the root cause of the problem, need time to cook a patch</p> <p>patches are at: <a href="http://review.whamcloud.com/5157" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5157</a>, <a href="http://review.whamcloud.com/5158" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5158</a>, <a href="http://review.whamcloud.com/5159" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5159</a>. Oleg is verifying them.</p> <p>Patches landed, issue no longer visible</p> <p>Saw this running racer today on 2.4.51-3-g9f5eea8. Reproduces <b>much</b> more readily if you do:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># llmount.sh # cd /mnt/lustre # touch 0 1 # while true; do lfs swap_layouts $((RANDOM % 2)) $((RANDOM % 2)) done &amp; # while true; do lfs swap_layouts $((RANDOM % 2)) $((RANDOM % 2)) done &amp; </pre> </div></div> <p>The same is true of <a href="https://jira.whamcloud.com/browse/LU-2766" title="lov_object.c:635:lov_layout_change()) ASSERTION( atomic_read(&amp;lov-&gt;lo_active_ios) == 0 ) failed" class="issue-link" data-issue-key="LU-2766"><del>LU-2766</del></a>.</p> <p>Do you think this is the same bug as was previously being hit here, or a different bug with the same symptom (this wasn't hit with racer, and was apparently fixed by landing the patch)? Reopening this bug (which is marked a blocker) that was fixed and landed for 2.4.0 complicates the tracking process.</p> <p>It might be better to close this one again and open a separate bug related to racer + layout swap. While the symptom (LASSERT()) is serious, the stress of swapping the layout of two files in a loop is very unlikely to be seen in real life, so I won't rank this as a blocker.</p> <p>Unfortunately this is still present in 2.4. Oleg's description matches up very well with what I'm seeing on 2.4 and 2.5, so reopening seems best. Based on stack traces and the shared reproducer, I assume this is really the same issue as <a href="https://jira.whamcloud.com/browse/LU-2766" title="lov_object.c:635:lov_layout_change()) ASSERTION( atomic_read(&amp;lov-&gt;lo_active_ios) == 0 ) failed" class="issue-link" data-issue-key="LU-2766"><del>LU-2766</del></a>.</p> <p>Duplicate of <a href="https://jira.whamcloud.com/browse/LU-2766" title="lov_object.c:635:lov_layout_change()) ASSERTION( atomic_read(&amp;lov-&gt;lo_active_ios) == 0 ) failed" class="issue-link" data-issue-key="LU-2766"><del>LU-2766</del></a>, the same issue with different symptom.</p> <p>TGCC site hits the same issue/LBUG running an MPI-IO app on 2.1.3 (<a href="https://jira.whamcloud.com/browse/LU-3524" title="Lustre 2.1.3: lov_io.c:212:lov_sub_get()) ASSERTION( stripe &lt; lio-&gt;lis_stripe_count ) failed" class="issue-link" data-issue-key="LU-3524"><del>LU-3524</del></a>), I mean the original/title "(lov_io.c:212:lov_sub_get()) ASSERTION( stripe &lt; lio-&gt;lis_stripe_count ) failed" LBUG for this ticket, so based on this can we definitely trust that this particular problem is layout related ?</p> Duplicate LU-1876 Related LU-2766 Development Rank 1|hzvfq7: Rank (Obsolete) 6194 Severity