https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2093 Lustre <p>Second time I see a crash like this. First time was test 118m, now test 27m, but same assertion/stack.<br/> Has to do with OOS apparently:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">[10419.559628] Lustre: DEBUG MARKER: == sanity test 27m: create file <span class="code-keyword">while</span> OST0 was full ==================== 22:00:08 (1349402408) [10437.623359] LustreError: 16640:0:(vvp_io.c:1038:vvp_io_commit_write()) Write page 37435 of inode ffff88016e63bb20 failed -28 [10437.775336] LustreError: 12752:0:(osp_precreate.c:275:osp_precreate_send()) l ustre-OST0000-osc-MDT0000: can<span class="code-quote">'t precreate: rc = -28[10439.841663] LustreError: 12937:0:(lod_qos.c:1147:lod_alloc_qos()) can'</span>t decla re <span class="code-keyword">new</span> object on #0: -28 [10439.843142] LustreError: 12937:0:(lod_qos.c:1159:lod_alloc_qos()) Didn't find any OSTs? [10439.844380] LustreError: 12937:0:(lod_qos.c:1163:lod_alloc_qos()) ASSERTION( nfound == stripe_cnt ) failed: [10439.845971] LustreError: 12937:0:(lod_qos.c:1163:lod_alloc_qos()) LBUG [10439.847156] Pid: 12937, comm: mdt00_004 [10439.848067] [10439.848067] Call Trace: [10439.848838] [&lt;ffffffffa074d915&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] [10439.850378] [&lt;ffffffffa074df27&gt;] lbug_with_loc+0x47/0xb0 [libcfs] [10439.851372] [&lt;ffffffffa0c39217&gt;] lod_alloc_qos.clone.0+0x8e7/0x1170 [lod] [10439.852475] [&lt;ffffffffa0c3b303&gt;] lod_qos_prep_create+0x693/0x18e4 [lod] [10439.853557] [&lt;ffffffffa0c36a8b&gt;] lod_declare_striped_object+0x14b/0x920 [lod] [10439.854839] [&lt;ffffffffa0c37568&gt;] lod_declare_object_create+0x308/0x4f0 [lod] [10439.855969] [&lt;ffffffffa06ffc4f&gt;] mdd_declare_object_create_internal+0xaf/0x1d0 [mdd] [10439.857238] [&lt;ffffffffa0710aca&gt;] mdd_create+0x39a/0x1550 [mdd] [10439.858190] [&lt;ffffffffa0b7bbc9&gt;] mdt_reint_open+0x1079/0x1860 [mdt] [10439.859199] [&lt;ffffffffa071686e&gt;] ? md_ucred+0x1e/0x60 [mdd] [10439.860114] [&lt;ffffffffa0b46655&gt;] ? mdt_ucred+0x15/0x20 [mdt] [10439.861049] [&lt;ffffffffa0b660a1&gt;] mdt_reint_rec+0x41/0xe0 [mdt] [10439.862006] [&lt;ffffffffa0b5f483&gt;] mdt_reint_internal+0x4e3/0x7e0 [mdt] [10439.863046] [&lt;ffffffffa0b5fa4d&gt;] mdt_intent_reint+0x1ed/0x500 [mdt] [10439.864064] [&lt;ffffffffa0b5b3fe&gt;] mdt_intent_policy+0x38e/0x770 [mdt] [10439.865125] [&lt;ffffffffa022ddda&gt;] ldlm_lock_enqueue+0x2ea/0x890 [ptlrpc] [10439.866208] [&lt;ffffffffa0254fc7&gt;] ldlm_handle_enqueue0+0x4e7/0x1010 [ptlrpc] [10439.867328] [&lt;ffffffffa0b5b936&gt;] mdt_enqueue+0x46/0x130 [mdt] [10439.868292] [&lt;ffffffffa0b4f1f2&gt;] mdt_handle_common+0x932/0x1740 [mdt] [10439.869535] [&lt;ffffffffa0b500d5&gt;] mdt_regular_handle+0x15/0x20 [mdt] [10439.870577] [&lt;ffffffffa0283743&gt;] ptlrpc_server_handle_request+0x463/0xe70 [ptlrpc] [10439.871790] [&lt;ffffffffa074e66e&gt;] ? cfs_timer_arm+0xe/0x10 [libcfs] [10439.873067] [&lt;ffffffffa027c431&gt;] ? ptlrpc_wait_event+0xb1/0x2a0 [ptlrpc] [10439.874154] [&lt;ffffffff81051f73&gt;] ? __wake_up+0x53/0x70 [10439.875000] [&lt;ffffffffa02862ce&gt;] ptlrpc_main+0xb8e/0x1960 [ptlrpc] [10439.876019] [&lt;ffffffffa0285740&gt;] ? ptlrpc_main+0x0/0x1960 [ptlrpc] [10439.877234] [&lt;ffffffff8100c14a&gt;] child_rip+0xa/0x20 [10439.878049] [&lt;ffffffffa0285740&gt;] ? ptlrpc_main+0x0/0x1960 [ptlrpc] [10439.879049] [&lt;ffffffffa0285740&gt;] ? ptlrpc_main+0x0/0x1960 [ptlrpc] [10439.880047] [&lt;ffffffff8100c140&gt;] ? child_rip+0x0/0x20 [10439.880889] [10439.881508] Kernel panic - not syncing: LBUG </pre> </div></div> LU-2093

sanity test 27m 118m ASSERTION( nfound == stripe_cnt ) failed Out of space

Bug Blocker Resolved Fixed Alex Zhuravlev Oleg Drokin Thu, 4 Oct 2012 22:04:47 +0000 Fri, 19 Apr 2013 16:20:13 +0000 Fri, 19 Apr 2013 16:20:13 +0000 Lustre 2.4.0 Lustre 2.4.0 0 3 <p>It seems when running with USE_OFD=yes this issue hits much-much more frequently for me.</p> <p>Anyway, here's the debug log dumped during ofd run.</p> <p>Apparently ORI-213 is the same or very similar issue</p> <p>please try <a href="http://review.whamcloud.com/4210" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4210</a><br/> also, could you run sanity with full debug enabled?</p> <p>Just hit it again running racer on a very fresh master (with this particular patch included).</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">[17138.598512] LustreError: 9189:0:(vvp_io.c:1038:vvp_io_commit_write()) Write page 975 of inode ffff88007df36b20 failed -28 [17138.599687] LustreError: 9189:0:(vvp_io.c:1038:vvp_io_commit_write()) Skipped 8192 previous similar messages [17162.396113] LustreError: 16540:0:(file.c:2333:ll_inode_revalidate_fini()) failure -116 inode 144115205322946107 [17162.396697] LustreError: 16540:0:(file.c:2333:ll_inode_revalidate_fini()) Skipped 6 previous similar messages [17170.436387] LustreError: 28970:0:(lod_qos.c:1147:lod_alloc_qos()) can't declare <span class="code-keyword">new</span> object on #0: -28 [17170.437162] LustreError: 28970:0:(lod_qos.c:1147:lod_alloc_qos()) Skipped 4 previous similar messages [17180.616649] LustreError: 25274:0:(ofd_obd.c:1222:ofd_create()) unable to precreate: -28 [17180.617596] LustreError: 25274:0:(ofd_obd.c:1222:ofd_create()) Skipped 339 previous similar messages [17180.618456] LustreError: 24438:0:(osp_precreate.c:275:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -28 [17180.619540] LustreError: 24438:0:(osp_precreate.c:275:osp_precreate_send()) Skipped 339 previous similar messages [17205.641098] LustreError: 24130:0:(lod_qos.c:1159:lod_alloc_qos()) Didn't find any OSTs? [17205.642043] LustreError: 24130:0:(lod_qos.c:1163:lod_alloc_qos()) ASSERTION( nfound == stripe_cnt ) failed: [17205.643006] LustreError: 24130:0:(lod_qos.c:1163:lod_alloc_qos()) LBUG [17205.643845] Pid: 24130, comm: mdt01_002 [17205.644498] [17205.644500] Call Trace: [17205.645703] [&lt;ffffffffa0f14915&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] [17205.646309] [&lt;ffffffffa0f14f27&gt;] lbug_with_loc+0x47/0xb0 [libcfs] [17205.646872] [&lt;ffffffffa097f217&gt;] lod_alloc_qos.clone.0+0x8e7/0x1170 [lod] [17205.647511] [&lt;ffffffffa0981303&gt;] lod_qos_prep_create+0x693/0x18e4 [lod] [17205.648144] [&lt;ffffffffa0832d92&gt;] ? osd_declare_inode_qid+0x1a2/0x270 [osd_ldiskfs] [17205.649115] [&lt;ffffffffa097ca8b&gt;] lod_declare_striped_object+0x14b/0x920 [lod] [17205.649994] [&lt;ffffffffa097d568&gt;] lod_declare_object_create+0x308/0x4f0 [lod] [17205.650602] [&lt;ffffffffa072bc4f&gt;] mdd_declare_object_create_internal+0xaf/0x1d0 [mdd] [17205.651497] [&lt;ffffffffa073caca&gt;] mdd_create+0x39a/0x1550 [mdd] [17205.652060] [&lt;ffffffffa08c1bc9&gt;] mdt_reint_open+0x1079/0x1860 [mdt] [17205.652646] [&lt;ffffffffa074286e&gt;] ? md_ucred+0x1e/0x60 [mdd] [17205.653248] [&lt;ffffffffa088c655&gt;] ? mdt_ucred+0x15/0x20 [mdt] [17205.653807] [&lt;ffffffffa08ac0a1&gt;] mdt_reint_rec+0x41/0xe0 [mdt] [17205.654443] [&lt;ffffffffa08a5483&gt;] mdt_reint_internal+0x4e3/0x7e0 [mdt] [17205.655038] [&lt;ffffffffa08a5a4d&gt;] mdt_intent_reint+0x1ed/0x500 [mdt] [17205.655630] [&lt;ffffffffa08a13fe&gt;] mdt_intent_policy+0x38e/0x770 [mdt] [17205.656247] [&lt;ffffffffa11d0dda&gt;] ldlm_lock_enqueue+0x2ea/0x890 [ptlrpc] [17205.657122] [&lt;ffffffffa11f80d7&gt;] ldlm_handle_enqueue0+0x4e7/0x1010 [ptlrpc] [17205.657748] [&lt;ffffffffa08a1936&gt;] mdt_enqueue+0x46/0x130 [mdt] [17205.658413] [&lt;ffffffffa08951f2&gt;] mdt_handle_common+0x932/0x1740 [mdt] [17205.658681] [&lt;ffffffffa08960d5&gt;] mdt_regular_handle+0x15/0x20 [mdt] [17205.659050] [&lt;ffffffffa1226853&gt;] ptlrpc_server_handle_request+0x463/0xe70 [ptlrpc] [17205.659479] [&lt;ffffffffa0f1566e&gt;] ? cfs_timer_arm+0xe/0x10 [libcfs] [17205.659755] [&lt;ffffffffa121f541&gt;] ? ptlrpc_wait_event+0xb1/0x2a0 [ptlrpc] [17205.660018] [&lt;ffffffff81051f73&gt;] ? __wake_up+0x53/0x70 [17205.660288] [&lt;ffffffffa12293ea&gt;] ptlrpc_main+0xb9a/0x1960 [ptlrpc] [17205.660557] [&lt;ffffffffa1228850&gt;] ? ptlrpc_main+0x0/0x1960 [ptlrpc] [17205.660815] [&lt;ffffffff8100c14a&gt;] child_rip+0xa/0x20 [17205.661094] [&lt;ffffffffa1228850&gt;] ? ptlrpc_main+0x0/0x1960 [ptlrpc] [17205.662229] [&lt;ffffffffa1228850&gt;] ? ptlrpc_main+0x0/0x1960 [ptlrpc] [17205.662490] [&lt;ffffffff8100c140&gt;] ? child_rip+0x0/0x20 [17205.662727] [17205.666053] Kernel panic - not syncing: LBUG [17205.666056] Pid: 24130, comm: mdt01_002 Not tainted 2.6.32-debug #6 [17205.666058] Call Trace: [17205.666066] [&lt;ffffffff814f75e4&gt;] ? panic+0xa0/0x168 [17205.666094] [&lt;ffffffffa0f14f7b&gt;] ? lbug_with_loc+0x9b/0xb0 [libcfs] [17205.666115] [&lt;ffffffffa097f217&gt;] ? lod_alloc_qos.clone.0+0x8e7/0x1170 [lod] [17205.666132] [&lt;ffffffffa0981303&gt;] ? lod_qos_prep_create+0x693/0x18e4 [lod] [17205.666160] [&lt;ffffffffa0832d92&gt;] ? osd_declare_inode_qid+0x1a2/0x270 [osd_ldiskfs] [17205.666177] [&lt;ffffffffa097ca8b&gt;] ? lod_declare_striped_object+0x14b/0x920 [lod] [17205.666203] [&lt;ffffffffa097d568&gt;] ? lod_declare_object_create+0x308/0x4f0 [lod] [17205.666222] [&lt;ffffffffa072bc4f&gt;] ? mdd_declare_object_create_internal+0xaf/0x1d0 [mdd] [17205.666240] [&lt;ffffffffa073caca&gt;] ? mdd_create+0x39a/0x1550 [mdd] [17205.666272] [&lt;ffffffffa08c1bc9&gt;] ? mdt_reint_open+0x1079/0x1860 [mdt] [17205.666288] [&lt;ffffffffa074286e&gt;] ? md_ucred+0x1e/0x60 [mdd] [17205.666309] [&lt;ffffffffa088c655&gt;] ? mdt_ucred+0x15/0x20 [mdt] [17205.666332] [&lt;ffffffffa08ac0a1&gt;] ? mdt_reint_rec+0x41/0xe0 [mdt] [17205.666355] [&lt;ffffffffa08a5483&gt;] ? mdt_reint_internal+0x4e3/0x7e0 [mdt] [17205.666378] [&lt;ffffffffa08a5a4d&gt;] ? mdt_intent_reint+0x1ed/0x500 [mdt] [17205.666400] [&lt;ffffffffa08a13fe&gt;] ? mdt_intent_policy+0x38e/0x770 [mdt] [17205.666462] [&lt;ffffffffa11d0dda&gt;] ? ldlm_lock_enqueue+0x2ea/0x890 [ptlrpc] [17205.666521] [&lt;ffffffffa11f80d7&gt;] ? ldlm_handle_enqueue0+0x4e7/0x1010 [ptlrpc] [17205.666543] [&lt;ffffffffa08a1936&gt;] ? mdt_enqueue+0x46/0x130 [mdt] [17205.666564] [&lt;ffffffffa08951f2&gt;] ? mdt_handle_common+0x932/0x1740 [mdt] [17205.666585] [&lt;ffffffffa08960d5&gt;] ? mdt_regular_handle+0x15/0x20 [mdt] [17205.666649] [&lt;ffffffffa1226853&gt;] ? ptlrpc_server_handle_request+0x463/0xe70 [ptlrpc] [17205.666671] [&lt;ffffffffa0f1566e&gt;] ? cfs_timer_arm+0xe/0x10 [libcfs] [17205.666732] [&lt;ffffffffa121f541&gt;] ? ptlrpc_wait_event+0xb1/0x2a0 [ptlrpc] [17205.666739] [&lt;ffffffff81051f73&gt;] ? __wake_up+0x53/0x70 [17205.666797] [&lt;ffffffffa12293ea&gt;] ? ptlrpc_main+0xb9a/0x1960 [ptlrpc] [17205.666856] [&lt;ffffffffa1228850&gt;] ? ptlrpc_main+0x0/0x1960 [ptlrpc] [17205.666862] [&lt;ffffffff8100c14a&gt;] ? child_rip+0xa/0x20 [17205.666921] [&lt;ffffffffa1228850&gt;] ? ptlrpc_main+0x0/0x1960 [ptlrpc] [17205.666981] [&lt;ffffffffa1228850&gt;] ? ptlrpc_main+0x0/0x1960 [ptlrpc] [17205.666987] [&lt;ffffffff8100c140&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p>Crashdump is at /exports/crashdumps/192.168.10.211-2012-10-08-21\:08\:23 (modules included)</p> <p>please try with <a href="http://review.whamcloud.com/#change,4241" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,4241</a></p> <p>landed</p> Related Development Rank 1|hzv56f: Rank (Obsolete) 4372