https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2129 Lustre <p>Lustre: DEBUG MARKER: ----<del>============= acceptance-small: parallel-scale ============</del>---- Tue Oct 9 09:38:01 PDT 2012<br/> Lustre: DEBUG MARKER: excepting tests: parallel_grouplock<br/> Lustre: DEBUG MARKER: Using TIMEOUT=20<br/> LustreError: 13309:0:(osd_io.c:1062:osd_ldiskfs_write_record()) sdd1: error reading offset 12288 (block 3): rc = -28<br/> LustreError: 13309:0:(llog_osd.c:175:llog_osd_write_blob()) lustre-MDT0000-osd: error writing log buffer: rc = -28<br/> LustreError: 13309:0:(mgs_llog.c:738:record_lcfg()) failed -28<br/> LustreError: 13309:0:(mgs_llog.c:1074:mgs_write_log_direct_all()) MGS: writing log lustre-params: rc = -28<br/> LustreError: 13309:0:(osd_io.c:1062:osd_ldiskfs_write_record()) sdd1: error reading offset 12288 (block 3): rc = -28<br/> LustreError: 13309:0:(llog_osd.c:175:llog_osd_write_blob()) lustre-MDT0000-osd: error writing log buffer: rc = -28<br/> LustreError: 13309:0:(mgs_llog.c:738:record_lcfg()) failed -28<br/> LustreError: 13309:0:(mgs_llog.c:1074:mgs_write_log_direct_all()) MGS: writing log lustre-OST0000: rc = -28<br/> LustreError: 13309:0:(mgs_llog.c:2956:mgs_write_log_param()) err -28 on param 'sys.jobid_var=procname_uid'<br/> LustreError: 13309:0:(mgs_handler.c:926:mgs_iocontrol()) MGS: setparam err: rc = -28<br/> LustreError: 11580:0:(llog_osd.c:579:llog_osd_next_block()) ASSERTION( last_rec-&gt;lrh_index == tail-&gt;lrt_index ) failed: <br/> LustreError: 11580:0:(llog_osd.c:579:llog_osd_next_block()) LBUG<br/> Pid: 11580, comm: ll_mgs_0003</p> <p>Call Trace:<br/> <span class="error">&#91;&lt;ffffffffa0991905&gt;&#93;</span> libcfs_debug_dumpstack+0x55/0x80 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0991f17&gt;&#93;</span> lbug_with_loc+0x47/0xb0 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0cd6435&gt;&#93;</span> llog_osd_next_block+0xa55/0xa60 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa151d436&gt;&#93;</span> ? lustre_pack_reply_flags+0xb6/0x210 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa153a80c&gt;&#93;</span> llog_origin_handle_next_block+0x50c/0x780 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa1747f93&gt;&#93;</span> mgs_handle+0xb13/0x11e0 <span class="error">&#91;mgs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa09a2621&gt;&#93;</span> ? libcfs_debug_msg+0x41/0x50 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa152d8dc&gt;&#93;</span> ptlrpc_server_handle_request+0x41c/0xe00 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff8104e309&gt;&#93;</span> ? __wake_up_common+0x59/0x90<br/> <span class="error">&#91;&lt;ffffffff810533f3&gt;&#93;</span> ? __wake_up+0x53/0x70<br/> <span class="error">&#91;&lt;ffffffffa152eecc&gt;&#93;</span> ptlrpc_main+0xc0c/0x19f0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa152e2c0&gt;&#93;</span> ? ptlrpc_main+0x0/0x19f0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff8100c14a&gt;&#93;</span> child_rip+0xa/0x20<br/> <span class="error">&#91;&lt;ffffffffa152e2c0&gt;&#93;</span> ? ptlrpc_main+0x0/0x19f0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa152e2c0&gt;&#93;</span> ? ptlrpc_main+0x0/0x19f0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff8100c140&gt;&#93;</span> ? child_rip+0x0/0x20</p> <p>Kernel panic - not syncing: LBUG<br/> Pid: 11580, comm: ll_mgs_0003 Not tainted 2.6.32-279.5.1.el6_lustre.x86_64 #1<br/> Call Trace:<br/> <span class="error">&#91;&lt;ffffffff814fd58a&gt;&#93;</span> ? panic+0xa0/0x168<br/> <span class="error">&#91;&lt;ffffffffa0991f6b&gt;&#93;</span> ? lbug_with_loc+0x9b/0xb0 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0cd6435&gt;&#93;</span> ? llog_osd_next_block+0xa55/0xa60 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa151d436&gt;&#93;</span> ? lustre_pack_reply_flags+0xb6/0x210 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa153a80c&gt;&#93;</span> ? llog_origin_handle_next_block+0x50c/0x780 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa1747f93&gt;&#93;</span> ? mgs_handle+0xb13/0x11e0 <span class="error">&#91;mgs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa09a2621&gt;&#93;</span> ? libcfs_debug_msg+0x41/0x50 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa152d8dc&gt;&#93;</span> ? ptlrpc_server_handle_request+0x41c/0xe00 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff8104e309&gt;&#93;</span> ? __wake_up_common+0x59/0x90<br/> <span class="error">&#91;&lt;ffffffff810533f3&gt;&#93;</span> ? __wake_up+0x53/0x70<br/> <span class="error">&#91;&lt;ffffffffa152eecc&gt;&#93;</span> ? ptlrpc_main+0xc0c/0x19f0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa152e2c0&gt;&#93;</span> ? ptlrpc_main+0x0/0x19f0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff8100c14a&gt;&#93;</span> ? child_rip+0xa/0x20<br/> <span class="error">&#91;&lt;ffffffffa152e2c0&gt;&#93;</span> ? ptlrpc_main+0x0/0x19f0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa152e2c0&gt;&#93;</span> ? ptlrpc_main+0x0/0x19f0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff8100c140&gt;&#93;</span> ? child_rip+0x0/0x20</p> &nbsp;<a href="https://build.whamcloud.com/job/lustre-master/951/">https://build.whamcloud.com/job/lustre-master/951/</a> <br/> On opensfs with 1 mds, 4 oss, 24 clients LU-2129

ASSERTION( last_rec->lrh_index == tail->lrt_index )

Bug Blocker Resolved Fixed Mikhail Pershin Minh Diep NFBlocker Tue, 9 Oct 2012 12:57:35 +0000 Tue, 20 Nov 2012 14:23:19 +0000 Tue, 20 Nov 2012 14:23:19 +0000 Lustre 2.4.0 Lustre 2.4.0 0 5 <p>This happens in my testing at random tests<br/> last occurence today in the morning, crashdump available:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">[10933.785890] LustreError: 11126:0:(lmv_obd.c:1190:lmv_statfs()) can't stat MDS #0 (lustre-MDT0000-mdc-ffff880096259bf0), error -5 [10933.786846] LustreError: 11126:0:(llite_lib.c:1558:ll_statfs_internal()) md_statfs fails: rc = -5 [10935.038782] Lustre: DEBUG MARKER: == replay-single test 21: |X| open(O_CREAT), unlink touch <span class="code-keyword">new</span>, replay, close (test mds_cleanup_orphans) == 06:04:11 (1349777051) [10935.209519] Turning device loop0 (0x700000) read-only [10935.233110] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [10935.239897] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [10935.555060] Removing read-only on unknown block (0x700000) [10945.911054] LDISKFS-fs (loop0): recovery complete [10945.912650] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. quota=on. Opts: [10945.944018] LustreError: 11387:0:(llog_osd.c:579:llog_osd_next_block()) ASSERTION( last_rec-&gt;lrh_index == tail-&gt;lrt_index ) failed: [10945.944868] LustreError: 11387:0:(llog_osd.c:579:llog_osd_next_block()) LBUG [10945.945387] Pid: 11387, comm: ll_mgs_0001 [10945.945389] [10945.945389] Call Trace: [10945.945410] [&lt;ffffffffa0c24915&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] [10945.945424] [&lt;ffffffffa0c24f27&gt;] lbug_with_loc+0x47/0xb0 [libcfs] [10945.945469] [&lt;ffffffffa1124af5&gt;] llog_osd_next_block+0xa55/0xa60 [obdclass] [10945.945521] [&lt;ffffffffa1288dc6&gt;] ? lustre_pack_reply_flags+0xb6/0x210 [ptlrpc] [10945.945542] [&lt;ffffffffa12a542c&gt;] llog_origin_handle_next_block+0x55c/0x780 [ptlrpc] [10945.945549] [&lt;ffffffffa066ef73&gt;] mgs_handle+0xb13/0x11e0 [mgs] [10945.945557] [&lt;ffffffffa0c356d1&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] [10945.945576] [&lt;ffffffffa1297853&gt;] ptlrpc_server_handle_request+0x463/0xe70 [ptlrpc] [10945.945587] [&lt;ffffffffa0c2566e&gt;] ? cfs_timer_arm+0xe/0x10 [libcfs] [10945.945614] [&lt;ffffffffa1290541&gt;] ? ptlrpc_wait_event+0xb1/0x2a0 [ptlrpc] [10945.945619] [&lt;ffffffff81051f73&gt;] ? __wake_up+0x53/0x70 [10945.945646] [&lt;ffffffffa129a3ea&gt;] ptlrpc_main+0xb9a/0x1960 [ptlrpc] [10945.945673] [&lt;ffffffffa1299850&gt;] ? ptlrpc_main+0x0/0x1960 [ptlrpc] [10945.945676] [&lt;ffffffff8100c14a&gt;] child_rip+0xa/0x20 [10945.945702] [&lt;ffffffffa1299850&gt;] ? ptlrpc_main+0x0/0x1960 [ptlrpc] [10945.945744] [&lt;ffffffffa1299850&gt;] ? ptlrpc_main+0x0/0x1960 [ptlrpc] [10945.945748] [&lt;ffffffff8100c140&gt;] ? child_rip+0x0/0x20 [10945.945749] [10945.948136] Kernel panic - not syncing: LBUG [10945.948138] Pid: 11387, comm: ll_mgs_0001 Not tainted 2.6.32-debug #6 [10945.948139] Call Trace: [10945.948146] [&lt;ffffffff814f75e4&gt;] ? panic+0xa0/0x168 [10945.948155] [&lt;ffffffffa0c24f7b&gt;] ? lbug_with_loc+0x9b/0xb0 [libcfs] [10945.948198] [&lt;ffffffffa1124af5&gt;] ? llog_osd_next_block+0xa55/0xa60 [obdclass] [10945.948242] [&lt;ffffffffa1288dc6&gt;] ? lustre_pack_reply_flags+0xb6/0x210 [ptlrpc] [10945.948285] [&lt;ffffffffa12a542c&gt;] ? llog_origin_handle_next_block+0x55c/0x780 [ptlrpc] [10945.948296] [&lt;ffffffffa066ef73&gt;] ? mgs_handle+0xb13/0x11e0 [mgs] [10945.948312] [&lt;ffffffffa0c356d1&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] [10945.948354] [&lt;ffffffffa1297853&gt;] ? ptlrpc_server_handle_request+0x463/0xe70 [ptlrpc] [10945.948368] [&lt;ffffffffa0c2566e&gt;] ? cfs_timer_arm+0xe/0x10 [libcfs] [10945.948409] [&lt;ffffffffa1290541&gt;] ? ptlrpc_wait_event+0xb1/0x2a0 [ptlrpc] [10945.948413] [&lt;ffffffff81051f73&gt;] ? __wake_up+0x53/0x70 [10945.948452] [&lt;ffffffffa129a3ea&gt;] ? ptlrpc_main+0xb9a/0x1960 [ptlrpc] [10945.948492] [&lt;ffffffffa1299850&gt;] ? ptlrpc_main+0x0/0x1960 [ptlrpc] [10945.948496] [&lt;ffffffff8100c14a&gt;] ? child_rip+0xa/0x20 [10945.948540] [&lt;ffffffffa1299850&gt;] ? ptlrpc_main+0x0/0x1960 [ptlrpc] [10945.948620] [&lt;ffffffffa1299850&gt;] ? ptlrpc_main+0x0/0x1960 [ptlrpc] [10945.948624] [&lt;ffffffff8100c140&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p>I've added debug to get more information about this issue:<br/> <a href="http://review.whamcloud.com/#change,4240" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,4240</a></p> <p>Oleg, please use it if you can reproduce this bug</p> <p>Just reproduced</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[49201.778332] Lustre: DEBUG MARKER: == replay-single test 40: cause recovery in ptlrpc, ensure IO continues == 12:28:41 (1350318521) [49201.795828] Lustre: DEBUG MARKER: multiop /mnt/lustre/f.replay-single.40 OS_c [49214.016506] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. quota=on. Opts: [49214.071707] LustreError: 2626:0:(llog_osd.c:588:llog_osd_next_block()) ASSERTION( last_rec-&gt;lrh_index == tail-&gt;lrt_index ) failed: Rec idx 0, tail idx -30720 in llog #0x2#0x1#00000000 offset 92312, len 2200 [49214.072538] LustreError: 2626:0:(llog_osd.c:588:llog_osd_next_block()) LBUG [49214.072817] Pid: 2626, comm: ll_mgs_0001 [49214.073065] [49214.073065] Call Trace: [49214.073473] [&lt;ffffffffa0b34915&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] [49214.073788] [&lt;ffffffffa0b34f27&gt;] lbug_with_loc+0x47/0xb0 [libcfs] [49214.074118] [&lt;ffffffffa0ecf5f9&gt;] llog_osd_next_block+0xa89/0xa90 [obdclass] [49214.074455] [&lt;ffffffffa1092a96&gt;] ? lustre_pack_reply_flags+0xb6/0x210 [ptlrpc] [49214.074931] [&lt;ffffffffa10af05c&gt;] llog_origin_handle_next_block+0x55c/0x780 [ptlrpc] [49214.075406] [&lt;ffffffffa06bef73&gt;] mgs_handle+0xb13/0x11e0 [mgs] [49214.075679] [&lt;ffffffffa0b456d1&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] [49214.075980] [&lt;ffffffffa10a1483&gt;] ptlrpc_server_handle_request+0x463/0xe70 [ptlrpc] [49214.076443] [&lt;ffffffffa0b3566e&gt;] ? cfs_timer_arm+0xe/0x10 [libcfs] [49214.076736] [&lt;ffffffffa109a171&gt;] ? ptlrpc_wait_event+0xb1/0x2a0 [ptlrpc] [49214.077051] [&lt;ffffffff81051f73&gt;] ? __wake_up+0x53/0x70 [49214.077330] [&lt;ffffffffa10a401a&gt;] ptlrpc_main+0xb9a/0x1960 [ptlrpc] [49214.077710] [&lt;ffffffffa10a3480&gt;] ? ptlrpc_main+0x0/0x1960 [ptlrpc] [49214.077993] [&lt;ffffffff8100c14a&gt;] child_rip+0xa/0x20 [49214.078267] [&lt;ffffffffa10a3480&gt;] ? ptlrpc_main+0x0/0x1960 [ptlrpc] [49214.078583] [&lt;ffffffffa10a3480&gt;] ? ptlrpc_main+0x0/0x1960 [ptlrpc] [49214.078860] [&lt;ffffffff8100c140&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p>As I suspected the llog contains partially written records. Thus must be result of llog_osd_write_blob() called with buffer to write. In that case there are 3 separated writes actually - record header, buffer and record tail. So we can have first write successful but others not which causes assertion upon read. </p> <p>Possible solution would be the locking with write lock around those 3 writes and read lock in llog_osd_next_block(). Given the fact "3 writes" case is not common (used only for config llogs) that shouldn't cause lot of contentions.</p> <p>Meanwhile I wonder can we avoid those '3 writes' at all, demanding llog_write be not called with external buffer but only with fully prepared llog record. That means mgs should prepare llog record with copying lustre_cfg buffer. Such overhead is not critical for MGS but we are avoiding that buggy case with 3 separate writes in llog_osd </p> <p>Fix with locking, write lock envelopes problematic multiple writes while dt_read() is protected with read lock.<br/> <a href="http://review.whamcloud.com/4303" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4303</a></p> <p>Another and probably better way could be removal of those multiple writes and write llog record only by single block but would require more work, though will cleanup llog nicely</p> <p>patch is ready for landing, need confirmation from Oleg that bug it not seen anymore</p> <p>Patch landed, closing bug.</p> Development Rank 1|hzv9uv: Rank (Obsolete) 5131 Severity