https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4528 Lustre <p>When running sanity on the latest master build, hit following LBUG on client:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Lustre: DEBUG MARKER: == sanity test 60a: llog sanity tests run from kernel module ============ 15:06:20 (1390431980) Lustre: DEBUG MARKER: test_60 run 3219 - from kernel mode Lustre: 9324:0:(llog_test.c:1018:llog_test_setup()) Setup llog-test device over MGS device Lustre: 9324:0:(llog_test.c:110:llog_test_1()) 1a: create a log with name: aee6dfbf Lustre: 9324:0:(llog_test.c:127:llog_test_1()) 1b: close newly-created log Lustre: 9324:0:(llog_test.c:151:llog_test_2()) 2a: re-open a log with name: aee6dfbf Lustre: 9324:0:(llog_test.c:232:llog_test_3()) 3a: write one create_rec Lustre: 9324:0:(llog_test.c:244:llog_test_3()) 3b: write 10 cfg log records with 8 bytes bufs Lustre: 9324:0:(llog_test.c:265:llog_test_3()) 3c: write 1000 more log records Lustre: 9324:0:(llog_test.c:280:llog_test_3()) 3d: write log more than BITMAP_SIZE, return -ENOSPC Lustre: 9324:0:(llog_test.c:311:llog_test_3()) 3d: wrote 64364 more records before end of llog is reached Lustre: 9324:0:(llog_test.c:340:llog_test_4()) 4a: create a catalog log with name: aee6dfc0 Lustre: 9324:0:(llog_test.c:355:llog_test_4()) 4b: write 1 record into the catalog Lustre: 9324:0:(llog_test.c:370:llog_test_4()) 4c: cancel 1 log record Lustre: 9324:0:(llog_test.c:382:llog_test_4()) 4d: write 65536 more log records Lustre: 9324:0:(llog_test.c:398:llog_test_4()) 4e: add 5 large records, one record per block Lustre: 9324:0:(llog_test.c:418:llog_test_4()) 4f: put newly-created catalog Lustre: 9324:0:(llog_test.c:516:llog_test_5()) 5a: re-open catalog by id Lustre: 9324:0:(llog_test.c:529:llog_test_5()) 5b: print the catalog entries.. we expect 2 Lustre: 9325:0:(llog_test.c:447:cat_print_cb()) seeing record at index 1 - [0x1:0xc:0x0] in log [0xa:0xb:0x0] Lustre: 9325:0:(llog_test.c:447:cat_print_cb()) seeing record at index 2 - [0x1:0xd:0x0] in log [0xa:0xb:0x0] Lustre: 9324:0:(llog_test.c:541:llog_test_5()) 5c: Cancel 65536 records, see one log zapped Lustre: 9324:0:(llog_test.c:549:llog_test_5()) 5c: print the catalog entries.. we expect 1 Lustre: 9324:0:(llog_test.c:561:llog_test_5()) 5d: add 1 record to the log with many canceled empty pages Lustre: 9324:0:(llog_test.c:569:llog_test_5()) 5e: print plain log entries.. expect 6 Lustre: 9324:0:(llog_test.c:581:llog_test_5()) 5f: print plain log entries reversely.. expect 6 Lustre: 9324:0:(llog_test.c:595:llog_test_5()) 5g: close re-opened catalog Lustre: 9324:0:(llog_test.c:625:llog_test_6()) 6a: re-open log aee6dfbf using client API Lustre: MGS: non-config logname received: aee6dfbf Lustre: 9324:0:(llog_test.c:657:llog_test_6()) 6b: process log aee6dfbf using client API Lustre: 9324:0:(llog_test.c:661:llog_test_6()) 6b: processed 64364 records Lustre: 9324:0:(llog_test.c:668:llog_test_6()) 6c: process log aee6dfbf reversely using client API Lustre: 9324:0:(llog_test.c:672:llog_test_6()) 6c: processed 64364 records Lustre: 9324:0:(llog_test.c:820:llog_test_7()) 7a: test llog_logid_rec LustreError: 9324:0:(osd_io.c:1396:osd_ldiskfs_write_record()) loop0: error reading offset 1794048 (block 438): rc = -28 LustreError: 9324:0:(llog_osd.c:156:llog_osd_write_blob()) lustre-MDT0000-osd: error writing log record: rc = -28 LustreError: 9324:0:(osd_internal.h:970:osd_trans_exec_op()) ASSERTION( oti-&gt;oti_declare_ops_rb[rb] &gt; 0 ) failed: rb = 0 LustreError: 9324:0:(osd_internal.h:970:osd_trans_exec_op()) LBUG Pid: 9324, comm: lctl Message from Call Trace: [&lt;ffffffffa0369895&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] [&lt;ffffffffa0369e97&gt;] lbug_with_loc+0x47/0xb0 [libcfs] [&lt;ffffffffa0ccdb3b&gt;] osd_trans_exec_op+0x2ab/0x2e0 [osd_ldiskfs] [&lt;ffffffffa0ce0583&gt;] osd_attr_set+0xe3/0x540 [osd_ldiskfs] [&lt;ffffffffa0ced713&gt;] ? osd_punch+0x1d3/0x620 [osd_ldiskfs] [&lt;ffffffffa04b5081&gt;] llog_osd_write_blob+0x211/0x850 [obdclass] [&lt;ffffffffa04b906d&gt;] llog_osd_write_rec+0xacd/0x1390 [obdclass] syslogd@client- [&lt;ffffffffa04a74f8&gt;] llog_write_rec+0xc8/0x290 [obdclass] [&lt;ffffffffa04a8c6d&gt;] llog_write+0x2ad/0x420 [obdclass] [&lt;ffffffffa04594fd&gt;] llog_test_7_sub+0x16d/0xaf0 [llog_test] 3 at Jan 22 15:0 [&lt;ffffffffa0459fbf&gt;] llog_test_7+0x13f/0xa10 [llog_test] [&lt;ffffffffa045bd4a&gt;] llog_run_tests+0x14ba/0x1e50 [llog_test] [&lt;ffffffffa04b179e&gt;] ? llog_setup+0x3be/0x910 [obdclass] [&lt;ffffffffa045ce59&gt;] llog_test_setup+0x779/0x8c0 [llog_test] [&lt;ffffffffa04e2ebb&gt;] obd_setup+0x19b/0x290 [obdclass] [&lt;ffffffff81168043&gt;] ? kmem_cache_alloc_trace+0x1a3/0x1b0 6:22 ... kern [&lt;ffffffffa04c601b&gt;] ? class_new_export+0x72b/0x960 [obdclass] [&lt;ffffffffa04e31b8&gt;] class_setup+0x208/0x870 [obdclass] [&lt;ffffffffa04eb72c&gt;] class_process_config+0xc6c/0x1ad0 [obdclass] [&lt;ffffffff811685ac&gt;] ? __kmalloc+0x20c/0x220 [&lt;ffffffffa04a4d6d&gt;] ? obd_ioctl_getdata+0x12d/0x1120 [obdclass] [&lt;ffffffffa04bd3fb&gt;] class_handle_ioctl+0x3bb/0x1ef0 [obdclass] [&lt;ffffffffa04a42ab&gt;] obd_class_ioctl+0x4b/0x190 [obdclass] [&lt;ffffffff81195382&gt;] vfs_ioctl+0x22/0xa0 [&lt;ffffffff81195524&gt;] do_vfs_ioctl+0x84/0x580 [&lt;ffffffff81182bf1&gt;] ? __fput+0x1a1/0x210 [&lt;ffffffff81195aa1&gt;] sys_ioctl+0x81/0xa0 [&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b </pre> </div></div> LU-4528

osd_trans_exec_op()) ASSERTION( oti->oti_declare_ops_rb[rb] > 0 ) failed: rb = 0

Bug Blocker Resolved Fixed Mikhail Pershin Sarah Liu Wed, 22 Jan 2014 23:17:31 +0000 Thu, 16 May 2019 14:16:38 +0000 Fri, 9 May 2014 15:02:18 +0000 Lustre 2.6.0 Lustre 2.4.2 Lustre 2.5.1 Lustre 2.6.0 0 9 <p>Mike,<br/> Could you have a look and comment on this one?<br/> Thank you!</p> <p>yes</p> <p>per Alex comment this issue may happen when transaction is trying to do undo in case of some errors during processing. It was -28 (ENOSPC) error during llog testing and current transaction starts to undo. There can be other cases but with the same scenario.</p> <p>I have no idea how to fix that because we cannot just include all credits for undo in the current transaction because that will increase number of credits dramatically. This is somehow expected and turned off in production code, so it is not critical failure. I wonder why to have this assertion and break test runs if we know about that issue and can't make it works properly in all cases. Maybe Alex and/or Andreas have some ideas about how to handle this gracefully.</p> <p>Meanwhile, this particular case can be fixed by declaring attr_set operation in llog_osd_declare_write_rec(), that is missing and that is why it fails in case of error.</p> <p><a href="https://jira.whamcloud.com/browse/LU-4901" title="Remove triple buffer update from llog_osd_write_blob()" class="issue-link" data-issue-key="LU-4901"><del>LU-4901</del></a> may solve this issue naturally in other way</p> <p>Mike, I thought that the declare for each update also allowed the "undo" update in case of error handling? Is that not the case for these llog updates because they are potentially split across multiple transaction handles?</p> <p>I'd definitely prefer to fix this via <a href="https://jira.whamcloud.com/browse/LU-4901" title="Remove triple buffer update from llog_osd_write_blob()" class="issue-link" data-issue-key="LU-4901"><del>LU-4901</del></a> instead of adding more workarounds in the transaction credit accounting to handle this case. In that case, consider <a href="https://jira.whamcloud.com/browse/LU-4901" title="Remove triple buffer update from llog_osd_write_blob()" class="issue-link" data-issue-key="LU-4901"><del>LU-4901</del></a> a 2.6.0 blocker in order to fix this bug.</p> <p>Closing <a href="https://jira.whamcloud.com/browse/LU-4901" title="Remove triple buffer update from llog_osd_write_blob()" class="issue-link" data-issue-key="LU-4901"><del>LU-4901</del></a> so there is only a single blocker bug for this problem. This bug will be kept open as a 2.6 blocker. </p> <p>I agree that <a href="https://jira.whamcloud.com/browse/LU-4901" title="Remove triple buffer update from llog_osd_write_blob()" class="issue-link" data-issue-key="LU-4901"><del>LU-4901</del></a> is preferable but I didn't estimate how complex patch may be. At the same time the fix is simple and straightforward - there is just setattr declaration is missing. Therefore I will start with <a href="https://jira.whamcloud.com/browse/LU-4901" title="Remove triple buffer update from llog_osd_write_blob()" class="issue-link" data-issue-key="LU-4901"><del>LU-4901</del></a> patch but switch to the simple fix just to get rid of this issue if there will be any unexpected complexity.</p> <p><a href="http://review.whamcloud.com/#/c/10108/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/10108/</a> - patch removes 3-steps write for llog records, eliminating the extra credits need for error case.</p> <p>Is it possible to provide a workaround patch? I'm hitting this issue all the time by running racer.</p> <p>the link above is the patch, try it. If you need something simpler then add dt_declare_attr_set() in llog_osd_declare_write_rec() right after punch declaration.</p> <p>Patch landed to Master. Please reopen ticket if more work is needed.</p> <p>Hi,</p> <p>It seems we hit the same issue in lustre 2.4.2.<br/> Would it be possible to have the patch backported in b2_4 ?</p> <p>thanks,<br/> Grégoire.</p> <p>b2_5 path <a href="http://review.whamcloud.com/11751" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/11751</a></p> <p>Peter, this can be improved with <a href="http://review.whamcloud.com/#/c/12169/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/12169/</a> IMO</p> <p>Alex</p> <p>Then could you please open a new ticket to track that work and link it to this one?</p> <p>Thanks</p> <p>Peter</p> <p>Peter, I'm not sure how to link, but that's it - <a href="https://jira.whamcloud.com/browse/LU-5794" title="improve credit checks in osd-ldiskfs" class="issue-link" data-issue-key="LU-5794"><del>LU-5794</del></a></p> <p>thanks, Alex</p> <p>Thanks Alex. For future reference you find Link under the More tab <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>You can also add links between patches by clicking the "+" on the right side of the <tt>Issue Links</tt> header.</p> Duplicate LU-4899 Related LU-5794 LU-4901 LU-6556 LU-5303 LU-4183 Development Rank 1|hzwdg7: Rank (Obsolete) 12386 Severity