https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2991 Lustre <p>1 mds, 2 oss with 150 osts each<br/> 1 client mounted doing<br/> for i in $(seq 1 300); do<br/> lfs setstripe /mnt/lustre/dir$i<br/> touch /mnt/lustre/dir$i/file<br/> lfs getstripe /mnt/lustre/dir$i<br/> done</p> <p>rm -rf /mnt/lustre/dir*</p> <p>LustreError: 3627:0:(osd_internal.h:909:osd_trans_exec_op()) ASSERTION( oti-&gt;oti_declare_ops_rb<span class="error">&#91;rb&#93;</span> &gt; 0 ) failed: rb = 2<br/> LustreError: 3627:0:(osd_internal.h:909:osd_trans_exec_op()) LBUG<br/> Pid: 3627, comm: mdt02_001</p> <p>Call Trace:<br/> <span class="error">&#91;&lt;ffffffffa04cd895&gt;&#93;</span> libcfs_debug_dumpstack+0x55/0x80 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa04cde97&gt;&#93;</span> lbug_with_loc+0x47/0xb0 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0d1c688&gt;&#93;</span> osd_trans_exec_op+0x128/0x160 <span class="error">&#91;osd_ldiskfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0d26878&gt;&#93;</span> osd_xattr_set+0x278/0x380 <span class="error">&#91;osd_ldiskfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0edef30&gt;&#93;</span> lod_generate_and_set_lovea+0x350/0x720 <span class="error">&#91;lod&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0eea6d6&gt;&#93;</span> lod_striping_create+0x1e6/0x320 <span class="error">&#91;lod&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0eea983&gt;&#93;</span> lod_object_create+0x173/0x260 <span class="error">&#91;lod&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0c3b540&gt;&#93;</span> mdd_object_create_internal+0xa0/0x1c0 <span class="error">&#91;mdd&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0c4e503&gt;&#93;</span> mdd_create+0x963/0x1470 <span class="error">&#91;mdd&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0ee7460&gt;&#93;</span> ? lod_index_lookup+0x0/0x30 <span class="error">&#91;lod&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0e4bf84&gt;&#93;</span> mdt_reint_open+0x13d4/0x20d0 <span class="error">&#91;mdt&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa04ea7ae&gt;&#93;</span> ? upcall_cache_get_entry+0x28e/0x860 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa07bb7ec&gt;&#93;</span> ? lustre_msg_add_version+0x6c/0xc0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0638cf0&gt;&#93;</span> ? lu_ucred+0x20/0x30 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0e368c1&gt;&#93;</span> mdt_reint_rec+0x41/0xe0 <span class="error">&#91;mdt&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0e2ff23&gt;&#93;</span> mdt_reint_internal+0x4e3/0x7d0 <span class="error">&#91;mdt&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0e304dd&gt;&#93;</span> mdt_intent_reint+0x1ed/0x4f0 <span class="error">&#91;mdt&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0e2c0ae&gt;&#93;</span> mdt_intent_policy+0x3ae/0x750 <span class="error">&#91;mdt&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0774351&gt;&#93;</span> ldlm_lock_enqueue+0x361/0x8d0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa079a3e7&gt;&#93;</span> ldlm_handle_enqueue0+0x4f7/0x10b0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0e2c586&gt;&#93;</span> mdt_enqueue+0x46/0x110 <span class="error">&#91;mdt&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0e21168&gt;&#93;</span> mdt_handle_common+0x628/0x1620 <span class="error">&#91;mdt&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0e5a285&gt;&#93;</span> mds_regular_handle+0x15/0x20 <span class="error">&#91;mdt&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa07cc27c&gt;&#93;</span> ptlrpc_server_handle_request+0x41c/0xdf0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa04ce5de&gt;&#93;</span> ? cfs_timer_arm+0xe/0x10 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa04dfd8f&gt;&#93;</span> ? lc_watchdog_touch+0x6f/0x170 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa07c38b9&gt;&#93;</span> ? ptlrpc_wait_event+0xa9/0x290 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff81052223&gt;&#93;</span> ? __wake_up+0x53/0x70<br/> <span class="error">&#91;&lt;ffffffffa07cd7c5&gt;&#93;</span> ptlrpc_main+0xb75/0x1870 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa07ccc50&gt;&#93;</span> ? ptlrpc_main+0x0/0x1870 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff8100c0ca&gt;&#93;</span> child_rip+0xa/0x20<br/> <span class="error">&#91;&lt;ffffffffa07ccc50&gt;&#93;</span> ? ptlrpc_main+0x0/0x1870 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa07ccc50&gt;&#93;</span> ? ptlrpc_main+0x0/0x1870 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff8100c0c0&gt;&#93;</span> ? child_rip+0x0/0x20</p> <p>LustreError: dumping log to /tmp/lustre-log.1363711711.3627</p> <a href="https://build.whamcloud.com/job/lustre-master/1329/">https://build.whamcloud.com/job/lustre-master/1329/</a> LU-2991

(osd_internal.h:909:osd_trans_exec_op()) ASSERTION( oti->oti_declare_ops_rb[rb] > 0 ) failed: rb = 2

Bug Minor Resolved Fixed Bruno Faccini Minh Diep MB Tue, 19 Mar 2013 21:26:13 +0000 Mon, 15 Apr 2013 08:27:28 +0000 Mon, 15 Apr 2013 08:27:28 +0000 Lustre 2.4.0 Lustre 2.4.0 0 7 <p>Looks like the transaction credit debugging caught a case where we didn't declare any credits for the rollback or something. The <a href="http://review.whamcloud.com/5698" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5698</a> change will disable this accounting for 2.3.90 and beyond, but I suspect we are missing some error handling code?</p> <p>Minh,<br/> Where/How have you been able to run such "2 oss with 150 osts each" configuration ?<br/> Is it easily reproducible ?<br/> Have you been able to get a crash-dump from occurrence(s) you already got ?</p> <p>I ran this on our lab creating osts on LVM. I see if I can get the crash dump. I'll ping you offline to see if you can access the system</p> <p>Assert comes from Patch/commit 28fc8fcc8bfe092c9a1a8c192ab6fe22d92820e7 for <a href="https://jira.whamcloud.com/browse/LU-2640" title="deactivate OSD_EXEC_OP() operation accounting if operation is being undone" class="issue-link" data-issue-key="LU-2640"><del>LU-2640</del></a>.<br/> Could it be the number of credits exceeds/overflows the size of entries (unsigned char) in oti_declare_ops_rb[] ??</p> <p>Both the transaction credits counters declaration and the Assert comes from Patch/commit 28fc8fcc8bfe092c9a1a8c192ab6fe22d92820e7 for <a href="https://jira.whamcloud.com/browse/LU-2640" title="deactivate OSD_EXEC_OP() operation accounting if operation is being undone" class="issue-link" data-issue-key="LU-2640"><del>LU-2640</del></a>.</p> <p>with wide striping - yes. </p> <p>Thank's Alex, and assume this may have occur whith the 150 OSTs Minh was using !</p> <p>I changed the size of the transaction credits counters from uchar to ushort within patch <a href="http://review.whamcloud.com/5830" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5830</a> for master I just pushed to gerrit.</p> <p>Minh, if no build failures (I still expect side-effects during build+run when changing size !!) can you give it a try ??</p> <p>Humm, tests of build for my patch failed in sanity/test_24v with same LBUG :</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>04:13:52:Lustre: DEBUG MARKER: == sanity test 24v: list directory with large files (handle hash collision, bug: 17560) == 04:04:38 (1364209478) 04:13:52:LustreError: 11418:0:(osd_internal.h:909:osd_trans_exec_op()) ASSERTION( oti-&gt;oti_declare_ops_rb[rb] &gt; 0 ) failed: rb = 5 04:13:52:LustreError: 11418:0:(osd_internal.h:909:osd_trans_exec_op()) LBUG 04:13:52:Pid: 11418, comm: mdt00_000 04:13:52: 04:13:53:Call Trace: 04:13:53: [&lt;ffffffffa04d7895&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] 04:13:53: [&lt;ffffffffa04d7e97&gt;] lbug_with_loc+0x47/0xb0 [libcfs] 04:13:53: [&lt;ffffffffa0d44360&gt;] osd_object_ref_del+0x2d0/0x310 [osd_ldiskfs] 04:13:53: [&lt;ffffffffa0f157fb&gt;] lod_ref_del+0x3b/0xd0 [lod] 04:13:53: [&lt;ffffffffa0c6acd5&gt;] mdo_ref_del+0x35/0xc0 [mdd] 04:13:54: [&lt;ffffffffa0c72446&gt;] mdd_unlink+0x6c6/0xe20 [mdd] 04:13:54: [&lt;ffffffffa0e61a18&gt;] mdo_unlink+0x18/0x50 [mdt] 04:13:54: [&lt;ffffffffa0e64cd9&gt;] mdt_reint_unlink+0x739/0xfd0 [mdt] 04:13:54: [&lt;ffffffffa0e616d1&gt;] mdt_reint_rec+0x41/0xe0 [mdt] 04:13:54: [&lt;ffffffffa0e5ad53&gt;] mdt_reint_internal+0x4e3/0x7d0 [mdt] 04:13:55: [&lt;ffffffffa0e5b084&gt;] mdt_reint+0x44/0xe0 [mdt] 04:13:55: [&lt;ffffffffa0e49078&gt;] mdt_handle_common+0x648/0x1660 [mdt] 04:13:56: [&lt;ffffffffa0e85125&gt;] mds_regular_handle+0x15/0x20 [mdt] 04:13:56: [&lt;ffffffffa07f91dc&gt;] ptlrpc_server_handle_request+0x41c/0xdf0 [ptlrpc] 04:13:56: [&lt;ffffffffa04d85de&gt;] ? cfs_timer_arm+0xe/0x10 [libcfs] 04:13:57: [&lt;ffffffffa07f0819&gt;] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] 04:13:57: [&lt;ffffffff81052223&gt;] ? __wake_up+0x53/0x70 04:13:57: [&lt;ffffffffa07fa725&gt;] ptlrpc_main+0xb75/0x1870 [ptlrpc] 04:13:58: [&lt;ffffffffa07f9bb0&gt;] ? ptlrpc_main+0x0/0x1870 [ptlrpc] 04:13:58: [&lt;ffffffff8100c0ca&gt;] child_rip+0xa/0x20 04:13:58: [&lt;ffffffffa07f9bb0&gt;] ? ptlrpc_main+0x0/0x1870 [ptlrpc] 04:13:58: [&lt;ffffffffa07f9bb0&gt;] ? ptlrpc_main+0x0/0x1870 [ptlrpc] 04:13:58: [&lt;ffffffff8100c0c0&gt;] ? child_rip+0x0/0x20 04:13:58: 04:13:59:Kernel panic - not syncing: LBUG 04:13:59:Pid: 11418, comm: mdt00_000 Not tainted 2.6.32-279.19.1.el6_lustre.gc4681d8.x86_64 #1 04:13:59:Call Trace: 04:14:00: [&lt;ffffffff814e9811&gt;] ? panic+0xa0/0x168 04:14:00: [&lt;ffffffffa04d7eeb&gt;] ? lbug_with_loc+0x9b/0xb0 [libcfs] 04:14:00: [&lt;ffffffffa0d44360&gt;] ? osd_object_ref_del+0x2d0/0x310 [osd_ldiskfs] 04:14:00: [&lt;ffffffffa0f157fb&gt;] ? lod_ref_del+0x3b/0xd0 [lod] 04:14:00: [&lt;ffffffffa0c6acd5&gt;] ? mdo_ref_del+0x35/0xc0 [mdd] 04:14:01: [&lt;ffffffffa0c72446&gt;] ? mdd_unlink+0x6c6/0xe20 [mdd] 04:14:01: [&lt;ffffffffa0e61a18&gt;] ? mdo_unlink+0x18/0x50 [mdt] 04:14:01: [&lt;ffffffffa0e64cd9&gt;] ? mdt_reint_unlink+0x739/0xfd0 [mdt] 04:14:01: [&lt;ffffffffa0e616d1&gt;] ? mdt_reint_rec+0x41/0xe0 [mdt] 04:14:01: [&lt;ffffffffa0e5ad53&gt;] ? mdt_reint_internal+0x4e3/0x7d0 [mdt] 04:14:01: [&lt;ffffffffa0e5b084&gt;] ? mdt_reint+0x44/0xe0 [mdt] 04:14:02: [&lt;ffffffffa0e49078&gt;] ? mdt_handle_common+0x648/0x1660 [mdt] 04:14:02: [&lt;ffffffffa0e85125&gt;] ? mds_regular_handle+0x15/0x20 [mdt] 04:14:02: [&lt;ffffffffa07f91dc&gt;] ? ptlrpc_server_handle_request+0x41c/0xdf0 [ptlrpc] 04:14:02: [&lt;ffffffffa04d85de&gt;] ? cfs_timer_arm+0xe/0x10 [libcfs] 04:14:02: [&lt;ffffffffa07f0819&gt;] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc] 04:14:03: [&lt;ffffffff81052223&gt;] ? __wake_up+0x53/0x70 04:14:03: [&lt;ffffffffa07fa725&gt;] ? ptlrpc_main+0xb75/0x1870 [ptlrpc] 04:14:03: [&lt;ffffffffa07f9bb0&gt;] ? ptlrpc_main+0x0/0x1870 [ptlrpc] 04:14:03: [&lt;ffffffff8100c0ca&gt;] ? child_rip+0xa/0x20 04:14:03: [&lt;ffffffffa07f9bb0&gt;] ? ptlrpc_main+0x0/0x1870 [ptlrpc] 04:14:03: [&lt;ffffffffa07f9bb0&gt;] ? ptlrpc_main+0x0/0x1870 [ptlrpc] 04:14:03: [&lt;ffffffff8100c0c0&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p>Unless my very little change of counter's size introduced a regression, I am afraid that recent landed patches from <a href="https://jira.whamcloud.com/browse/LU-2640" title="deactivate OSD_EXEC_OP() operation accounting if operation is being undone" class="issue-link" data-issue-key="LU-2640"><del>LU-2640</del></a> may need some re-work ...<br/> I will try to run this build on Toro and reproduce in order to get a crash-dump for analysis, but Minh if you already have been able to reproduce with your own test/configuration, just tell me.</p> <p>I am trying to reproduce on a single Toro node and loop devices but no success/LBUG till now ...<br/> I added "max_loop=164" Kernel boot parameter and then ran the following script based on your reproducer <br/> original text :</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>export OSTCOUNT=150 /usr/lib64/lustre/tests/llmount.sh while true ; do for i in $(seq 1 300); do mkdir /mnt/lustre/dir$i lfs setstripe -c 150 /mnt/lustre/dir$i touch /mnt/lustre/dir$i/file lfs getstripe /mnt/lustre/dir$i done rm -rf /mnt/lustre/dir* done </pre> </div></div> <p>may be I need more Nodes/VMs to trigger it ??</p> <p>Yes, I think you might need more than one node to reproduce it. Did auto test provide a crash dump?</p> <p>Yes, according to MDS Console log, but then how/where can I find it ?</p> <p>I hit this during your patch test. I am not sure if it's related, likely not. I will reproduce this bug on master and collect the crash dump.</p> <p>-----------<del>[ cut here ]</del>-----------<br/> kernel BUG at fs/jbd2/transaction.c:1033!<br/> invalid opcode: 0000 <a href="#1" target="_blank" rel="noopener">1</a> SMP<br/> last sysfs file: /sys/devices/system/cpu/cpu15/cache/index2/shared_cpu_map<br/> CPU 8<br/> Modules linked in: osp(U) lod(U) mdt(U) mgs(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) lquota(U) mdd(U) lustre(U) lov(U) osc(U) mdc(U) fid(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) sha512_generic sha256_generic libcfs(U) ldiskfs(U) jbd2 nfsd exportfs nfs lockd fscache nfs_acl auth_rpcgss sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa microcode serio_raw mlx4_ib ib_mad ib_core mlx4_en mlx4_core i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support ioatdma i7core_edac edac_core ses enclosure sg igb dca ext3 jbd mbcache sr_mod cdrom sd_mod crc_t10dif pata_acpi ata_generic ata_piix mpt2sas scsi_transport_sas raid_class dm_mirror dm_region_hash dm_log dm_mod <span class="error">&#91;last unloaded: scsi_wait_scan&#93;</span></p> <p>Pid: 674, comm: mdt00_003 Not tainted 2.6.32-279.19.1.el6_lustre.gc4681d8.x86_64 #1 Supermicro X8DTH-i/6/iF/6F/X8DTH<br/> RIP: 0010:<span class="error">&#91;&lt;ffffffffa044c86d&gt;&#93;</span> <span class="error">&#91;&lt;ffffffffa044c86d&gt;&#93;</span> jbd2_journal_dirty_metadata+0x10d/0x150 <span class="error">&#91;jbd2&#93;</span><br/> RSP: 0018:ffff88032d16d6b0 EFLAGS: 00010246<br/> RAX: ffff88035d5bf380 RBX: ffff880359dd4678 RCX: ffff880f985172d0<br/> RDX: 0000000000000000 RSI: ffff880f985172d0 RDI: 0000000000000000<br/> RBP: ffff88032d16d6d0 R08: a010000000000000 R09: f049b2734454b402<br/> R10: 0000000000000000 R11: ffff880eb0d34000 R12: ffff88038e527ba8<br/> R13: ffff880f985172d0 R14: ffff8808298c2800 R15: 0000000000001000<br/> FS: 00007fac03683700(0000) GS:ffff880028280000(0000) knlGS:0000000000000000<br/> CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b<br/> CR2: 00000000006dbf98 CR3: 0000000826eb6000 CR4: 00000000000006e0<br/> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000<br/> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400<br/> Process mdt00_003 (pid: 674, threadinfo ffff88032d16c000, task ffff88032d213540)<br/> Stack:<br/> ffff880359dd4678 ffffffffa0d82b50 ffff880f985172d0 0000000000000000<br/> &lt;d&gt; ffff88032d16d710 ffffffffa04721bb ffff88032d16d710 ffffffffa047f898<br/> &lt;d&gt; 0000000000001000 0000000000001000 0000000000001000 ffff880f985172d0<br/> Call Trace:<br/> <span class="error">&#91;&lt;ffffffffa04721bb&gt;&#93;</span> __ldiskfs_handle_dirty_metadata+0x7b/0x100 <span class="error">&#91;ldiskfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa047f898&gt;&#93;</span> ? ldiskfs_bread+0x18/0x80 <span class="error">&#91;ldiskfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0d6d1ec&gt;&#93;</span> osd_ldiskfs_write_record+0xec/0x330 <span class="error">&#91;osd_ldiskfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0d6e038&gt;&#93;</span> osd_write+0x148/0x2a0 <span class="error">&#91;osd_ldiskfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa066aeb5&gt;&#93;</span> dt_record_write+0x45/0x130 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0c8acd0&gt;&#93;</span> ? md_capainfo+0x20/0x30 <span class="error">&#91;mdd&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa063788e&gt;&#93;</span> llog_osd_write_blob+0x2fe/0x730 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa063b157&gt;&#93;</span> llog_osd_write_rec+0x6b7/0x1200 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0d679a5&gt;&#93;</span> ? iam_path_fini+0x25/0x30 <span class="error">&#91;osd_ldiskfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa06073b8&gt;&#93;</span> llog_write_rec+0xc8/0x290 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa060f57d&gt;&#93;</span> llog_cat_add_rec+0xad/0x480 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa06071b1&gt;&#93;</span> llog_add+0x91/0x1d0 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0f71ef7&gt;&#93;</span> osp_sync_add_rec+0x247/0x8a0 <span class="error">&#91;osp&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0d650af&gt;&#93;</span> ? osd_oi_delete+0x2af/0x4b0 <span class="error">&#91;osd_ldiskfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0f725fb&gt;&#93;</span> osp_sync_add+0x7b/0x80 <span class="error">&#91;osp&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0f664e6&gt;&#93;</span> osp_object_destroy+0x106/0x150 <span class="error">&#91;osp&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0f22347&gt;&#93;</span> lod_object_destroy+0x1a7/0x350 <span class="error">&#91;lod&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0c83a19&gt;&#93;</span> mdd_finish_unlink+0x229/0x380 <span class="error">&#91;mdd&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0c86748&gt;&#93;</span> mdd_unlink+0x9c8/0xe20 <span class="error">&#91;mdd&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0e6ba18&gt;&#93;</span> mdo_unlink+0x18/0x50 <span class="error">&#91;mdt&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0e6ecd9&gt;&#93;</span> mdt_reint_unlink+0x739/0xfd0 <span class="error">&#91;mdt&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0e6b6d1&gt;&#93;</span> mdt_reint_rec+0x41/0xe0 <span class="error">&#91;mdt&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0e64d53&gt;&#93;</span> mdt_reint_internal+0x4e3/0x7d0 <span class="error">&#91;mdt&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0e65084&gt;&#93;</span> mdt_reint+0x44/0xe0 <span class="error">&#91;mdt&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0e53078&gt;&#93;</span> mdt_handle_common+0x648/0x1660 <span class="error">&#91;mdt&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0e8f125&gt;&#93;</span> mds_regular_handle+0x15/0x20 <span class="error">&#91;mdt&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa07fc1dc&gt;&#93;</span> ptlrpc_server_handle_request+0x41c/0xdf0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa04f45de&gt;&#93;</span> ? cfs_timer_arm+0xe/0x10 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0505d8f&gt;&#93;</span> ? lc_watchdog_touch+0x6f/0x170 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa07f3819&gt;&#93;</span> ? ptlrpc_wait_event+0xa9/0x290 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff81052223&gt;&#93;</span> ? __wake_up+0x53/0x70<br/> <span class="error">&#91;&lt;ffffffffa07fd725&gt;&#93;</span> ptlrpc_main+0xb75/0x1870 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa07fcbb0&gt;&#93;</span> ? ptlrpc_main+0x0/0x1870 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff8100c0ca&gt;&#93;</span> child_rip+0xa/0x20<br/> <span class="error">&#91;&lt;ffffffffa07fcbb0&gt;&#93;</span> ? ptlrpc_main+0x0/0x1870 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa07fcbb0&gt;&#93;</span> ? ptlrpc_main+0x0/0x1870 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff8100c0c0&gt;&#93;</span> ? child_rip+0x0/0x20<br/> Code: c6 9c 03 00 00 4c 89 f7 e8 a1 ff 09 e1 48 8b 33 ba 01 00 00 00 4c 89 e7 e8 11 ec ff ff 4c 89 f0 66 ff 00 66 66 90 e9 73 ff ff ff &lt;0f&gt; 0b eb fe 0f 0b eb fe 0f 0b 66 0f 1f 84 00 00 00 00 00 eb f5<br/> RIP <span class="error">&#91;&lt;ffffffffa044c86d&gt;&#93;</span> jbd2_journal_dirty_metadata+0x10d/0x150 <span class="error">&#91;jbd2&#93;</span><br/> RSP &lt;ffff88032d16d6b0&gt;</p> <p>I will file a separate bug if I hit this on master</p> <p>According to stack-trace, I don't think it is related, but as I already said, with item size changes some weird consequences can show-up ...</p> <p>Also, Chris confirmed that actually auto-test generated crash-dumps are not saved, it is in their to-do list.</p> <p>Thus I rely on your testing to reproduce the LBUG, in // I try to build a configuration with more Nodes/VMs too, like the failed auto-test one.</p> <p>I have reproduced it on master and got the debug log with debug=-1. I also did a crash dump but since this is a LBUG, not sure if a crash dump is useful.</p> <p>crash-dump is not really useful in this case since it can only confirm that oti-&gt;oti_declare_ops_rb<span class="error">&#91;OSD_OT_XATTR_SET&#93;</span> = 0 which caused the Assert to trigger.</p> <p>But the lustre-logs are much more interesting since they confirm that the thread that trigger the Assert was gathering transaction credits stats for a work-load where wide-striping was used, and each time near the 256 (in fact 253 + ??) "fatal" value. So I still suspect an overflow due to uchar size ...</p> <p>Minh, I know that for the crash-dump you were running with a genuine master build, but was it also the case for the full/debug=1 trace case ?</p> <p>In fact, I would like to know if you got such Assert/LBUG when running with my patch ??</p> <p>to check the theory about uchar size, you could put LASSERT(currect_value &lt; 255); just before increment?</p> <p>You are right Alex, I thought I can find it by counting accurate traces in the full debug log from Minh, but seems we miss these traces in fact ...</p> <p>Minh, I just submitted <a href="http://review.whamcloud.com/5869" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5869</a> to trigger if we reach UCHAR_MAX, can you give it a try and expose it to your reproducer+configuration ??</p> <p>Ok, fact that we can reach UCHAR_MAX and overflow oti_declare_ops counter has been definitely proven.</p> <p>Then, changing the size of both oti_declare_ops/oti_declare_ops_rb from uchar to ushort, as per 1st patch/change 5830, seems to allow us to go further, but then we finally trigger the same Assert, now because oti_rollback is true but oti_declare_ops_rb is Null !!…</p> <p>Seems that we missed something in the rollback logic … So, we continue to debug by adding traces/prints to see how counters progress and find where things break.</p> <p>notice the checks in rollback path are not quite correct. in short, from ldiskfs/zfs point of view, there is no point to declare rollback changes: if you have insert for a name declared, then you'll be able to remove that name as part of rollback. this is why we plan to just disable these checks in production (until a better mechanism to track changes is implemented).</p> <p>Alex, can you confirm that if we disable these check in production, this ASSERTION will no happen?</p> <p>the assertions in OSD - yes. but I think we still need to understand the root cause for kernel BUG at fs/jbd2/transaction.c:1033!</p> <p>Minh,</p> <p>Yes, undefine of compile-time OSD_TRACK_DECLARES will fully disable the concerned transaction credits counters feature/code.<br/> Concerning the "fs/jbd2/transaction.c:1033" kernel BUG()/Oops, am I wrong when I think it was a one-shoot and never re-occurred during the next heavy testing you ran ?<br/> Also did you get a crash-dump at that time ? And did you open a new ticket for it finally ?</p> <p>Disable by taking out the define OSD_TRACK_DECLARES results in compile error</p> <p>/home/minh.diep/build/lu2991/lustre/osd-ldiskfs/osd_handler.c: In function ‘osd_trans_create’:<br/> /home/minh.diep/build/lu2991/lustre/osd-ldiskfs/osd_handler.c:683: error: ‘struct osd_thread_info’ has no member named ‘oti_declare_ops’<br/> /home/minh.diep/build/lu2991/lustre/osd-ldiskfs/osd_handler.c:684: error: ‘struct osd_thread_info’ has no member named ‘oti_declare_ops_rb’<br/> /home/minh.diep/build/lu2991/lustre/osd-ldiskfs/osd_handler.c:685: error: ‘struct osd_thread_info’ has no member named ‘oti_declare_ops_cred’<br/> /home/minh.diep/build/lu2991/lustre/osd-ldiskfs/osd_handler.c:686: error: ‘struct osd_thread_info’ has no member named ‘oti_rollback’<br/> cc1: warnings being treated as errors<br/> /home/minh.diep/build/lu2991/lustre/osd-ldiskfs/osd_handler.c: In function ‘osd_trans_start’:<br/> /home/minh.diep/build/lu2991/lustre/osd-ldiskfs/osd_handler.c:717: error: unused variable ‘last_credits’<br/> /home/minh.diep/build/lu2991/lustre/osd-ldiskfs/osd_handler.c:716: error: unused variable ‘last_printed’</p> <p>Bruno, no bug was filed for the kernel BUG and haven't hit it again</p> <p>Humm, in osd_trans_create() function oti_declare_ops<span class="error">&#91;_rb,_cred&#93;</span> and rollback fields memset()s/initializations need to be put under #ifdef OSD_TRACK_DECLARES.</p> <p>Same for last_credits/last_printed variables declares in osd_trans_start().</p> <p>Then we may run your tests again and see if we trigger Oops in jbd2 again.</p> <p>ok, fixed the compile issue and ran the test for couple hours now. No ASSERTION. I will try to reproduce the Oops but in the mean time, this bug's priority can be lowered.</p> <p>Patch-set #3 submitted according to the fact patch-set #1 did not correctly initialized/memset()s the new sized fields.</p> <p>If it still shows the same wrong behavior we will definitely revert unset/undefine OSD_TRACK_DECLARES.</p> <p>I had to re-base patch #6 due to auto-tests failures like racer/test_1 <span class="error">&#91;LustreError: 31880:0:(file.c:930:ll_file_io_generic()) ASSERTION( io-&gt;u.ci_rw.crw_count == count ) failed: 21504 != 4194304&#93;</span> which has been fixed with <a href="https://jira.whamcloud.com/browse/LU-3035" title="Failure on racer: ASSERTION( io-&gt;u.ci_rw.crw_count == count ) failed: 785408 != 4194304 " class="issue-link" data-issue-key="LU-3035"><del>LU-3035</del></a> ...</p> <p>Wow too bad, now patch-set #6 experienced 2 auto-tests failures (sanity/test_56w linked to <a href="https://jira.whamcloud.com/browse/LU-3092" title="sanity 56w fails on current master." class="issue-link" data-issue-key="LU-3092"><del>LU-3092</del></a> and sanity/test_65ic linked to <a href="https://jira.whamcloud.com/browse/LU-3099" title="Test failure on test suite sanity, subtest test_65ic" class="issue-link" data-issue-key="LU-3099"><del>LU-3099</del></a>), so a new re-base is required to integrate <a href="https://jira.whamcloud.com/browse/LU-3057" title="Hang at sanity test_56x and test_56w" class="issue-link" data-issue-key="LU-3057"><del>LU-3057</del></a> fix since both <a href="https://jira.whamcloud.com/browse/LU-3092" title="sanity 56w fails on current master." class="issue-link" data-issue-key="LU-3092"><del>LU-3092</del></a>/<a href="https://jira.whamcloud.com/browse/LU-3099" title="Test failure on test suite sanity, subtest test_65ic" class="issue-link" data-issue-key="LU-3099"><del>LU-3099</del></a> have been dup'ed to it ...</p> <p>So, patch-set #7 just submitted after rebase ...</p> <p>Again 4x, and different from the preceding!, auto-tests failures. But all in review-zfs group, with at least 2 of them being known/perf ZFS issues (<a href="https://jira.whamcloud.com/browse/LU-2872" title="Test timeout failure on test suite sanity-quota test_1" class="issue-link" data-issue-key="LU-2872"><del>LU-2872</del></a>, <a href="https://jira.whamcloud.com/browse/LU-2547" title="test: recovery-small test_24a, test_24b: multiop didn&#39;t fail fsync: rc 0" class="issue-link" data-issue-key="LU-2547"><del>LU-2547</del></a>) still under investigations. Thus, considering that this bug+patch is pure osd-ldiskfs stuff and since all review (pure ldiskfs ?) group tests have been successful, we may reasonably think that patch is ok and verified ...</p> Development Rank 1|hzvlq7: Rank (Obsolete) 7290 Severity