https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7045 Lustre <p>Test Lustre master branch on RHEL 7.1 client and server, conf-sanity test 81 hung and syslog on MDS showed that:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Jun 10 16:28:25 shadow-15vm8 kernel: ------------[ cut here ]------------ Jun 10 16:28:25 shadow-15vm8 kernel: WARNING: at /var/lib/jenkins/workspace/lustre-master/arch/x86_64/build_type/server/distro/el7/ib_stack/inkernel/BUILD/BUILD/lustre-2.7.54/ldiskfs/ext4_jbd2.c:260 __ldiskfs_handle_dirty_metadata+0x1c2/0x220 [ldiskfs]() Jun 10 16:28:25 shadow-15vm8 kernel: Modules linked in: osp(OF) mdd(OF) lod(OF) mdt(OF) lfsck(OF) mgs(OF) mgc(OF) osd_ldiskfs(OF) lquota(OF) fid(OF) fld(OF) ksocklnd(OF) ptlrpc(OF) obdclass(OF) lnet(OF) libcfs(OF) ldiskfs(OF) sha512_generic dm_mod nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd fscache xprtrdma sunrpc ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ppdev pcspkr serio_raw parport_pc virtio_balloon i2c_piix4 parport ext4 mbcache jbd2 ata_generic pata_acpi virtio_blk cirrus syscopyarea sysfillrect sysimgblt drm_kms_helper ttm 8139too virtio_pci 8139cp virtio_ring virtio mii drm ata_piix libata i2c_core floppy [last unloaded: libcfs] Jun 10 16:28:25 shadow-15vm8 kernel: Jun 10 16:28:25 shadow-15vm8 kernel: CPU: 1 PID: 18882 Comm: llog_process_th Tainted: GF W O-------------- 3.10.0-229.4.2.el7_lustre.g1fee634.x86_64 #1 Jun 10 16:28:25 shadow-15vm8 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007 Jun 10 16:28:25 shadow-15vm8 kernel: 0000000000000000 000000005e4d00bc ffff88006d6a77f0 ffffffff816050da Jun 10 16:28:25 shadow-15vm8 kernel: ffff88006d6a7828 ffffffff8106e34b ffff88000083fc98 ffff88007b5877e0 Jun 10 16:28:25 shadow-15vm8 kernel: ffff88005c362708 ffffffffa062745c 0000000000000325 ffff88006d6a7838 Jun 10 16:28:25 shadow-15vm8 kernel: Call Trace: Jun 10 16:28:25 shadow-15vm8 kernel: [&lt;ffffffff816050da&gt;] dump_stack+0x19/0x1b Jun 10 16:28:25 shadow-15vm8 kernel: [&lt;ffffffff8106e34b&gt;] warn_slowpath_common+0x6b/0xb0 Jun 10 16:28:25 shadow-15vm8 kernel: [&lt;ffffffff8106e49a&gt;] warn_slowpath_null+0x1a/0x20 Jun 10 16:28:25 shadow-15vm8 kernel: [&lt;ffffffffa05d26b2&gt;] __ldiskfs_handle_dirty_metadata+0x1c2/0x220 [ldiskfs] Jun 10 16:28:25 shadow-15vm8 kernel: [&lt;ffffffffa05e30c1&gt;] ldiskfs_getblk+0x131/0x200 [ldiskfs] Jun 10 16:28:25 shadow-15vm8 kernel: [&lt;ffffffffa05e31b7&gt;] ldiskfs_bread+0x27/0xc0 [ldiskfs] Jun 10 16:28:25 shadow-15vm8 kernel: [&lt;ffffffffa0bca919&gt;] osd_ldiskfs_write_record+0x169/0x350 [osd_ldiskfs] Jun 10 16:28:25 shadow-15vm8 kernel: [&lt;ffffffffa0bcabdf&gt;] osd_write+0xdf/0x200 [osd_ldiskfs] Jun 10 16:28:25 shadow-15vm8 kernel: [&lt;ffffffffa0765399&gt;] dt_record_write+0x39/0x120 [obdclass] Jun 10 16:28:25 shadow-15vm8 kernel: [&lt;ffffffffa0eed261&gt;] osp_write_local_file.isra.16+0x284/0x30c [osp] Jun 10 16:28:25 shadow-15vm8 kernel: [&lt;ffffffffa0eed7ca&gt;] osp_last_used_init+0x1e7/0xaa6 [osp] Jun 10 16:28:25 shadow-15vm8 kernel: [&lt;ffffffffa0ecef84&gt;] osp_init0.isra.17+0x1b44/0x2060 [osp] Jun 10 16:28:25 shadow-15vm8 kernel: [&lt;ffffffffa0ecf549&gt;] osp_device_alloc+0xa9/0x1a0 [osp] Jun 10 16:28:25 shadow-15vm8 kernel: [&lt;ffffffffa074b6c4&gt;] obd_setup+0x114/0x2a0 [obdclass] Jun 10 16:28:25 shadow-15vm8 kernel: [&lt;ffffffffa074f430&gt;] class_setup+0x2a0/0x820 [obdclass] Jun 10 16:28:25 shadow-15vm8 kernel: [&lt;ffffffffa0753b0c&gt;] class_process_config+0x224c/0x2e50 [obdclass] Jun 10 16:28:25 shadow-15vm8 kernel: [&lt;ffffffffa0571967&gt;] ? libcfs_debug_msg+0x57/0x80 [libcfs] Jun 10 16:28:25 shadow-15vm8 kernel: [&lt;ffffffffa0754f6f&gt;] class_config_llog_handler+0x85f/0x16e0 [obdclass] Jun 10 16:28:25 shadow-15vm8 kernel: [&lt;ffffffffa071945a&gt;] llog_process_thread+0x7aa/0xe90 [obdclass] Jun 10 16:28:25 shadow-15vm8 kernel: [&lt;ffffffffa071a4f0&gt;] ? llog_backup+0x500/0x500 [obdclass] Jun 10 16:28:25 shadow-15vm8 kernel: [&lt;ffffffffa071a53c&gt;] llog_process_thread_daemonize+0x4c/0x80 [obdclass] Jun 10 16:28:25 shadow-15vm8 kernel: [&lt;ffffffff8109739f&gt;] kthread+0xcf/0xe0 Jun 10 16:28:25 shadow-15vm8 kernel: [&lt;ffffffff810972d0&gt;] ? kthread_create_on_node+0x140/0x140 Jun 10 16:28:25 shadow-15vm8 kernel: [&lt;ffffffff81614f7c&gt;] ret_from_fork+0x7c/0xb0 Jun 10 16:28:25 shadow-15vm8 kernel: [&lt;ffffffff810972d0&gt;] ? kthread_create_on_node+0x140/0x140 Jun 10 16:28:25 shadow-15vm8 kernel: ---[ end trace ddd376db706c5663 ]--- Jun 10 16:28:25 shadow-15vm8 kernel: LDISKFS-fs: ldiskfs_getblk:805: aborting transaction: error 28 in __ldiskfs_handle_dirty_metadata Jun 10 16:28:25 shadow-15vm8 kernel: LDISKFS-fs error (device dm-0): ldiskfs_getblk:805: inode #132: block 16827: comm llog_process_th: journal_dirty_metadata failed: handle type 0 started at line 967, credits 6/0, errcode -28 Jun 10 16:28:25 shadow-15vm8 kernel: Aborting journal on device dm-0-8. Jun 10 16:28:25 shadow-15vm8 kernel: LDISKFS-fs (dm-0): Remounting filesystem read-only Jun 10 16:28:25 shadow-15vm8 kernel: LustreError: 18882:0:(osd_io.c:1645:osd_ldiskfs_write_record()) dm-0: error reading offset 84152 (block 20): rc = -28 Jun 10 16:28:25 shadow-15vm8 kernel: LDISKFS-fs error (device dm-0) in osd_trans_stop:1052: error 28 Jun 10 16:28:25 shadow-15vm8 kernel: LustreError: 18458:0:(osd_handler.c:829:osd_trans_commit_cb()) transaction @0xffff8800700b43c0 commit error: 2 Jun 10 16:28:25 shadow-15vm8 kernel: LustreError: 18882:0:(osd_handler.c:1055:osd_trans_stop()) lustre-MDT0000-osd: failed to stop transaction: rc = -28 Jun 10 16:28:25 shadow-15vm8 kernel: LustreError: 18882:0:(osp_dev.c:283:osp_init_last_objid()) lustre-OST2917-osc-MDT0000: can't initialize lov_objid: rc = -28 Jun 10 16:28:25 shadow-15vm8 kernel: LustreError: 18882:0:(osp_dev.c:369:osp_last_used_init()) lustre-OST2917-osc-MDT0000: Can not get ids -28 from old objid! Jun 10 16:28:25 shadow-15vm8 kernel: LustreError: 18882:0:(obd_config.c:558:class_setup()) setup lustre-OST2917-osc-MDT0000 failed (-28) Jun 10 16:28:25 shadow-15vm8 kernel: LustreError: 18882:0:(obd_config.c:1601:class_config_llog_handler()) MGC10.1.4.179@tcp: cfg command failed: rc = -28 Jun 10 16:28:25 shadow-15vm8 kernel: Lustre: cmd=cf003 0:lustre-OST2917-osc-MDT0000 1:lustre-OST2917_UUID 2:10.1.4.178@tcp Jun 10 16:28:25 shadow-15vm8 kernel: LustreError: 18477:0:(mgc_request.c:527:do_requeue()) failed processing log: -28 </pre> </div></div> <p>The Maloo instances:<br/> <a href="https://testing.hpdd.intel.com/sub_tests/ebb78d16-1044-11e5-8fc4-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/ebb78d16-1044-11e5-8fc4-5254006e85c2</a></p> RHEL 7.1 client and server LU-7045

NOT enough transaction credits for osp_write_local_file()

Bug Critical Resolved Fixed nasf nasf Wed, 26 Aug 2015 11:30:10 +0000 Wed, 14 Oct 2015 14:19:49 +0000 Wed, 14 Oct 2015 14:19:49 +0000 Lustre 2.8.0 Lustre 2.8.0 0 10 <p>Here are some failure instances:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LDISKFS-fs error (device dm-0): ldiskfs_getblk:805: inode #132: block 16827: comm llog_process_th: journal_dirty_metadata failed: handle type 0 started at line 967, credits 6/0, errcode -28 LDISKFS-fs error (device dm-0): ldiskfs_getblk:810: inode #123: block 17157: comm llog_process_th: journal_dirty_metadata failed: handle type 0 started at line 1083, credits 6/0, errcode -28 LDISKFS-fs error (device dm-0): ldiskfs_getblk:810: inode #123: block 17148: comm llog_process_th: journal_dirty_metadata failed: handle type 0 started at line 1083, credits 6/0, errcode -28 LDISKFS-fs error (device dm-0): ldiskfs_getblk:810: inode #133: block 16800: comm llog_process_th: journal_dirty_metadata failed: handle type 0 started at line 1083, credits 6/0, errcode -28 </pre> </div></div> <p>The directly reason for the failure is that when the the ldiskfs_bread() wanted to allocate new block for osp_write_local_file(), it found that all the declared 6 credits have been exhausted.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-keyword">static</span> ssize_t osd_declare_write(<span class="code-keyword">const</span> struct lu_env *env, struct dt_object *dt, <span class="code-keyword">const</span> struct lu_buf *buf, loff_t _pos, struct thandle *handle) { ... <span class="code-keyword">if</span> (osd_extents_enabled(sb, inode)) { /* * many concurrent threads may grow tree by the time * our transaction starts. so, consider 2 is a min depth * <span class="code-keyword">for</span> every level we may need to allocate a <span class="code-keyword">new</span> block * and take some entries from the old one. so, 3 blocks * to allocate (bitmap, gd, itself) + old block - 4 per * level. */ depth = inode != NULL ? ext_depth(inode) : 0; depth = max(depth, 1) + 1; credits = depth; /* <span class="code-keyword">if</span> not append, then split may need to modify * existing blocks moving entries into the <span class="code-keyword">new</span> ones */ <span class="code-keyword">if</span> (_pos == -1) credits += depth; <span class="code-comment">/* blocks to store data: bitmap,gd,itself */</span> credits += blocks * 3; } <span class="code-keyword">else</span> { credits = osd_calc_bkmap_credits(sb, inode, size, _pos, blocks); } /* <span class="code-keyword">if</span> inode is created as part of the transaction, * then it's counted already by the creation method */ <span class="code-keyword">if</span> (inode != NULL) credits++; ... } </pre> </div></div> <p>Generally, for OSP on the MDT, we will NOT enable "extents" feature, so the logic path should be: osd_declare_write() =&gt; osd_calc_bkmap_credits(). According to the block# (to be modified), we can know the write offset, and then we can know how the osd_declare_write() will prepare the transaction credits: it should be (1 + 2) * 3 + 1 = 10. But according to the failure log, the declared credits is 6, such value was calculated via the logic path with "extents" feature enable! That is conflict with our former conclusion for the OSP on the MDT. So I suspect that the "extents" feature has been enabled by wrong for some unknown reason. I will make a patch to verify that.</p> <p>whether extents feature is enabled on MDT or not shouldn't make credits calculation wrong. this logic is per-inode for the reason.</p> <p>If the "extents" is disabled, then the credits for osp_write_local_file() at the block 16xxx (or 17xxx) should be 10, otherwise, the credits will be 6. According to the error message, it is 6, so the "extents" feature has been enabled when declare the credits. But on the MDT, it should NOT.</p> <p>What I suspect is that: from the inode view, the "extents" is NOT enabled, so when allocate the block, it used traditional triple-indirect blocks mechanism; but from the super block view, it is enabled, so when declare the credits, it used "extents" mode. Such difference will cause the credits trouble. Currently, I cannot find which code will cause such confused logic, but I will make patch to verify whether such case exists or not.</p> <p>extents is per-inode and once inode is allocated osd_extents_enabled() uses that specific inode to learn the exact method:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">if</span> (inode != NULL) { <span class="code-keyword">if</span> (LDISKFS_I(inode)-&gt;i_flags &amp; LDISKFS_EXTENTS_FL) <span class="code-keyword">return</span> 1; } <span class="code-keyword">else</span> <span class="code-keyword">if</span> (LDISKFS_HAS_INCOMPAT_FEATURE(sb, LDISKFS_FEATURE_INCOMPAT_EXTENTS)) { <span class="code-keyword">return</span> 1; } </pre> </div></div> <p>But we do not know whether the "inode" is NULL or not when declare. It can be NULL.</p> <p>ldiskfs_new_inode():</p> <blockquote> <p> if (LDISKFS_HAS_INCOMPAT_FEATURE(sb, LDISKFS_FEATURE_INCOMPAT_EXTENTS)) {<br/> /* set extent flag only for directory, file and normal symlink*/<br/> if (S_ISDIR(mode) || S_ISREG(mode) || S_ISLNK(mode)) {<br/> ldiskfs_set_inode_flag(inode, LDISKFS_INODE_EXTENTS);</p></blockquote> <p>I am also confused considering the ldiskfs_new_inode(). But if "extents" was not enabled, how to explain the declared credits is "6", not "10"?</p> <p>I'd rather think it was osd_calc_bkmap_credits() returning (1 + 2), then osd_declare_write() adding 1, so 4 and the remaining 2 came from osd_declare_inode_qid()</p> <p>Hi Nasf,</p> <p>Could you please create a patch for master branch? Thank you.</p> <p>Fan Yong (fan.yong@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/16330" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16330</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7045" title="NOT enough transaction credits for osp_write_local_file()" class="issue-link" data-issue-key="LU-7045"><del>LU-7045</del></a> osd: enough credits for single indirect block write<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 95e2c1d58959af73dd6c08357e398f5593d7d4aa</p> <p>Debug patch shows that when the issue happened, the "extents" feature is disabled on the MDT. So the block allocation follows the traditional triple indirect blocks mechanism. The patch <a href="http://review.whamcloud.com/16330" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16330</a> should have fixed the issue.</p> <p>triple indirect? the patch changes single-indirect logic only.</p> <p>I mean that the non-extents block allocation method. In our case, the patch fixes the improper logic that was used for optimising single indirect block.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/16330/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16330/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7045" title="NOT enough transaction credits for osp_write_local_file()" class="issue-link" data-issue-key="LU-7045"><del>LU-7045</del></a> osd: enough credits for single indirect block write<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 85c6c099d83b19480dd4160de57e7ffac5b312af</p> <p>Landed for 2.8</p> Blocker LU-6843 Related Development Rank 1|hzxldz: Rank (Obsolete) 9223372036854775807 Severity