[LU-15726] Introduce / use min journal credit for ldiskfs

• Status: Closed
• Project: Lustre
• Type: Bug
• Priority: Minor
• Reporter: Shaun Tancheff
• Assignee: Shaun Tancheff
• Resolution: Fixed

Description

lustre/ldiskfs consumes more journal credits than ext4. Try to place nice with jbd2 and increase the requested journal credits as needed.

Issue Links

• is related to: LU-13135 quick checks in OSDs to ignore quota ...
• is related to: LU-14195 Support for linux kernel version 5.10

Comments

[Gerrit Updater]

"Shaun Tancheff <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/47009
Subject: LU-15726 ldiskfs: Introduce and use min journal credits
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 0bcea9046aec1bdf022621395ca7693c50132cac

[Shaun Tancheff]

Not an improvement

[James A Simmons]

Starting with 5.10 kernels the way xattr credits is handled has changed so that the ext4-xattr-disable-credits-check.patch is not good enough to work around this issue. We need a real solution to this problem so I'm reopening this ticket.

[Xinliang Liu]

I tried the latest b2_15 branch(2.15.3-RC1) on 5.10 kernel, it seems that without patch https://review.whamcloud.com/47009 it can't mount client(all-in-one) or start MDS (muli-node). So branch b2_15 must lack of a patch or some patches from branch master for fixing this credit related issue. Which ones?

And here the warning log from the kernel:

[ 8189.170458] ------------[ cut here ]------------
[ 8189.170504] WARNING: CPU: 0 PID: 115468 at /tmp/rpmbuild-lustre-openeuler-AL963B8M/BUILD/lustre-2.15.3_RC1_5_g4aaae55_dirty/ldiskfs/ext4_jbd2.c:336 __ldiskfs_handle_dirty_metadata+0x18c/0x2e0 [ldiskfs]
[ 8189.170506] Modules linked in: ofd(OE) ost(OE) osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) osd_ldiskfs(OE) lquota(OE) loop ldiskfs(OE) lustre(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) kso
cklnd(OE) lnet(OE) libcfs(OE) dm_flakey dm_mod crc32_generic rfkill sunrpc virtio_balloon vfat fat sch_fq_codel fuse ext4 mbcache jbd2 virtio_gpu virtio_net virtio_dma_buf net_failover virtio_blk failover ghash_ce sha2_ce sha256_arm64 sha1_ce virtio_pci virtio_pci_
modern_dev virtio_mmio virtio_rng virtio virtio_ring aes_neon_bs aes_neon_blk aes_ce_blk crypto_simd cryptd aes_ce_cipher [last unloaded: libcfs]
[ 8189.170583] CPU: 0 PID: 115468 Comm: mdt00_001 Kdump: loaded Tainted: G        W  OE     5.10.0-152.0.0.78.oe2203sp2.aarch64 #1
[ 8189.170585] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015
[ 8189.170588] pstate: 60000005 (nZCv daif -PAN -UAO -TCO BTYPE=--)
[ 8189.170610] pc : __ldiskfs_handle_dirty_metadata+0x18c/0x2e0 [ldiskfs]
[ 8189.170631] lr : __ldiskfs_handle_dirty_metadata+0x9c/0x2e0 [ldiskfs]
[ 8189.170632] sp : ffff8000109e36a0
[ 8189.170634] x29: ffff8000109e36c0 x28: 0000000000000000
[ 8189.170638] x27: 0000000000000002 x26: 0000000000000001
[ 8189.170641] x25: 0000000000000001 x24: ffff0bf31c309af8
[ 8189.170645] x23: 0000000000000372 x22: ffffde4408547508
[ 8189.170648] x21: 00000000ffffffe4 x20: ffff0bf318c45a10
[ 8189.170651] x19: ffff0bf31c1ff1a0 x18: 0000000000000020
[ 8189.170654] x17: 0000000000000000 x16: ffffde4435035f10
[ 8189.170658] x15: ffffffffffffffff x14: 0000000000000000
[ 8189.170661] x13: 0000000000191000 x12: 0000000000000000
[ 8189.170665] x11: 0000000000000000 x10: 00000000ffffffff
[ 8189.170668] x9 : ffffde44084c254c x8 : ffff0bf34f812000
[ 8189.170671] x7 : 0000000000000000 x6 : 0000000000000000
[ 8189.170674] x5 : 61c8864680b583eb x4 : 0000000000116011
[ 8189.170678] x3 : ffff0bf31f327800 x2 : 0000000000000001
[ 8189.170681] x1 : 00000000007be000 x0 : 0000000000000030
[ 8189.170685] Call trace:
[ 8189.170706]  __ldiskfs_handle_dirty_metadata+0x18c/0x2e0 [ldiskfs]
[ 8189.170727]  ldiskfs_getblk+0x150/0x210 [ldiskfs]
[ 8189.170748]  ldiskfs_bread+0x1c/0xd4 [ldiskfs]
[ 8189.170765]  osd_ldiskfs_write_record+0x4a4/0x8fc [osd_ldiskfs]
[ 8189.170779]  osd_write+0x104/0x6e4 [osd_ldiskfs]
[ 8189.170842]  dt_record_write+0x38/0xf0 [obdclass]
[ 8189.170943]  tgt_client_data_write+0x12c/0x180 [ptlrpc]
[ 8189.171012]  tgt_client_data_update+0x4fc/0x86c [ptlrpc]
[ 8189.171079]  tgt_client_new+0x610/0xcb0 [ptlrpc]
[ 8189.171117]  mdt_obd_connect+0x5b0/0x940 [mdt]
[ 8189.171370]  target_handle_connect+0x10e4/0x3b00 [ptlrpc]
[ 8189.171465]  tgt_request_handle+0x174/0xd9c [ptlrpc]
[ 8189.171545]  ptlrpc_server_handle_request.isra.0+0x3d4/0x11fc [ptlrpc]
[ 8189.171613]  ptlrpc_main+0xdb0/0x1670 [ptlrpc]
[ 8189.171620]  kthread+0x108/0x13c
[ 8189.171624]  ret_from_fork+0x10/0x18
[ 8189.171626] ---[ end trace ce1929bc2ec68092 ]---
[ 8189.171631] LDISKFS-fs: ldiskfs_getblk:882: aborting transaction: error 28 in __ldiskfs_handle_dirty_metadata
[ 8189.174222] LDISKFS-fs error (device dm-0): ldiskfs_getblk:882: inode #91: block 31655: comm mdt00_001: journal_dirty_metadata failed: handle type 0 started at line 1982, credits 7/0, errcode -28
[ 8189.178341] Aborting journal on device dm-0-8.
[ 8189.179762] LDISKFS-fs (dm-0): Remounting filesystem read-only
[ 8189.181278] LustreError: 115468:0:(osd_io.c:2123:osd_ldiskfs_write_record()) lustre-MDT0000: error reading offset 8192 (block 2, size 128, offs 8192), credits 7/1: rc = -28
[ 8189.184623] LDISKFS-fs error (device dm-0) in osd_trans_stop:2092: error 28
[ 8189.184635] LustreError: 115449:0:(osd_handler.c:1790:osd_trans_commit_cb()) transaction @0x00000000ce92c156 commit error: 2
 

[Xinliang Liu]

After a long bisect on branch master, find out that  branch b2_15 with commit ef90a02d12  can run on kernel 5.10 with no crash.

But don't know why? Does this issue still exist in 5.10+ kernel for non root-owned files? Does anyone have any ideas on this? @Alex Zhuravlev

Anyway, just cherry-picked it to branch b2_15 and make a note it is related to this issue: https://review.whamcloud.com/c/fs/lustre-release/+/51776

[James A Simmons]

I noticed the same thing for Ubuntu 5.15 kernels. Patch 51776 fixes this issue. Shaun can you close this ticket.

[Shaun Tancheff]

Resolved with LU-13135