[LU-1842] Quota enforcement landing

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.4.0
• Fix Version/s: Lustre 2.4.0, Lustre 2.1.4, Lustre 1.8.9
• Type: New Feature
• Priority: Minor
• Reporter: Johann Lombardi (Inactive)
• Assignee: Johann Lombardi (Inactive)
• Resolution: Fixed

Sub-Tasks

Key	Summary	Type	Status	Assignee
LU-1318	Quota Enforcement DLM&FID	Technical task	Closed	Johann Lombardi
LU-1789	Lightweight connection support	Technical task	Resolved	Johann Lombardi
LU-1818	configuration-related changes for the...	Technical task	Resolved	Niu Yawei
LU-1830	defects in ldiskfs quota accounting i...	Technical task	Resolved	Niu Yawei
LU-1920	Create and attach Test Plan for quota...	Technical task	Resolved	Cliff White

• Key: Summary
• Summary: Type
• Type: Status
• Status: Assignee

Description

Tracker bug for landing orion_quota to master.

Issue Links

• is duplicated by: LU-1300 Quota enforcement support for OSD Res...
• is related to: LU-1789 Lightweight connection support
• is related to: LU-1318 Quota Enforcement DLM&FID
• is related to: LU-5975 Consider removing quotacheck compatib...
• is related to: LU-5978 Remove migration of old admin quota f...
• is related to: LU-5980 Eliminate old quota formats
• is related to: LUDOC-89 Quota Enforcement Doc Changes

Comments

[Johann Lombardi (Inactive)]

quota: define quota records for glb/slv indexes
http://review.whamcloud.com/#change,3886

[Johann Lombardi (Inactive)]

ldlm: support for sending GL ASTs to multiple locks
http://review.whamcloud.com/3887

[Johann Lombardi (Inactive)]

quota: quota identifier unification
http://review.whamcloud.com/3888

[Niu Yawei (Inactive)]

quota: ldiskfs local enforcement
http://review.whamcloud.com/3915

[Niu Yawei (Inactive)]

quota: zfs local enforcement
http://review.whamcloud.com/3933

[Johann Lombardi (Inactive)]

protocol: add support for OBD_IDX_READ
http://review.whamcloud.com/#change,3942

[Johann Lombardi (Inactive)]

quota: helper routines to manage FID_SEQ_QUOTA_GLB
http://review.whamcloud.com/#change,3937

[Johann Lombardi (Inactive)]

mdt: add lvbo support to MDT namespace
http://review.whamcloud.com/#change,3951

[Niu Yawei (Inactive)]

protocol: new quota RPC format
http://review.whamcloud.com/3977

[Niu Yawei (Inactive)]

iam: store key & rec in LE order for quota IAM files
http://review.whamcloud.com/#change,3989

[Niu Yawei (Inactive)]

lfs: lfs quota commands changes
http://review.whamcloud.com/4022

[Niu Yawei (Inactive)]

test: t-f changes for the new quota
http://review.whamcloud.com/4031

[Niu Yawei (Inactive)]

test: new s-q
http://review.whamcloud.com/4040

[Johann Lombardi (Inactive)]

quota: add quota disk operations
http://review.whamcloud.com/4042

[Niu Yawei (Inactive)]

osp: osp-on-ost device
http://review.whamcloud.com/4068

[Niu Yawei (Inactive)]

quota: migrate old quota admin files
http://review.whamcloud.com/4077

[Johann Lombardi (Inactive)]

quota: remove quota code
http://review.whamcloud.com/#change,4092

[Niu Yawei (Inactive)]

quota: lquota entry
http://review.whamcloud.com/4101

[Johann Lombardi (Inactive)]

ldlm: add support for quota LVB
http://review.whamcloud.com/4107

[Niu Yawei (Inactive)]

quota: qsd request
http://review.whamcloud.com/4109

[Johann Lombardi (Inactive)]

quota: remove quota code from lvfs
http://review.whamcloud.com/#change,4110

[Niu Yawei (Inactive)]

quota: qsd entry operations
http://review.whamcloud.com/4119

[Johann Lombardi (Inactive)]

quota: setup/shutdown qmt device
http://review.whamcloud.com/#change,4126

[Niu Yawei (Inactive)]

quota: qsd lock
http://review.whamcloud.com/4129

[Johann Lombardi (Inactive)]

quota: add per-filesystem information
http://review.whamcloud.com/4156

[Johann Lombardi (Inactive)]

quota: add quotactl support on qmt
http://review.whamcloud.com/4160

[Johann Lombardi (Inactive)]

quota: add quota locks support on QMT
http://review.whamcloud.com/#change,4166

[Johann Lombardi (Inactive)]

quota: add acq/rel logic on QMT
http://review.whamcloud.com/#change,4178

[Johann Lombardi (Inactive)]

quota: add core QSD code
http://review.whamcloud.com/#change,4182

[Johann Lombardi (Inactive)]

test: re-enable all sanity-quota tests
http://review.whamcloud.com/#change,4200

[James A Simmons]

Patch http://review.whamcloud.com/#change,4156 introduced a kernel oops when mounting a MGT on a stand alone MGS system. The oops is:

2012-10-05 09:52:05 [ 355.128991] Lustre: DEBUG MARKER: mkdir -p /tmp/lustre/mgs
2012-10-05 09:52:05 [ 355.485941] Lustre: DEBUG MARKER: test -b /dev/sda1
2012-10-05 09:52:05 [ 355.830922] Lustre: DEBUG MARKER: mkdir -p /tmp/lustre/mgs; mount -t lustre ^I^I /dev/sda1 /tmp/lustre/mgs
2012-10-05 09:52:06 [ 356.265280] LDISKFS-fs (sda1): mounted filesystem with ordered data mode. quota=on. Opts:
2012-10-05 09:52:06 [ 356.381871] LustreError: 8414:0:(qsd_lib.c:355:qsd_init()) MGS: fail to extract filesystem name
2012-10-05 09:52:06 [ 356.449872] BUG: unable to handle kernel NULL pointer dereference at (null)
2012-10-05 09:52:06 [ 356.457029] IP: [<ffffffffa0cdf43a>] osd_oi_table_put+0x10a/0x1d0 [osd_ldiskfs]
2012-10-05 09:52:06 [ 356.464534] PGD 5c85dc067 PUD 5c96f8067 PMD 0
2012-10-05 09:52:06 [ 356.469223] Oops: 0000 1 SMP
2012-10-05 09:52:06 [ 356.472645] last sysfs file: /sys/module/ldiskfs/initstate
2012-10-05 09:52:06 [ 356.478213] CPU 2
2012-10-05 09:52:06 [ 356.480088] Modules linked in: fsfilt_ldiskfs(U) exportfs osd_ldiskfs(U) mdd(U) ldiskfs(U) mbcache jbd2 lustre(U) lov(U) osc(U) mdc(U) lquo
ta(U) fid(U) fld(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) sha512_generic sha256_generic libcfs(U) autofs4 ib_srp scsi_transport_srp scsi_tgt ipmi_devintf ipmi_si ipmi_msghandler nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ib_sa mlx4_ib ib_mad ib_core dm_mirror dm_region_hash dm_log dm_multipath dm_mod sg power_meter sr_mod cdrom sd_mod crc_t10dif ses enclosure microcode dcdbas serio_raw ata_generic pata_acpi ata_piix iTCO_wdt iTCO_vendor_support megaraid_sas i7core_edac edac_core ipv6 nfs lockd fscache nfs_acl auth_rpcgss sunrpc mlx4_en mlx4_core bnx2 [last unloaded: scsi_wait_scan]
2012-10-05 09:52:06 [ 356.565516]
2012-10-05 09:52:06 [ 356.567099] Pid: 8414, comm: mount.lustre Not tainted 2.6.32-279.9.1.el6.head.x86_64 #1 Dell Inc. PowerEdge R610/0F0XJ6
2012-10-05 09:52:06 [ 356.578177] RIP: 0010:[<ffffffffa0cdf43a>] [<ffffffffa0cdf43a>] osd_oi_table_put+0x10a/0x1d0 [osd_ldiskfs]
2012-10-05 09:52:06 [ 356.588109] RSP: 0018:ffff8805c463d948 EFLAGS: 00010246
2012-10-05 09:52:06 [ 356.593503] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
2012-10-05 09:52:06 [ 356.600717] RDX: 0000000000000040 RSI: 0000000000000000 RDI: ffff8805c4590000
2012-10-05 09:52:06 [ 356.607930] RBP: ffff8805c463d978 R08: 0000000000000000 R09: ffff8805f0091c80
2012-10-05 09:52:06 [ 356.615156] R10: ffff880330096440 R11: 0000000000000000 R12: 0000000000000000
2012-10-05 09:52:06 [ 356.622371] R13: 0000000000000040 R14: 0000000000000000 R15: 5a5a5a5a5a5a5a5a
2012-10-05 09:52:06 [ 356.629590] FS: 00007f371446e700(0000) GS:ffff88033ac20000(0000) knlGS:0000000000000000
2012-10-05 09:52:06 [ 356.637820] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
2012-10-05 09:52:06 [ 356.643649] CR2: 0000000000000000 CR3: 000000031701a000 CR4: 00000000000006e0
2012-10-05 09:52:06 [ 356.650864] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
2012-10-05 09:52:06 [ 356.658077] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
2012-10-05 09:52:06 [ 356.665297] Process mount.lustre (pid: 8414, threadinfo ffff8805c463c000, task ffff8805c466f500)
2012-10-05 09:52:06 [ 356.674219] Stack:
2012-10-05 09:52:06 [ 356.676322] ffff8805c463d968 ffff8805ca6da000 ffff8805c463da08 00000000ffffffea
2012-10-05 09:52:06 [ 356.683788] <d> ffff8805ca6da000 ffff8805ca6db4e8 ffff8805c463d998 ffffffffa0cdf51d
2012-10-05 09:52:06 [ 356.691845] <d> ffff8805c463da08 ffff8805ca6da000 ffff8805c463d9e8 ffffffffa0cd373e
2012-10-05 09:52:06 [ 356.701234] Call Trace:
2012-10-05 09:52:06 [ 356.703777] [<ffffffffa0cdf51d>] osd_oi_fini+0x1d/0x120 [osd_ldiskfs]
2012-10-05 09:52:06 [ 356.710395] [<ffffffffa0cd373e>] osd_device_alloc+0x43e/0x810 [osd_ldiskfs]
2012-10-05 09:52:06 [ 356.717559] [<ffffffffa055acf7>] obd_setup+0x1d7/0x2f0 [obdclass]
2012-10-05 09:52:06 [ 356.723846] [<ffffffffa055b018>] class_setup+0x208/0x890 [obdclass]
2012-10-05 09:52:06 [ 356.730304] [<ffffffffa05625cc>] class_process_config+0xc0c/0x1ca0 [obdclass]
2012-10-05 09:52:06 [ 356.737678] [<ffffffffa03a8be0>] ? cfs_alloc+0x30/0x60 [libcfs]
2012-10-05 09:52:06 [ 356.743791] [<ffffffffa0569b63>] ? lustre_cfg_new+0x353/0x710 [obdclass]
2012-10-05 09:52:06 [ 356.750685] [<ffffffffa056a4b9>] do_lcfg+0x149/0x480 [obdclass]
2012-10-05 09:52:06 [ 356.756796] [<ffffffffa056a884>] lustre_start_simple+0x94/0x200 [obdclass]
2012-10-05 09:52:06 [ 356.763862] [<ffffffffa057b0a3>] lustre_fill_super+0x1093/0x1af0 [obdclass]
2012-10-05 09:52:06 [ 356.770996] [<ffffffff8116a48c>] ? pcpu_alloc+0x3ac/0xa50
2012-10-05 09:52:06 [ 356.776570] [<ffffffff8127a22a>] ? strlcpy+0x4a/0x60
2012-10-05 09:52:06 [ 356.781715] [<ffffffff8117de33>] ? sget+0x3e3/0x480
2012-10-05 09:52:06 [ 356.786764] [<ffffffff8117cf10>] ? set_anon_super+0x0/0x100
2012-10-05 09:52:06 [ 356.792527] [<ffffffffa057a010>] ? lustre_fill_super+0x0/0x1af0 [obdclass]
2012-10-05 09:52:06 [ 356.799572] [<ffffffff8117e37f>] get_sb_nodev+0x5f/0xa0
2012-10-05 09:52:06 [ 356.804993] [<ffffffffa05656a5>] lustre_get_sb+0x25/0x30 [obdclass]
2012-10-05 09:52:06 [ 356.811430] [<ffffffff8117dfdb>] vfs_kern_mount+0x7b/0x1b0
2012-10-05 09:52:06 [ 356.817090] [<ffffffff8117e182>] do_kern_mount+0x52/0x130
2012-10-05 09:52:06 [ 356.822665] [<ffffffff8119c862>] do_mount+0x2d2/0x8d0
2012-10-05 09:52:06 [ 356.827890] [<ffffffff8119cef0>] sys_mount+0x90/0xe0
2012-10-05 09:52:06 [ 356.833030] [<ffffffff8100b0f2>] system_call_fastpath+0x16/0x1b
2012-10-05 09:52:06 [ 356.839120] Code: f8 b9 19 00 00 00 48 89 d7 f3 48 ab 48 8b 3b e8 7d 97 6c ff 48 c7 03 00 00 00 00 41 83 c4 01 45 39 e5 76 3d 49 63 c4 49 8d 1c c6 <48> 8b 03 48 85 c0 74 e8 48 8b 38 48 85 ff 74 35 48 3b 78 08 0f
2012-10-05 09:52:06 [ 356.861652] RIP [<ffffffffa0cdf43a>] osd_oi_table_put+0x10a/0x1d0 [osd_ldiskfs]
2012-10-05 09:52:06 [ 356.869243] RSP <ffff8805c463d948>
2012-10-05 09:52:06 [ 356.872820] CR2: 0000000000000000
2012-10-05 09:52:07 [ 0.000000] Initializing cgroup subsys cpuset
2012-10-05 09:52:07 [ 0.000000] Initializing cgroup subsys cpu

[Johann Lombardi (Inactive)]

Hi James,

Could you please give a try to this patch? http://review.whamcloud.com/4203

It should address the qsd setup issue.
That said, there is still a bug related to oi scrub since it should definitely properly clean up if the quota slave instance failed to be started. I will talk to Fanyong.

Thanks in advance.

[Johann Lombardi (Inactive)]

For the record, i have verified that the patch fixes the problem.
Thanks James!

[Niu Yawei (Inactive)]

utils: tunefs.lustre --quota
http://review.whamcloud.com/420

[Johann Lombardi (Inactive)]

That said, there is still a bug related to oi scrub since it should definitely properly clean up if the quota slave instance failed to be started. I will talk to Fanyong.

Patch for the oi scrub init issue (independent of quota) is available here:
b2_3: http://review.whamcloud.com/#change,4218
master: http://review.whamcloud.com/#change,4219

[Johann Lombardi (Inactive)]

For the record, LUDOC-89 is the bug to track changes to the manual.

[James A Simmons]

Appears master + http://review.whamcloud.com/#change,4218 fixed my problem.

[chas williams - CONTRACTOR]

Hello,

I am seeing a problem with ldiskfs and quotas in 2.4. http://review.whamcloud.com/4098 turned on quotas by default when mounting. However, when my ldiskfs goes to mount, it will fails since quotas and usage accounting are contraindicated. from the "first class accounting" ldiskfs patch:

+#ifdef CONFIG_QUOTA
+ /* Enable space tracking during mount, enforcement can be enabled/disable
+ * later with quota_on/off */
+ if (EXT4_HAS_RO_COMPAT_FEATURE(sb, EXT4_FEATURE_RO_COMPAT_QUOTA) &&
+ !(sb->s_flags & MS_RDONLY)) {
+ ret = ext4_acct_on(sb);
+ if (ret)

{ + ext4_msg(sb, KERN_ERR, "Can't enable usage tracking on " + "a filesystem with the QUOTA feature set"); + goto failed_mount4; + }

+ }

am i confused about what is going on here?

[Johann Lombardi (Inactive)]

This code snippet just enables space accounting automatically at mount time if the quota feature is enabled on the backend ldiskfs filesystem.

Could you please tell us more about the problem you see with ldiskfs and quota in 2.4?

[chas williams - CONTRACTOR]

i ported the ldiskfs patches to sles11sp2's kernel (basically 3.0). so it is possible to run quota and usage accounting at the same time? i guess i need to look harder at the ldiskfs patches.

[Johann Lombardi (Inactive)]

Quota enforcement actually requires accounting to be enabled. The only possible combinations are:

• no accounting & no limit enforcement
• space accounting enabled & no limit enforcement
• space accounting enabled & quota limit enforced

[chas williams - CONTRACTOR]

how clever. apparently i have to have the quota_v2 module loaded for this feature. however, ldiskfs doesnt depend on this module and doesnt seem to automagically load it either. this doesnt appear to be happening because QFMT_VFS_V1 isn't mapped to any particular module name in linux/quota.h:

#define INIT_QUOTA_MODULE_NAMES {\

{QFMT_VFS_OLD, "quota_v1"}

,\

{QFMT_VFS_V0, "quota_v2"}

,\
{0, NULL}}

[Wally Wang (Inactive)]

For the older OS(SLES11 SP1) that does not have QFMT_VFS_V1 support, is there a problem enabling it with QFMT_VFS_V0 instead of QFMT_VFS_V1?

[James A Simmons]

Looking at the new quota code it appears to be independent of what version of linux quota you are using. From Lustre 2.1.X I see the main dependency was from the lustre quota ioctl in the fsfilt layer which did vary between linux quota versions. Now that code is linux independent.

[Gerrit Updater]

Andreas Dilger ([e-mail]) uploaded a new patch: https://review.whamcloud.com/33694
Subject: LU-1842 ptlrpc: clean up ptlrpc_client setup
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: fa1de82769802f5b772ff625e738e15f23b15848