[LU-4291] Lustre 1.8.9 client on RHEL 6.4 does not play nice with SELINUX while mounting 2.4.1 filesystems

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 1.8.9
• Type: Bug
• Priority: Major
• Reporter: Jason Hill (Inactive)
• Assignee: Oleg Drokin
• Resolution: Fixed
• Severity: 3

Description

Lustre 1.8.9 client built locally from HPDD Git for kernel 2.6.32-358.23.2.el6.x86_64, using distribution OFED. Client successfully mounts all 1.8.9 filesystems, but when you ask it to mount a 2.4.1 filesystem it crashes with the following stack trace:

-----------[ cut here ]-----------
kernel BUG at security/selinux/ss/services.c:625!
invalid opcode: 0000 1 SMP
last sysfs file: /sys/devices/pci0000:00/0000:00:0a.0/0000:02:00.4/usb7/7-1/speed
CPU 11
Modules linked in: mgc(U) lustre(U) lov(U) mdc(U) lquota(U) osc(U) ptlrpc(U) obdclass(U) lvfs(U) ko2iblnd(U) lnet(U) libcfs(U) nfs lockd fscache auth_rpcgss nfs_acl mptctl mptbase autofs4 sunrpc nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 tcp_bic power_meter sg mlx4_ib ib_sa ib_mad ib_core mlx4_en mlx4_core hpilo hpwdt bnx2 myri10ge(U) dca microcode serio_raw k10temp amd64_edac_mod edac_core edac_mce_amd i2c_piix4 shpchp ext4 jbd2 mbcache sd_mod crc_t10dif hpsa pata_acpi ata_generic pata_atiixp ahci radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]

Pid: 7454, comm: lsof Not tainted 2.6.32-358.23.2.el6.x86_64 #1 HP ProLiant DL385 G7
RIP: 0010:[<ffffffff8123982b>] [<ffffffff8123982b>] context_struct_compute_av+0x40b/0x420
RSP: 0018:ffff88083854db18 EFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff88083854dca8 RCX: 0000000000000100
RDX: 0000000000000f3c RSI: 00000000ffffffff RDI: 0000000000000010
RBP: ffff88083854db98 R08: 00000000000135f0 R09: ffff88083854dca8
R10: 0000000000000010 R11: 0000000000000000 R12: 0000000000000007
R13: ffff880c3a556248 R14: 0000000000000796 R15: 000000000000079e
FS: 00007f9638d787a0(0000) GS:ffff88084e480000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000003a4dadaf50 CR3: 00000008391ac000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process lsof (pid: 7454, threadinfo ffff88083854c000, task ffff8808395e0040)
Stack:
ffff880839c1bc80 0007880800000007 ffff88041cc613c8 ffff880c3a556248
<d> ffff88083a6825f0 00000000b4bb9d11 0000000000000007 0000000000000000
<d> 000700073854dbd8 ffffffff81223621 ffff88083854dbe8 ffff88083854dca8
Call Trace:
[<ffffffff81223621>] ? avc_has_perm+0x71/0x90
[<ffffffff81239d05>] security_compute_av+0xf5/0x2c0
[<ffffffff8122328e>] avc_has_perm_noaudit+0x14e/0x470
[<ffffffff812235fb>] avc_has_perm+0x4b/0x90
[<ffffffff812253c4>] inode_has_perm+0x54/0xa0
[<ffffffff811a2100>] ? mntput_no_expire+0x30/0x110
[<ffffffff8122599b>] dentry_has_perm+0x5b/0x80
[<ffffffff81225a4c>] selinux_inode_getattr+0x2c/0x30
[<ffffffff8121d033>] security_inode_getattr+0x23/0x30
[<ffffffff81186d2f>] vfs_getattr+0x2f/0x80
[<ffffffff81186de0>] vfs_fstatat+0x60/0x80
[<ffffffff81186f2b>] vfs_stat+0x1b/0x20
[<ffffffff81186f54>] sys_newstat+0x24/0x50
[<ffffffff810dc937>] ? audit_syscall_entry+0x1d7/0x200
[<ffffffff810dc685>] ? __audit_syscall_exit+0x265/0x290
[<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
Code: ff ff ff e8 08 53 e3 ff 85 c0 0f 84 34 ff ff ff 0f b7 75 8e 48 c7 c7 28 22 7c 81 31 c0 e8 52 43 2d 00 e9 1d ff ff ff 0f 0b eb fe <0f> 0b 0f 1f 00 eb fb 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00
RIP [<ffffffff8123982b>] context_struct_compute_av+0x40b/0x420
RSP <ffff88083854db18>

To get this trace, set SELINUX=permissive in /etc/selinux/config.

We do have a kdump from this node in the permissive mode.

Setting SELINUX=disabled and the behavior goes away.

This is for an internet facing server (data transfer node), and our cyber policy strongly suggests running SELINUX on the web facing systems for the center.

This isn't critical as there's a workaround, but it's serious and we do need to get the reason that Lustre is tickling SELINUX figured out and patched so we can move forward with putting the new Lustre filesystems on the data transfer nodes.

Comments

[Oleg Drokin]

Historically 1.8.9 with selinux in any state but off was not supported, so can you please turn it off completely?
Even if it worked before for you, the code path was not verified, so now at the slight change broken things revealed itself.

[Jason Hill (Inactive)]

Oleg – thanks for the update. What is the stance for 2.4.X for our reference?

[Oleg Drokin]

There were some patches from Xyratex to allow operating a client and a server with SELinux enabled (no enforcement available, just to make it not crash), but to my knowledge we do not actively test this configuration.

[James A Simmons]

Lustre 2.4 is missing the patch from LU-2655. Lustre 1.8 would need to back port that and a bunch of patches from LU-589. Is SELinux a hard requirement?

[James Nunez (Inactive)]

Jason,

As Oleg pointed out, there are patches in b2_5 and beyond that allow clients and servers to operate with SELinux enabled.

Is there something else we need to do for this ticket or should we close it?

Thanks,
James

[Jason Hill (Inactive)]

James,

Go ahead and close this. My apologies for not responding sooner. 1 Month latencies are unacceptable.

Thanks!

–
-Jason

[James Nunez (Inactive)]

Thank you for the update, Jason.