Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1692513 - unable to mount disk at `/var/lib/containers` via `systemd` unit when `container-selinux` policy installed
Summary: unable to mount disk at `/var/lib/containers` via `systemd` unit when `contai...
Keywords:
Status: MODIFIED
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: container-selinux
Version: 8.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: 8.0
Assignee: Lokesh Mandvekar
QA Contact: atomic-bugs@redhat.com
URL:
Whiteboard:
Depends On:
Blocks: 1693806
TreeView+ depends on / blocked
 
Reported: 2019-03-25 18:47 UTC by Ben Breard
Modified: 2019-04-11 20:52 UTC (History)
12 users (show)

Fixed In Version: container-selinux-2.94
Doc Type: Bug Fix
Doc Text:
Systemd needs to create the container directories with the correct labels.
Clone Of:
: 1693806 1695669 (view as bug list)
Environment:
Last Closed:
Type: Bug
Target Upstream Version:


Attachments (Terms of Use)

Description Ben Breard 2019-03-25 18:47:14 UTC
Description of problem:

SELinux is blocking the "Where" line of mount units from creating mount points

Version-Release number of selected component (if applicable):

$ cat /etc/os-release 
NAME="Red Hat Enterprise Linux CoreOS"
VERSION="410.8.20190322.0"
ID="rhcos"
ID_LIKE="rhel fedora"
VERSION_ID="4.1"
PRETTY_NAME="Red Hat Enterprise Linux CoreOS 410.8.20190322.0"
ANSI_COLOR="0;31"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="OpenShift Container Platform"
REDHAT_BUGZILLA_PRODUCT_VERSION="4.1"
REDHAT_SUPPORT_PRODUCT="OpenShift Container Platform"
REDHAT_SUPPORT_PRODUCT_VERSION="4.1"
OSTREE_VERSION=410.8.20190322.0

(also present in Fedora CoreOS 3-22 build from F29)

How reproducible:

100%

Steps to Reproduce:
1. add a second disk and create a partition
2. Create the following mount unit that points to the correct disk:

$ systemctl cat var-lib-containers.mount 
# /etc/systemd/system/var-lib-containers.mount
[Mount]
What=/dev/sdb1
Where=/var/lib/containers
Type=xfs
Options=defaults
[Install]
WantedBy=local-fs.target


3. systemctl daemon-reload ; systemctl start var-lib-containers.mount


Actual results:

The mount fails due to the mount point not being present. (but systemd should be able to create it) 

Expected results:

/var/lib/containers should be created, but SELInux is blocking it. setenforce=0 gets around this problem.

Additional info:
Mar 25 18:40:13 host.local systemd[1]: var-lib-containers.mount: Failed to check directory /var/lib/containers: No such file or directory
Mar 25 18:40:13 host.local systemd[1]: Mounting /var/lib/containers...
Mar 25 18:40:13 host.local kernel: audit: type=1400 audit(1553539213.983:5): avc:  denied  { create } for  pid=1 comm="systemd" name="containers" scontext=system_u:system_r:init_t:s0 tcontext=system_u:object_r:container_var_lib_t:s0 tclass=dir permissive=0
Mar 25 18:40:13 host.local mount[1536]: mount: /var/lib/containers: mount point does not exist.
Mar 25 18:40:14 host.local systemd[1]: var-lib-containers.mount: Mount process exited, code=exited status=32
Mar 25 18:40:14 host.local systemd[1]: var-lib-containers.mount: Failed with result 'exit-code'.
Mar 25 18:40:14 host.local systemd[1]: Failed to mount /var/lib/containers.

Comment 2 Micah Abbott 2019-03-26 15:11:10 UTC
This doesn't happen for all mounts; for example, I created a similar mount unit for `/var/lib/foobar` and it succeeded:

```
$ rpm-ostree status
State: idle
Warning: failed to query journal: Cannot assign requested address (os error 99)
AutomaticUpdates: disabled
Deployments:
● pivot://docker-registry-default.cloud.registry.upshift.redhat.com/redhat-coreos/ootpa@sha256:84b54f4578bd04aa4e6fae2a7052a3b6df0bf5405dfbba1db44d59d282f3ee2f
              CustomOrigin: Provisioned from oscontainer
                   Version: 410.8.20190325.0 (2019-03-25T18:27:58Z)

$ sudo cat /etc/systemd/system/var-lib-foobar.mount
[Mount]
What=/dev/xvdb1
Where=/var/lib/foobar
Type=xfs
Options=defaults
[Install]
WantedBy=local-fs.target

$ sudo systemctl daemon-reload

$ sudo ls -la /var/lib/foobar
ls: cannot access '/var/lib/foobar': No such file or directory

$ sudo systemctl start var-lib-foobar.mount
$ sudo systemctl status var-lib-foobar.mount
● var-lib-foobar.mount - /var/lib/foobar
   Loaded: loaded (/etc/systemd/system/var-lib-foobar.mount; disabled; vendor preset: disabled)
   Active: active (mounted) since Tue 2019-03-26 14:41:42 UTC; 10s ago
    Where: /var/lib/foobar
     What: /dev/xvdb1
    Tasks: 0 (limit: 24828)
   Memory: 72.0K
   CGroup: /system.slice/var-lib-foobar.mount

Mar 26 14:41:42 ip-172-18-7-88 systemd[1]: Mounting /var/lib/foobar...
Mar 26 14:41:42 ip-172-18-7-88 systemd[1]: Mounted /var/lib/foobar.

$ sudo ls -la /var/lib/foobar
total 4
drwxr-xr-x.  2 root root    6 Mar 26 14:40 .
drwxr-xr-x. 27 root root 4096 Mar 26 14:41 ..
```

When I tested the failing scenario on a traditional RHEL 8 system, it succeeded for me.  I noticed that the version of `selinux-policy` on that host was older than what was on the RHCOS host:

traditional RHEL 8 - selinux-policy-3.14.1-51.el8.noarch
RHCOS - selinux-policy-3.14.1-61.el8.noarch

After upgrading the traditional RHEL 8 host, everything continued to work as expected.

There is something specific about the `/var/lib/containers` mount point on RHCOS, but it's not obvious to me what it is yet.

Comment 3 Micah Abbott 2019-03-26 16:23:13 UTC
I found that on the traditional RHEL 8 host, the `/var/lib/containers` directory is labeled differently than on the RHCOS host.

traditional RHEL 8 (after the mount unit):
```
$ ls -laZ /var/lib/containers
total 4
drwxr-xr-x.  2 root root system_u:object_r:unlabeled_t:s0    6 Mar 26 10:54 .
drwxr-xr-x. 35 root root system_u:object_r:var_lib_t:s0   4096 Mar 26 11:57 ..
```

RHCOS (defaults):

```
$ sudo ls -laZ /var/lib/containers
total 4
drwx------.  3 root root system_u:object_r:container_var_lib_t:s0   21 Mar 26 16:06 .
drwxr-xr-x. 27 root root system_u:object_r:var_lib_t:s0           4096 Mar 26 16:06 ..
drwx------.  8 root root system_u:object_r:container_var_lib_t:s0  136 Mar 26 16:06 storage
```

On the RHCOS host, `/var/lib/containers` is owned by the `cri-o` package.

Perhaps there is an SELinux policy difference since the traditional RHEL 8 host does not have `container-selinux` or `cri-o` installed.

Comment 4 Micah Abbott 2019-03-26 16:54:57 UTC
So the traditional RHEL 8 host is missing the `container_var_lib_t` (and associated SELinux types) by default:

```
$ sudo seinfo -x --type=container_var_lib_t

Types: 0

$ sudo sesearch --dontaudit -t container_var_lib_t
container_var_lib_t is not a valid type attribute

$ sudo semanage fcontext -l | grep container_var_lib_t

$ rpm -qa | grep selinux | sort
libselinux-2.8-6.el8.x86_64
libselinux-utils-2.8-6.el8.x86_64
python3-libselinux-2.8-6.el8.x86_64
rpm-plugin-selinux-4.14.2-9.el8.x86_64
selinux-policy-3.14.1-61.el8.noarch
selinux-policy-targeted-3.14.1-61.el8.noarch
```

When the `container-selinux` package is installed on the traditional RHEL 8 host, those types are now populated:

```
$ sudo seinfo -x --type container_var_lib_t                     
                                                                        
Types: 1                                                               

$ sudo sesearch --dontaudit -t container_var_lib_t
dontaudit abrt_helper_t non_security_file_type:blk_file { append getattr ioctl lock read write };
dontaudit abrt_helper_t non_security_file_type:chr_file { append getattr ioctl lock read write };
dontaudit abrt_helper_t non_security_file_type:fifo_file { append getattr ioctl lock read write };
dontaudit abrt_helper_t non_security_file_type:file { append getattr ioctl lock read write };
dontaudit abrt_helper_t non_security_file_type:lnk_file { append getattr ioctl lock read write };
dontaudit abrt_helper_t non_security_file_type:sock_file { append getattr ioctl lock read write };
dontaudit abrt_t file_type:lnk_file read;                                    
dontaudit abrt_t file_type:sock_file getattr;                     
dontaudit apmd_t file_type:fifo_file getattr;             
...

$  sudo semanage fcontext -l | grep container_var_lib_t
/exports(/.*)?                                     all files          system_u:object_r:container_var_lib_t:s0
/var/lib/containers(/.*)?                          all files          system_u:object_r:container_var_lib_t:s0
/var/lib/docker(/.*)?                              all files          system_u:object_r:container_var_lib_t:s0
/var/lib/docker-latest(/.*)?                       all files          system_u:object_r:container_var_lib_t:s0
/var/lib/lxc(/.*)?                                 all files          system_u:object_r:container_var_lib_t:s0
/var/lib/lxd(/.*)?                                 all files          system_u:object_r:container_var_lib_t:s0
/var/lib/ocid(/.*)?                                all files          system_u:object_r:container_var_lib_t:s0

$ rpm -qa | grep selinux | sort
container-selinux-2.75-1.git99e2cfd.module+el8+2769+577ad176.noarch
libselinux-2.8-6.el8.x86_64
libselinux-utils-2.8-6.el8.x86_64
python3-libselinux-2.8-6.el8.x86_64
rpm-plugin-selinux-4.14.2-9.el8.x86_64
selinux-policy-3.14.1-61.el8.noarch
selinux-policy-targeted-3.14.1-61.el8.noarch
```

With this policy installed, the traditional RHEL 8 host will suffer the same failure:

```
$ ls -laZ /var/lib/containers                  
ls: cannot access '/var/lib/containers': No such file or directory         

$ cat /etc/systemd/system/var-lib-containers.mount 
[Mount]
What=/dev/vdb1
Where=/var/lib/containers
Type=xfs
Options=defaults
[Install]
WantedBy=local-fs.target

$ sudo systemctl daemon-reload                                                    
$ sudo systemctl start var-lib-containers.mount            
Job for var-lib-containers.mount failed.                                            
See "systemctl status var-lib-containers.mount" and "journalctl -xe" for details.            

Mar 26 12:52:13 micah-rhel8-vm0326c systemd[1]: var-lib-containers.mount: Failed to check directory /var/lib/containers: No such file or directory
Mar 26 12:52:13 micah-rhel8-vm0326c systemd[1]: Mounting /var/lib/containers...
Mar 26 12:52:13 micah-rhel8-vm0326c mount[1149]: mount: /var/lib/containers: mount point does not exist.
Mar 26 12:52:13 micah-rhel8-vm0326c systemd[1]: var-lib-containers.mount: Mount process exited, code=exited status=32
Mar 26 12:52:13 micah-rhel8-vm0326c sudo[1145]: pam_unix(sudo:session): session closed for user root
Mar 26 12:52:13 micah-rhel8-vm0326c systemd[1]: var-lib-containers.mount: Failed with result 'exit-code'.
Mar 26 12:52:13 micah-rhel8-vm0326c systemd[1]: Failed to mount /var/lib/containers.
Mar 26 12:52:16 micah-rhel8-vm0326c dbus-daemon[643]: [system] Activating service name='org.fedoraproject.Setroubleshootd' requested by ':1.29' (uid=0 pid=611 comm="/usr/sbin/sedispatch " label="system_u:system_r:auditd_t:s0") (using servicehelper)
Mar 26 12:52:16 micah-rhel8-vm0326c dbus-daemon[643]: [system] Successfully activated service 'org.fedoraproject.Setroubleshootd'
Mar 26 12:52:17 micah-rhel8-vm0326c setroubleshoot[1152]: SELinux is preventing systemd from create access on the directory containers. For complete SELinux messages run: sealert -l 85ca15e5-4c63-44b0-989a-e7535189c316
Mar 26 12:52:17 micah-rhel8-vm0326c platform-python[1152]: SELinux is preventing systemd from create access on the directory containers.

                                                           *****  Plugin catchall (100. confidence) suggests   **************************

                                                           If you believe that systemd should be allowed create access on the containers directory by default.
                                                           Then you should report this as a bug.
                                                           You can generate a local policy module to allow this access.
                                                           Do
                                                           allow this access for now by executing:
                                                           # ausearch -c 'systemd' --raw | audit2allow -M my-systemd
                                                           # semodule -X 300 -i my-systemd.pp

```


So I think this is ultimately an `selinux-policy` or `container-selinux` issue.  @bbreard WDYT?

Comment 5 Ben Breard 2019-03-26 17:03:14 UTC
Brilliant analysis! I guess I'm leaning toward container-selinux being where this should land. 

This is blocking our means to recreate the docker-storage-setup utility via ignition.

Comment 6 Micah Abbott 2019-03-26 18:20:54 UTC
FWIW, mounting manually is still possible:

```
$ sudo ls -laZ /var/lib/containers
ls: cannot access '/var/lib/containers': No such file or directory
$ sudo mkdir -p /var/lib/containers
$ sudo ls -laZ /var/lib/containers
total 4
drwxr-xr-x.  2 root root unconfined_u:object_r:container_var_lib_t:s0    6 Mar 26 14:18 .
drwxr-xr-x. 35 root root system_u:object_r:var_lib_t:s0               4096 Mar 26 14:18 ..
$ sudo mount /dev/vdb1 /var/lib/containers
$ findmnt /var/lib/containers
TARGET              SOURCE    FSTYPE OPTIONS
/var/lib/containers /dev/vdb1 xfs    rw,relatime,seclabel,attr2,inode64,noquota
```

Comment 7 Ben Breard 2019-03-26 18:23:58 UTC
That is correct; I got the same result. I'm not sure how to create that directory first via ignition. I'm sure it's possible, but not obvious or user friendly.

Comment 9 Daniel Walsh 2019-03-28 11:50:05 UTC
We can fix this in container-selinux, but it is really a fix in selinux-policy.

Fixed in container-selinux-2.94


Note You need to log in before you can comment on or make changes to this bug.