Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1516569 - Unable to mount a secret inside another secrets mount point with kernel 3.10.0-693.5.2
Summary: Unable to mount a secret inside another secrets mount point with kernel 3.10....
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Pod
Version: 3.6.0
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ---
: 3.9.0
Assignee: Joel Smith
QA Contact: DeShuai Ma
URL:
Whiteboard:
: 1516887 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-11-22 23:24 UTC by Ryan Howe
Modified: 2018-03-28 14:13 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-03-28 14:13:03 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2018:0489 None None None 2018-03-28 14:13:48 UTC
Red Hat Knowledge Base (Article) 3251261 None None None 2017-12-08 10:52:44 UTC

Description Ryan Howe 2017-11-22 23:24:17 UTC
Description of problem:

I attach 2 secrets to a pod and try to mount one secret to /var/lib/base and the other secret to /var/lib/base/nested/. The nested secret does not show up as mounted to the pods. 


Version-Release number of selected component (if applicable):

> openshift version
openshift v3.6.173.0.49
kubernetes v1.6.1+5115d708d7
etcd 3.2.1

How reproducible:
100%


Steps to Reproduce:

# mkdir ./basedir; echo "basefile1" > ./basedir/basefile1 ; echo "basefile2" > ./basedir/basefile2
# mkdir ./nestdir;  echo "nestfile1" > ./nestdir/nestfile1 ; echo "nestfile2" > ./nestdir/nestfile2

# oc create secret generic basedir --from-file ./basedir
# oc create secret generic nestdir --from-file ./nestdir

# oc new-app --template=httpd-example
# oc volume dc/httpd-example  --add --mount-path=/var/lib/basedir -t secret --secret-name='basedir' --name basedir
# oc volume dc/httpd-example  --add --mount-path=/var/lib/basedir/nestdir -t secret --secret-name='nestdir' --name nestdir

-- Pod shows as running and healthy 

Actual results:
nested secret does not get mounted in pod. 


Expected results:
secret to get mounted

Additional info:


- Check Docker and Node for mounts 
# mount | grep -e nestdir -e basedir
tmpfs on /var/lib/origin/openshift.local.volumes/pods/61189515-cfd8-11e7-ad20-fa163ec5d77e/volumes/kubernetes.io~secret/basedir type tmpfs (rw,relatime,seclabel)
tmpfs on /var/lib/origin/openshift.local.volumes/pods/61189515-cfd8-11e7-ad20-fa163ec5d77e/volumes/kubernetes.io~secret/nestdir type tmpfs (rw,relatime,seclabel)

# docker inspect 83878ec9ac1a |  grep -e nestdir -e basedir

        "HostConfig": {
            "Binds": [
                "/var/lib/origin/openshift.local.volumes/pods/61189515-cfd8-11e7-ad20-fa163ec5d77e/volumes/kubernetes.io~secret/basedir:/var/lib/basedir:Z",
                "/var/lib/origin/openshift.local.volumes/pods/61189515-cfd8-11e7-ad20-fa163ec5d77e/volumes/kubernetes.io~secret/nestdir:/var/lib/basedir/nestdir:Z",
         ...
         ...
        "Mounts": [
            {
                "Source": "/var/lib/origin/openshift.local.volumes/pods/61189515-cfd8-11e7-ad20-fa163ec5d77e/volumes/kubernetes.io~secret/basedir",
                "Destination": "/var/lib/basedir",
                "Mode": "Z",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Source": "/var/lib/origin/openshift.local.volumes/pods/61189515-cfd8-11e7-ad20-fa163ec5d77e/volumes/kubernetes.io~secret/nestdir",
                "Destination": "/var/lib/basedir/nestdir",
                "Mode": "Z",
                "RW": true,
                "Propagation": "rprivate"
            },


- Check in the pod from the node for mounts
# nsenter -m -p  -t $(docker inspect --format "{{ .State.Pid }}" 83878ec9ac1a) << NSEOF
> mount | grep -e nestdir -e basedir
> ls /var/lib/basedir/ -R
> NSEOF

tmpfs on /var/lib/basedir type tmpfs (rw,relatime,seclabel)
/var/lib/basedir/:
basefile1  basefile2



- Logs from node and docker
Nov 22 18:05:38 test.node.com atomic-openshift-node[8740]: I1122 18:05:38.738478    8740 secret.go:186] Setting up volume basedir for pod 61189515-cfd8-11e7-ad20-fa163ec5d77e at /var/lib/origin/openshift.local.volumes/pods/61189515-cfd8-11e7-ad20-fa163ec5d77e/volumes/kubernetes.io~secret/basedir
Nov 22 18:05:38 test.node.com atomic-openshift-node[8740]: I1122 18:05:38.738872    8740 secret.go:186] Setting up volume nestdir for pod 61189515-cfd8-11e7-ad20-fa163ec5d77e at /var/lib/origin/openshift.local.volumes/pods/61189515-cfd8-11e7-ad20-fa163ec5d77e/volumes/kubernetes.io~secret/nestdir
Nov 22 18:05:38 test.node.com atomic-openshift-node[8740]: I1122 18:05:38.740926    8740 secret.go:217] Received secret nesttest/basedir containing (2) pieces of data, 20 total bytes
Nov 22 18:05:38 test.node.com atomic-openshift-node[8740]: I1122 18:05:38.741029    8740 atomic_writer.go:333] /var/lib/origin/openshift.local.volumes/pods/61189515-cfd8-11e7-ad20-fa163ec5d77e/volumes/kubernetes.io~secret/basedir: current paths:   [basefile1 basefile2]
Nov 22 18:05:38 test.node.com atomic-openshift-node[8740]: I1122 18:05:38.741048    8740 atomic_writer.go:345] /var/lib/origin/openshift.local.volumes/pods/61189515-cfd8-11e7-ad20-fa163ec5d77e/volumes/kubernetes.io~secret/basedir: new paths:       [basefile1 basefile2]
Nov 22 18:05:38 test.node.com atomic-openshift-node[8740]: I1122 18:05:38.741059    8740 atomic_writer.go:348] /var/lib/origin/openshift.local.volumes/pods/61189515-cfd8-11e7-ad20-fa163ec5d77e/volumes/kubernetes.io~secret/basedir: paths to remove: map[]
Nov 22 18:05:38 test.node.com atomic-openshift-node[8740]: I1122 18:05:38.741147    8740 atomic_writer.go:142] pod nesttest/httpd-example-3-b2g8r volume basedir: no update required for target directory /var/lib/origin/openshift.local.volumes/pods/61189515-cfd8-11e7-ad20-fa163ec5d77e/volumes/kubernetes.io~secret/basedir
Nov 22 18:05:38 test.node.com atomic-openshift-node[8740]: I1122 18:05:38.741286    8740 operation_generator.go:613] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/61189515-cfd8-11e7-ad20-fa163ec5d77e-basedir" (spec.Name: "basedir") pod "61189515-cfd8-11e7-ad20-fa163ec5d77e" (UID: "61189515-cfd8-11e7-ad20-fa163ec5d77e").
Nov 22 18:05:38 test.node.com atomic-openshift-node[8740]: I1122 18:05:38.741799    8740 secret.go:217] Received secret nesttest/nestdir containing (2) pieces of data, 20 total bytes
Nov 22 18:05:38 test.node.com atomic-openshift-node[8740]: I1122 18:05:38.741875    8740 atomic_writer.go:333] /var/lib/origin/openshift.local.volumes/pods/61189515-cfd8-11e7-ad20-fa163ec5d77e/volumes/kubernetes.io~secret/nestdir: current paths:   [nestfile1 nestfile2]
Nov 22 18:05:38 test.node.com atomic-openshift-node[8740]: I1122 18:05:38.741884    8740 atomic_writer.go:345] /var/lib/origin/openshift.local.volumes/pods/61189515-cfd8-11e7-ad20-fa163ec5d77e/volumes/kubernetes.io~secret/nestdir: new paths:       [nestfile1 nestfile2]
Nov 22 18:05:38 test.node.com atomic-openshift-node[8740]: I1122 18:05:38.741890    8740 atomic_writer.go:348] /var/lib/origin/openshift.local.volumes/pods/61189515-cfd8-11e7-ad20-fa163ec5d77e/volumes/kubernetes.io~secret/nestdir: paths to remove: map[]
Nov 22 18:05:38 test.node.com atomic-openshift-node[8740]: I1122 18:05:38.741925    8740 atomic_writer.go:142] pod nesttest/httpd-example-3-b2g8r volume nestdir: no update required for target directory /var/lib/origin/openshift.local.volumes/pods/61189515-cfd8-11e7-ad20-fa163ec5d77e/volumes/kubernetes.io~secret/nestdir
Nov 22 18:05:38 test.node.com atomic-openshift-node[8740]: I1122 18:05:38.742073    8740 operation_generator.go:613] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/61189515-cfd8-11e7-ad20-fa163ec5d77e-nestdir" (spec.Name: "nestdir") pod "61189515-cfd8-11e7-ad20-fa163ec5d77e" (UID: "61189515-cfd8-11e7-ad20-fa163ec5d77e").

Comment 1 Ryan Howe 2017-11-23 17:42:42 UTC
Issue is seen when booting wit kernel 3.10.0-693.5.2.el7.x86_64

After rolling kernel back to: 3.10.0-514.21.1.el7.x86_64 the issue is not seen and secrets are able to be mounted inside other secrets. 

Version of OpenShift did not change.

Comment 2 Paul Dwyer 2017-11-24 09:08:57 UTC
*** Bug 1516887 has been marked as a duplicate of this bug. ***

Comment 30 Joel Smith 2017-12-21 00:48:58 UTC
I have filed an issue upstream in the Kubernetes project:

https://github.com/kubernetes/kubernetes/issues/57421

and I have also posted a pull request of code changes to address the issue:

https://github.com/kubernetes/kubernetes/pull/57422

It's going to take some time for the community to review the pull request and make a decision about how to handle things. Assuming they like the patch as-is (or with minor modifications), we might see it merged early next year.

Red Hat still hasn't made any decisions about which old versions of OpenShift should receive the patch, but once we do, I'd then backport the patch to those versions.

Comment 31 Seth Jennings 2018-01-31 17:30:37 UTC
Merged in Origin:
https://github.com/openshift/origin/pull/18165

Comment 33 weiwei jiang 2018-02-07 09:10:21 UTC
Checked with 
# openshift version 
openshift v3.9.0-0.38.0
kubernetes v1.9.1+a0ce1bc657
etcd 3.2.8

and
# uname -r
3.10.0-843.el7.x86_64

Can not reproduce this issue, so verify this.

Comment 34 Joel Smith 2018-03-12 20:28:17 UTC
Thanks for reporting this bug. My investigation of this bug led to the discovery of CVE-2017-1002102 (#1551818).

Because the fixes for the security flaw also address this bug, you should be able to install an updated version to remedy this bug. I'm sure the errata tool will update with this bug with the information at some point, but in the mean time, you can get updated versions for each affected branch (the bug goes back to OpenShift 3.3):

3.3.1.46.11-1.git.4.e236015
3.4.1.44.38-1.git.4.bb8df08
3.5.5.31.48-1.git.4.ff6153e
3.6.173.0.96-1.git.4.e6301f8
3.7.23-1.git.5.83efd71

Comment 38 errata-xmlrpc 2018-03-28 14:13:03 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0489


Note You need to log in before you can comment on or make changes to this bug.