Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1366268 - [extras-rhel-7.2.7] oci runtime error: rootfs ("...") does not exist
Summary: [extras-rhel-7.2.7] oci runtime error: rootfs ("...") does not exist
Keywords:
Status: VERIFIED
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: oci-register-machine
Version: 7.2
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Lokesh Mandvekar
QA Contact: Martin Jenner
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-08-11 12:39 UTC by Ed Santiago
Modified: 2018-01-30 16:39 UTC (History)
3 users (show)

Fixed In Version: docker-latest-1.12.0-5.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:


Attachments (Terms of Use)

Description Ed Santiago 2016-08-11 12:39:45 UTC
docker-latest-1.12.0-4.el7.x86_64 -- pretty much nothing works:

    # docker run fedora echo hi
    /usr/bin/docker-latest: Error response from daemon: oci runtime error: rootfs ("/var/lib/docker-latest/devicemapper/mnt/35b92e43e1d11408a6add14576512f506cb9806b8b498b26304762d79c8ae5e5/rootfs") does not exist.

I've tried restarting docker-containerd; tried rm -rf /var/lib/docker* and docker-latest-storage-setup --reset; setenforce 0.

Comment 2 Lokesh Mandvekar 2016-08-11 18:08:01 UTC
Hmm, weird, seems to work with the "docker" 1.12.0 at https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=507118 ..Let me check for packaging bugs, or name mismatches RE: appending "-latest"

Comment 3 Lokesh Mandvekar 2016-08-11 20:13:02 UTC
removing MountFlags=slave from the unitfile seems to make this work.

Comment 4 Ed Santiago 2016-08-11 22:14:47 UTC
> removing MountFlags=slave from the unitfile seems to make this work.

Confirmed.

Comment 6 Daniel Walsh 2016-08-17 20:20:21 UTC
This is not the correct solution.  It will cause other issues.

Comment 7 Daniel Walsh 2016-08-17 20:27:51 UTC
I think this is caused by oci-register-machine.  Basically if the oci-register-machine sends a message to systemd, it will tell it the container is running with a specific mount point.  If this directory is not mounted in systemd mount namespace, oci-register-machine will fail.

I think it is probably better that we disable oci-register-machine on rhel for now.

Comment 8 Daniel Walsh 2016-08-17 21:18:20 UTC
Ok lets remove oci-register machine from RHEL7 at this time.  Or at least disable it from  docker-latest requires.

oci-register-machine tells systemd that a container PID is about to run on some path.  /var/lib/docker-latest/devicemapper/mnt/67d94e161c93131596eb64238705733dbf31e2a24c82ee5dfd56b06c1cc9eeb5/rootfs

systemd checks the path and if it does not exist throws an error back at oci-register-machine. which kills the container.

Since docker is in a different namespace then systemd, the path only exists in dockers mount namespace.

Removing MountFlags fixed this issue, but opens us up to other issues.

I think we are better off holding off on oci-register-machine for RHEL until we can run docker in the host namespace

Comment 9 Colin Walters 2016-08-17 21:26:57 UTC
Yes please, let's back it out.  See also https://lists.projectatomic.io/projectatomic-archives/atomic-devel/2016-May/msg00050.html

Comment 10 Daniel Walsh 2016-08-18 12:29:50 UTC
Well this is actually far worse then just oci-register-machine.

We have a problem with the mount namespace implementation the kernel in
RHEL7.  There is a bug which is being actively worked on in the kernel
but will not be fixed by rhel7.2.7 Maybe not by rhel7.3.  Basically this
bug causes docker to fail to properly start and cleanup containers when
it runs in the host mnt namespace.  We have solved this problem in
docker-1.10 by running the docker daemon in its own mount namespace. 

In docker-latest-1.12 we have tried the same procedure, but it does not
work.  The problem is that the docker daemon is no longer the parent of
the new container, docker-containerd is.
If we configure docker daemon to run in its own namespace, it will setup
the containers rootfs and then send a message to docker-containerd to
run the container, but docker-containerd is in a different namespace so
it fails to see the rootfs and the container fails.  The only way so far
we have figured out to fix this is to run both daemons on the host
namespace (As we do in Fedora), but this triggers the bug in the RHEL
kernel.  Since this is the way we run these daemons in Fedora, we have
not seen this issue.  We have tried playing around with systemd
directives JoinNamespaceOf, but so far have not been able to get this to
work.  We have also noticed that docker daemon will start containerd as
a child if the containerd is not running.  This means that the
containerd ends up being in the namespace of the docker daemon, so that
stuff works.  But we would need to remove the docker-containerd unit
file. We have not experimented fully with this yet.

Comment 12 Daniel Walsh 2016-08-19 14:18:46 UTC
THis is really a docker issue in that docker can not be setting the mount points up in a different namespace then systemd is in, since it will fail to see the mountpoints.

Comment 13 Daniel Walsh 2016-08-26 19:20:23 UTC
oci-register-machine is now disabled in rhel7 release.  If you move docker daemon back into the host namespace, it should work properly.

Comment 14 Ed Santiago 2016-09-02 20:19:22 UTC
docker-latest-1.12.1-2.el7.x86_64 does not demonstrate this problem.


Note You need to log in before you can comment on or make changes to this bug.