Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.

Bug 1509895

Summary: [DOCKER] Error copying file from docker container
Product: OpenShift Container Platform Reporter: Gan Huang <ghuang>
Component: ContainersAssignee: Tom Sweeney <tsweeney>
Status: CLOSED DUPLICATE QA Contact: DeShuai Ma <dma>
Severity: high Docs Contact:
Priority: high    
Version: 3.7.0CC: amurdaca, aos-bugs, ghuang, jokerman, mmccomas, pasik, vgoyal
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-11-07 04:47:27 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Description Gan Huang 2017-11-06 09:49:45 UTC
Description of problem:
Installation failed at:

Failure summary:


  1. Hosts:    qe-ghuang-auto-14065-gytx-mez-3.1106-oih.qe.rhcloud.com
     Play:     Create OpenShift certificates for master hosts
     Task:     Copy client binaries/symlinks out of CLI image for use on the host
     Message:  Error copying file from docker container: stdout=, stderr=Error response from daemon: error creating overlay mount to /var/lib/docker/overlay2/82470f5844ff880921511496f501a0a6cec885269f2cd5939ae8f4bf5be3ac88/merged: mountfrom error on re-exec cmd: fork/exec /proc/self/exe: cannot allocate memory
               

  2. Hosts:    qe-ghuang-auto-14065-gytx-mez-1.1106-oih.qe.rhcloud.com, qe-ghuang-auto-14065-gytx-mez-2.1106-oih.qe.rhcloud.com
     Play:     Configure masters
     Task:     restart master controllers
     Message:  Unable to restart service atomic-openshift-master-controllers: Job for atomic-openshift-master-controllers.service failed because the control process exited with error code. See "systemctl status atomic-openshift-master-controllers.service" and "journalctl -xe" for details.
               

  3. Hosts:    qe-ghuang-auto-14065-gytx-nrrz-1.1106-oih.qe.rhcloud.com, qe-ghuang-auto-14065-gytx-nrrz-2.1106-oih.qe.rhcloud.com, qe-ghuang-auto-14065-gytx-nzp-1.1106-oih.qe.rhcloud.com
     Play:     Configure nodes
     Task:     Wait for master API to become available before proceeding
     Message:  non-zero return code

Version-Release number of the following components:

openshift-ansible-3.7.0-0.189.0.git.0.d497c5e.el7.noarch.rpm
docker-1.12.6-61.git85d7426.el7.x86_64

# docker info
Containers: 1
 Running: 1
 Paused: 0
 Stopped: 0
Images: 2
Server Version: 1.12.6
Storage Driver: overlay2
 Backing Filesystem: xfs
Logging Driver: json-file
Cgroup Driver: systemd
Plugins:
 Volume: local
 Network: null overlay host bridge
 Authorization: rhel-push-plugin
Swarm: inactive
Runtimes: runc docker-runc
Default Runtime: docker-runc
Security Options: seccomp selinux
Kernel Version: 3.10.0-693.5.2.el7.x86_64
Operating System: Red Hat Enterprise Linux Server 7.4 (Maipo)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 3
CPUs: 1
Total Memory: 3.456 GiB
Name: qe-ghuang-auto-14065-gytx-mez-3


How reproducible:
sometimes

Steps to Reproduce:
1. Trigger containerized installation on RHEL


Actual results:
Please include the entire output from the last TASK line through the end of output if an error is generated

Expected results:

Additional info:
Logs attached

Similar issue: https://github.com/moby/moby/issues/8539

Comment 4 Gan Huang 2017-11-06 09:54:52 UTC
# free
              total        used        free      shared  buff/cache   available
Mem:        3623892      379976     2381476        8608      862440     2916832
Swap:             0           0           0

Comment 5 Scott Dodson 2017-11-06 13:45:04 UTC
I believe this is a docker problem. Here's the code that's executed, basically we start openshift3/ose container and `docker cp` some files out to /tmp.

https://github.com/openshift/openshift-ansible/blob/master/roles/openshift_cli/library/openshift_container_binary_sync.py#L71-L98


Possibly a problem related to use of overlayfs? This code hasn't changed in quite some time but the use of overlayfs is relatively new.