Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1511374 - container-engine can't pull images against authenticated registry
Summary: container-engine can't pull images against authenticated registry
Keywords:
Status: VERIFIED
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 3.7.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 3.7.z
Assignee: Michael Gugino
QA Contact: Johnny Liu
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-11-09 09:18 UTC by Gan Huang
Modified: 2018-11-21 13:24 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Docker as system container caused docker login to fail on host due to docker not installed. Consequence: Installer unable to pull openshift images from authenticated registries during containerized install due to lack of credentials. Fix: Create credentials manually on host to enable installer to pull needed images. Result: Installer can pull images from authenticated registry.
Clone Of:
Environment:
Last Closed:


Attachments (Terms of Use)

Description Gan Huang 2017-11-09 09:18:11 UTC
Description of problem:

Version-Release number of the following components:
openshift-ansible-3.7.4-1.git.0.254e849.el7.noarch.rpm

How reproducible:
always

Steps to Reproduce:
1. Triggering OCP with docker system container enabled, and specifying `oreg_url` to use an authenticated registry



Actual results:
Installation succeed, but all pods are pending status.

Expected results:

Additional info:

atomic-openshift-node indicated that it failed to pull image from the authenticated registry as lacking of the credentials, but docker pull from the host succeeded.
  
# journalctl -u atomic-openshift-node:

<--snip-->

Nov 09 02:46:41 host-172-16-120-121 atomic-openshift-node[23762]: I1109 02:46:41.713855   23773 helpers.go:412] Pulling image "registry.reg-aws.openshift.com:443/openshift3/ose-pod:v3.7.4" without credentials
Nov 09 02:46:42 host-172-16-120-121 atomic-openshift-node[23762]: I1109 02:46:42.443525   23773 kube_docker_client.go:331] Stop pulling image "registry.reg-aws.openshift.com:443/openshift3/ose-pod:v3.7.4": "Trying to pull repository registry.reg-aws.openshift.com:443/openshift3/ose-pod ... "
Nov 09 02:46:42 host-172-16-120-121 atomic-openshift-node[23762]: E1109 02:46:42.444197   23773 remote_runtime.go:91] RunPodSandbox from runtime service failed: rpc error: code = 2 desc = failed pulling image "registry.reg-aws.openshift.com:443/openshift3/ose-pod:v3.7.4": unauthorized: authentication required
Nov 09 02:46:42 host-172-16-120-121 atomic-openshift-node[23762]: E1109 02:46:42.444366   23773 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "router-1-deploy_default(20389bc5-c522-11e7-a1fa-fa163e9c054e)" failed: rpc error: code = 2 desc = failed pulling image "registry.reg-aws.openshift.com:443/openshift3/ose-pod:v3.7.4": unauthorized: authentication required
Nov 09 02:46:42 host-172-16-120-121 atomic-openshift-node[23762]: E1109 02:46:42.444391   23773 kuberuntime_manager.go:622] createPodSandbox for pod "router-1-deploy_default(20389bc5-c522-11e7-a1fa-fa163e9c054e)" failed: rpc error: code = 2 desc = failed pulling image "registry.reg-aws.openshift.com:443/openshift3/ose-pod:v3.7.4": unauthorized: authentication required
Nov 09 02:46:42 host-172-16-120-121 atomic-openshift-node[23762]: E1109 02:46:42.444615   23773 pod_workers.go:186] Error syncing pod 20389bc5-c522-11e7-a1fa-fa163e9c054e ("router-1-deploy_default(20389bc5-c522-11e7-a1fa-fa163e9c054e)"), skipping: failed to "CreatePodSandbox" for "router-1-deploy_default(20389bc5-c522-11e7-a1fa-fa163e9c054e)" with CreatePodSandboxError: "CreatePodSandbox for pod \"router-1-deploy_default(20389bc5-c522-11e7-a1fa-fa163e9c054e)\" failed: rpc error: code = 2 desc = failed pulling image \"registry.reg-aws.openshift.com:443/openshift3/ose-pod:v3.7.4\": unauthorized: authentication required"


<--snip-->

Comment 2 Michael Gugino 2017-11-10 17:05:38 UTC
Since docker is not running with system containers, we need a way to create docker auth credentials file for these hosts as 'docker login' will not work.

I have looked into this, and it appears that docker login creates a file at ~/.docker/config.json.

The credentials are stored in this file in the following json format:

{
	"auths": {
		"registry.example.com": {
			"auth": "<base64 encoded username:password>"
		}
	}
}

It should be possible to generate/update this file dynamically with an ansible module or python script.

That will take some time to prepare, but I should be able to complete it relatively soon.

We can also validate the credentials for the user using curl:
"curl --user x:2 https://registry.example.com/v2/"

Note, the trailing slash on /v2/ is required or the request will return a 404.

Successful user/password combo should return a 200 OK response, unsuccessful username/password combo will return a 401.

Comment 3 Michael Gugino 2017-11-15 14:51:02 UTC
PR Merged: https://github.com/openshift/openshift-ansible/pull/6094 in master.

Comment 4 Michael Gugino 2017-11-15 16:52:17 UTC
PR Created for 3.7 branch: https://github.com/openshift/openshift-ansible/pull/6120

Comment 6 Gan Huang 2017-12-07 09:38:44 UTC
Tested in openshift-ansible-3.7.13-1.git.0.a8e2fcb.el7.noarch.rpm

Installation failed:

TASK [openshift_master : Create credentials for registry auth (alternative)] ***
Thursday 07 December 2017  09:26:55 +0000 (0:00:00.035)       0:06:09.304 ***** 
fatal: [host-8-241-76.host.centralci.eng.rdu2.redhat.com]: FAILED! => {"failed": true, "msg": "The conditional check '(not docker_cli_auth_credentials_stat.stat.exists or oreg_auth_credentials_replace) | bool' failed. The error was: error while evaluating conditional ((not docker_cli_auth_credentials_stat.stat.exists or oreg_auth_credentials_replace) | bool): 'dict object' has no attribute 'stat'\n\nThe error appears to have been in '/home/slave5/workspace/Launch-Environment-Flexy/private-openshift-ansible/roles/openshift_master/tasks/registry_auth.yml': line 26, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n# check the registry to determine whether or not the credentials will work.\n- name: Create credentials for registry auth (alternative)\n  ^ here\n"}

Comment 7 Michael Gugino 2017-12-07 18:21:16 UTC
PR Created in Master: https://github.com/openshift/openshift-ansible/pull/6387

Comment 8 Michael Gugino 2017-12-07 20:02:19 UTC
PR merged.

Backport to 3.7 merged: https://github.com/openshift/openshift-ansible/pull/6391

Comment 11 Gan Huang 2017-12-13 01:59:30 UTC
Verified in openshift-ansible-3.7.15-1.git.0.b20e6be.el7.noarch.rpm


Note You need to log in before you can comment on or make changes to this bug.