Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1511870 - Failed to push image in proxy environment
Summary: Failed to push image in proxy environment
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 3.7.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 3.7.z
Assignee: Fabian von Feilitzsch
QA Contact: Gan Huang
Johnny Liu
URL:
Whiteboard:
: 1544073 1544682 1575050 (view as bug list)
Depends On: 1527210 1541625
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-11-10 10:35 UTC by Gan Huang
Modified: 2018-05-08 14:24 UTC (History)
14 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Kubernetes service IP was not added to no_proxy list for the docker-registry Consequence: Internal registry requests would be forced to use the proxy, preventing logins and pushes to the internal registry. Fix: Added the kubernetes service IP to the no_proxy list Result: The internal registry requests are no longer proxied, and logins and pushes to the internal registry succeed as expected.
Clone Of:
Environment:
Last Closed: 2018-05-08 14:24:44 UTC


Attachments (Terms of Use)

Comment 2 Johnny Liu 2017-11-10 11:34:47 UTC
This is blocking testing behind proxy

Comment 3 Gan Huang 2017-11-13 09:25:49 UTC
The registry version seems to be v2.6.2

# oc logs docker-registry-1-fc7f4 |grep distribution_version
time="2017-11-13T08:15:29.183080643Z" level=info msg="start registry" distribution_version="v2.6.2+unknown" kubernetes_version=v1.7.6+a08f5eeb62 openshift_version=v3.7.7

Comment 4 Scott Dodson 2017-11-13 16:25:49 UTC
I think this is a regression and therefore a blocker. I think the fix is to ensure that the kube service ip is added to the no_proxy list. There's another bug on this that's got more information, let me find that.

Comment 5 Scott Dodson 2017-11-13 16:52:07 UTC
I think this is a dupe of https://bugzilla.redhat.com/show_bug.cgi?id=1504464 and I think the correct way to fix this is to set KUBERNETES_MASTER='kubernetes.svc.default' on the registry whenever we configure a proxy so that it connects to the api server via dns name rather than ip address.

Since we actually regressed this in 3.6.z this is by definition not a 3.7 blocker. We'll fix this ASAP however, ideally on 3.7 GA day.

Comment 6 Gan Huang 2017-11-14 05:55:16 UTC
(In reply to Scott Dodson from comment #5)
> I think this is a dupe of
> https://bugzilla.redhat.com/show_bug.cgi?id=1504464 and I think the correct
> way to fix this is to set KUBERNETES_MASTER='kubernetes.svc.default' on the

I'm thinking that the correct route should be "kubernetes.default.svc" :)

I tried with setting KUBERNETES_MASTER='kubernetes.default.svc', things still don't work.

After appending `172.30.0.1` to NO_PROXY of docker-registry dc, build succeeded.

# oc env dc/docker-registry NO_PROXY=<--snip-->,172.30.0.1

Comment 10 Johnny Liu 2018-02-07 02:29:26 UTC
I think the fix should be backport to 3.7 branch, and fix this 3.7 bug, this is really a very basic functionality (sti build behind proxy)

Comment 11 Johnny Liu 2018-02-07 02:31:11 UTC
Before backport the PR to 3.7, pls fix Bug 1541625 together, Bug 1541625 is introduced by this PR.

Comment 12 Scott Dodson 2018-02-07 02:49:13 UTC
ACK, lets treat this as the bug to track the backport from master.

Comment 13 Scott Dodson 2018-02-09 15:07:03 UTC
Need to backport these two to release-3.7 for this bug

https://github.com/openshift/openshift-ansible/pull/7055
https://github.com/openshift/openshift-ansible/pull/6215

Comment 14 Scott Dodson 2018-02-12 18:49:24 UTC
*** Bug 1544073 has been marked as a duplicate of this bug. ***

Comment 15 Ben Parees 2018-02-13 14:55:47 UTC
*** Bug 1544682 has been marked as a duplicate of this bug. ***

Comment 16 Fabian von Feilitzsch 2018-02-13 21:22:24 UTC
backports: https://github.com/openshift/openshift-ansible/pull/7137

Comment 17 Scott Dodson 2018-02-22 21:51:25 UTC
In openshift-ansible-3.7.32-1

Comment 18 Johnny Liu 2018-02-23 06:02:30 UTC
Verified this bug with openshift-ansible-3.9.0-0.48.0.git.0.2fb33db.el7.noarch, and PASS.

# oc env dc docker-registry --list |grep -i proxy
NO_PROXY=.centralci.eng.rdu2.redhat.com,.cluster.local,.svc,169.254.169.254,172.16.120.106,172.16.120.64,172.31.0.1
HTTP_PROXY=http://file.rdu.redhat.com:3128
HTTPS_PROXY=http://file.rdu.redhat.com:3128

kubernetes svc IP (172.31.0.1) is added into NO_PROXY list, sti build succeed.

# oc get po -n install-test
NAME                             READY     STATUS      RESTARTS   AGE
mongodb-1-4w6ln                  1/1       Running     0          2h
nodejs-mongodb-example-1-build   0/1       Completed   0          2h
nodejs-mongodb-example-1-r5g6c   1/1       Running     0          2h

Comment 19 Scott Dodson 2018-05-08 14:24:44 UTC
Fixed in 	openshift-ansible-3.7.42-1 and later

Comment 20 Scott Dodson 2018-05-08 14:24:57 UTC
*** Bug 1575050 has been marked as a duplicate of this bug. ***


Note You need to log in before you can comment on or make changes to this bug.