Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1366031 - Random unauthorized error when concurrent builds are going using S3 registry
Summary: Random unauthorized error when concurrent builds are going using S3 registry
Keywords:
Status: CLOSED WORKSFORME
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Build
Version: 3.3.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: ---
Assignee: Cesar Wong
QA Contact: Wang Haoran
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-08-10 20:01 UTC by Vikas Laad
Modified: 2016-08-17 20:16 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-08-17 20:16:46 UTC


Attachments (Terms of Use)

Description Vikas Laad 2016-08-10 20:01:59 UTC
Description of problem:
I am testing Openshift 3.3 concurrent builds performance, when running 20 or 40 builds concurrently I see following errors randomly on few builds.

root@ip-172-31-35-249: ~/svt/openshift_performance/ose3_perf/scripts # oc logs cakephp-example-1-build -n p25
Downloading "https://github.com/openshift/cakephp-ex.git" ...

---> Installing application source...


Pushing image 172.26.140.59:5000/p25/cakephp-example:latest ...
error: build error: Failed to push image: unauthorized: authentication required

Version-Release number of selected component (if applicable):
openshift v3.3.0.17
kubernetes v1.3.0+507d3a7
etcd 2.3.0+git


How reproducible:
Can reproduce with concurrent builds.

Steps to Reproduce:
1. start more then 20 builds concurrently
2. few builds fail, check the logs

Actual results:
error: build error: Failed to push image: unauthorized: authentication required


Expected results:


Additional info:
Errors seen in the registry pod
===============================
time="2016-08-10T15:46:05.015606219-04:00" level=error msg="OpenShift access denied: User \"system:anonymous\" cannot get imagestreams/layers in project \"p25\"" go.version=go1.6.2 http.request.host="172.26.140.59:5000" http.request.id=46da4d8c-5bbc-4f1f-8388-79dfc3132bbf http.request.method=POST http.request.remoteaddr="172.20.3.1:40714" http.request.uri="/v2/p25/cakephp-example/blobs/uploads/" http.request.useragent="docker/1.10.3 go/go1.6.2 git-commit/05cbc5b-unsupported kernel/3.10.0-394.el7.x86_64 os/linux arch/amd64" instance.id=35cb1ef9-217d-4be0-a8b7-97fd8dbbc8a1 vars.name="p25/cakephp-example"
time="2016-08-10T15:46:05.018648846-04:00" level=debug msg="Origin auth: checking for access to repository:p25/cakephp-example:push" go.version=go1.6.2 http.request.host="172.26.140.59:5000" http.request.id=46da4d8c-5bbc-4f1f-8388-79dfc3132bbf http.request.method=POST http.request.remoteaddr="172.20.3.1:40714" http.request.uri="/v2/p25/cakephp-example/blobs/uploads/" http.request.useragent="docker/1.10.3 go/go1.6.2 git-commit/05cbc5b-unsupported kernel/3.10.0-394.el7.x86_64 os/linux arch/amd64" instance.id=35cb1ef9-217d-4be0-a8b7-97fd8dbbc8a1 vars.name="p25/cakephp-example"
time="2016-08-10T15:46:05.018713791-04:00" level=error msg="OpenShift access denied: User \"system:anonymous\" cannot get imagestreams/layers in project \"p25\"" go.version=go1.6.2 http.request.host="172.26.140.59:5000" http.request.id=631093fa-a1ee-45af-85e7-a27047573cf0 http.request.method=POST http.request.remoteaddr="172.20.3.1:40712" http.request.uri="/v2/p25/cakephp-example/blobs/uploads/" http.request.useragent="docker/1.10.3 go/go1.6.2 git-commit/05cbc5b-unsupported kernel/3.10.0-394.el7.x86_64 os/linux arch/amd64" instance.id=35cb1ef9-217d-4be0-a8b7-97fd8dbbc8a1 vars.name="p25/cakephp-example"

S3 Registry configuration 
=========================
version: 0.1
log:
  level: debug
http:
  addr: :5000
auth:
  openshift:
    realm: openshift
storage:
  cache:
    layerinfo: inmemory
  s3:
    accesskey: <key_id>
    secretkey: <secret>
    region: us-west-2
    bucket: aoe-perf-test
    encrypt: false
    secure: false
    v4auth: true
    rootdirectory: /registry
middleware:
  registry:
    - name: openshift
  repository:
    - name: openshift
      options:
        acceptschema2: false
        pullthrough: true
        enforcequota: false
        projectcachettl: 1m
  storage:
    - name: openshift

Comment 1 Vikas Laad 2016-08-10 20:36:29 UTC
If I start another build just after failed build it completes fine.

Comment 2 Michal Fojtik 2016-08-12 09:57:56 UTC
This seems to be that the builder failed to attach the correct credentials when pushing the image to the registry (which might be a race in the build controller where we fail to get the secret/token for the push).

Comment 3 Michal Fojtik 2016-08-12 10:00:46 UTC
Vikas, can you please also attach the full build log (also please set the BUILD_LOGLEVEL env var in the build config to 5, so we can see more details).

Comment 4 Vikas Laad 2016-08-17 20:15:50 UTC
I am not able to reproduce this issue with the latest build, I only see following which already has a bug open

error: build error: timeout after waiting 20s for Docker to create container


Note You need to log in before you can comment on or make changes to this bug.