Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1357659 - Registry "blob unknown error for every build using S3 as storage
Summary: Registry "blob unknown error for every build using S3 as storage
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Image Registry
Version: 3.3.0
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: ---
: ---
Assignee: Alexey Gladkov
QA Contact: Wei Sun
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-07-18 19:19 UTC by Mike Fiedler
Modified: 2016-08-11 08:53 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-08-08 17:11:57 UTC


Attachments (Terms of Use)

Description Mike Fiedler 2016-07-18 19:19:33 UTC
I opened this separately from https://bugzilla.redhat.com/show_bug.cgi?id=1357656, feel free to collapse them into one issue if desired.

Description of problem:

Every application builds pops a "blob unknown" error in the registry log when S3 is configured as the backend.   See below for the full error and S3 configuration.


Version-Release number of selected component (if applicable):

3.3.0.6


How reproducible: Always


Steps to Reproduce:
1. Configure registry for S3 (see config below)
2. Build the cakephp-example
3. Examine the docker-registry pod's logs

Actual results:
2016-07-18T19:02:55.184925204Z time="2016-07-18T15:02:55.184599642-04:00" level=error msg="response completed with error" err.code="blob unknown" err.detail=sha256:d6a6ee7df917db09124747b615a2ef7fb4651a427a170df0d0b12386a362c356 err.message="blob unknown to registry" go.version=go1.6.2 http.request.host="172.27.158.145:5000" http.request.id=f22b5425-f5f5-441e-8723-a533e506fa65 http.request.method=HEAD http.request.remoteaddr="172.20.0.1:57328" http.request.uri="/v2/x20/cakephp-example/blobs/sha256:d6a6ee7df917db09124747b615a2ef7fb4651a427a170df0d0b12386a362c356" http.request.useragent="docker/1.10.3 go/go1.6.2 git-commit/969e2ce-unsupported kernel/3.10.0-394.el7.x86_64 os/linux arch/amd64" http.response.contenttype="application/json; charset=utf-8" http.response.duration=23.967409ms http.response.status=404 http.response.written=157 instance.id=72c1fc39-d0fe-449c-b3fc-02e88f0bd115 vars.digest="sha256:d6a6ee7df917db09124747b615a2ef7fb4651a427a170df0d0b12386a362c356" vars.name="x20/cakephp-example"

Expected results:

No errors.

Additional info:

version: 0.1
log:
  level: debug
http:
  addr: :5000
auth:
  openshift:
    realm: openshift
storage:
  cache:
    layerinfo: inmemory
  s3:
    accesskey: <key>
    secretkey: <secret key>
    region: us-west-2
    bucket: aoe-perf-test
    encrypt: false
    secure: false
    v4auth: true
    rootdirectory: /registry
middleware:
  registry:
    - name: openshift
  repository:
    - name: openshift
      options:
        acceptschema2: false
        pullthrough: true
        enforcequota: false
        projectcachettl: 1m
  storage:
    - name: openshift

Comment 1 Alexey Gladkov 2016-08-03 15:52:37 UTC
It happens because docker checks for the presence of the blob. For this, docker client makes a HEAD request (you can see "http.request.method=HEAD")

https://docs.docker.com/registry/spec/api/#existing-layers

Comment 2 Michal Minar 2016-08-05 13:19:34 UTC
Yes, the registry log makes a mountain out of a molehill. Unfortunately, in this case we cannot lower a priority of the message because we cannot anticipate what client if about to do (in this it just checks for a presence of blob in the destination repository before pushing it there).

Comment 3 Mike Fiedler 2016-08-08 16:51:29 UTC
I don't think this belongs in ON_QA if there is no code change.   You can answer it as NOTABUG if you feel filling the logs with these error messages is acceptable to end-users.

Comment 4 Alexey Gladkov 2016-08-08 17:10:01 UTC
I have no right to put such status. Available status: NEW, ASSIGNED, POST, MODIFIED, ON_QA, VERIFIED, RELEASE_PENDING, CLOSED. Sorry.


Note You need to log in before you can comment on or make changes to this bug.