Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1357656 - Registry "manifest invalid" error for every build using S3 storage
Summary: Registry "manifest invalid" error for every build using S3 storage
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Image Registry
Version: 3.3.0
Hardware: x86_64
OS: All
Target Milestone: ---
: ---
Assignee: Alexey Gladkov
QA Contact: Wei Sun
Depends On:
TreeView+ depends on / blocked
Reported: 2016-07-18 19:14 UTC by Mike Fiedler
Modified: 2016-08-11 08:53 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2016-08-10 10:04:29 UTC
Target Upstream Version:

Attachments (Terms of Use)

Description Mike Fiedler 2016-07-18 19:14:02 UTC
Description of problem:

Every application builds pops a "manifest invalid" error in the registry log when S3 is configured as the backend.   See below for the full error and S3 configuration.

Version-Release number of selected component (if applicable):

How reproducible: Always

Steps to Reproduce:
1. Configure registry for S3 (see config below)
2. Build the cakephp-example
3. Examine the docker-registry pod's logs

Actual results:

2016-07-18T19:02:35.034236438Z time="2016-07-18T15:02:35.03394661-04:00" level=error msg="response completed with error" err.code=unknown err.detail="manifest invalid: manifest invalid" err.message="unknown error" go.version=go1.6.2 http.request.contenttype="application/vnd.docker.distribution.manifest.v2+json""" http.request.method=PUT http.request.remoteaddr="" http.request.uri="/v2/x19/cakephp-example/manifests/latest" http.request.useragent="docker/1.10.3 go/go1.6.2 git-commit/969e2ce-unsupported kernel/3.10.0-394.el7.x86_64 os/linux arch/amd64" http.response.contenttype="application/json; charset=utf-8" http.response.duration=7.306784ms http.response.status=500 http.response.written=136"x19/cakephp-example" vars.reference=latest

Expected results:

No errors.

Additional info:

version: 0.1
  level: debug
  addr: :5000
    realm: openshift
    layerinfo: inmemory
    accesskey: <key>
    secretkey: <secret key>
    region: us-west-2
    bucket: aoe-perf-test
    encrypt: false
    secure: false
    v4auth: true
    rootdirectory: /registry
    - name: openshift
    - name: openshift
        acceptschema2: false
        pullthrough: true
        enforcequota: false
        projectcachettl: 1m
    - name: openshift

Comment 1 Alexey Gladkov 2016-08-03 16:08:09 UTC
It happens because docker 1.10 and newer uses v2 schema by default. Docker client tries to send the manifest in v2 schema and if he fails, sends it in v1 schema [1].

In our docker-registry v2 schema is disabled by default for compatibility reasons [2]. You see these errors because the docker-registry refuses to accept the v2 schema (http.request.contenttype="application/vnd.docker.distribution.manifest.v2+json"), but next try will be accepted (http.request.contenttype="application/vnd.docker.distribution.manifest.v1+prettyjws").


Comment 2 Mike Fiedler 2016-08-08 16:50:54 UTC
I don't think this belongs in ON_QA if there is no code change.   You can answer it as NOTABUG if you feel filling the logs with these error messages is acceptable to end-users.

Comment 3 Alexey Gladkov 2016-08-08 17:09:44 UTC
I have no right to put such status. Available status: NEW, ASSIGNED, POST, MODIFIED, ON_QA, VERIFIED, RELEASE_PENDING, CLOSED.

Note You need to log in before you can comment on or make changes to this bug.