Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1696393 - Overcloud deploy fails pulling from ceph repository
Summary: Overcloud deploy fails pulling from ceph repository
Status: POST
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: ansible-role-container-registry
Version: 15.0 (Stein)
Hardware: Unspecified
OS: Unspecified
Target Milestone: beta
: 15.0 (Stein)
Assignee: Steve Baker
QA Contact: Sasha Smolyak
Depends On:
TreeView+ depends on / blocked
Reported: 2019-04-04 18:24 UTC by Alistair Tonner
Modified: 2019-04-16 13:04 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed:
Target Upstream Version:

Attachments (Terms of Use)

System ID Priority Status Summary Last Updated
Launchpad 1823821 None None None 2019-04-09 01:11:21 UTC
OpenStack gerrit 651051 None None None 2019-04-09 01:10:27 UTC

Description Alistair Tonner 2019-04-04 18:24:29 UTC
Description of problem: Overcloud deploy fails to pull from ceph repository with KeyError: Layers

Version-Release number of selected component (if applicable):

RHEL8 - 1845

ansible.noarch                                2.7.6-1.el8                                          @rhelosp-15.0-trunk
ansible-pacemaker.noarch                      1.0.4-0.20190129114541.0e4d7c0.el8ost                @rhelosp-15.0-trunk
ansible-role-atos-hsm.noarch                  0.1.1-0.20190306173142.f6f9c3f.el8ost                @rhelosp-15.0-trunk
ansible-role-chrony.noarch                    0.0.1-0.20190327040343.068668b.el8ost                @rhelosp-15.0-trunk
ansible-role-container-registry.noarch        1.0.1-0.20190219021249.d6a749a.el8ost                @rhelosp-15.0-trunk
ansible-role-redhat-subscription.noarch       1.0.2-0.20190215212927.13bf86d.el8ost                @rhelosp-15.0-trunk
ansible-role-thales-hsm.noarch                0.2.1-0.20190306204553.08b5efa.el8ost                @rhelosp-15.0-trunk
ansible-role-tripleo-modify-image.noarch      1.0.1-0.20190402220346.012209a.el8ost                @rhelosp-15.0-trunk
ansible-tripleo-ipsec.noarch                  9.0.1-0.20190220162047.f60ad6c.el8ost                @rhelosp-15.0-trunk
openstack-tripleo-common-containers.noarch    10.6.1-0.20190404000356.3398bec.el8ost               @rhelosp-15.0-trunk
openstack-tripleo-common.noarch               10.6.1-0.20190404000356.3398bec.el8ost               @rhelosp-15.0-trunk
openstack-tripleo-heat-templates.noarch       10.4.1-0.20190403221322.0d98720.el8ost               @rhelosp-15.0-trunk
openstack-tripleo-image-elements.noarch       10.3.1-0.20190325204940.253fe88.el8ost               @rhelosp-15.0-trunk
openstack-tripleo-puppet-elements.noarch      10.2.1-0.20190327211339.0f6cacb.el8ost               @rhelosp-15.0-trunk
openstack-tripleo-validations.noarch          10.3.1-0.20190403171315.a4c40f2.el8ost               @rhelosp-15.0-trunk
puppet-tripleo.noarch                         10.3.1-0.20190403180925.81d7714.el8ost               @rhelosp-15.0-trunk
python3-heat-agent-ansible.noarch             1.8.1-0.20190402070337.ad2a5d1.el8ost                @rhelosp-15.0-trunk
python3-tripleoclient-heat-installer.noarch   11.3.1-0.20190403170353.73cc438.el8ost               @rhelosp-15.0-trunk
python3-tripleoclient.noarch                  11.3.1-0.20190403170353.73cc438.el8ost               @rhelosp-15.0-trunk
python3-tripleo-common.noarch                 10.6.1-0.20190404000356.3398bec.el8ost               @rhelosp-15.0-trunk

How reproducible:


Steps to Reproduce:
  Run attached script to deploy overcloud

Actual results:

IN :

Using config files: ['/tmp/tmpbpv5mulo']
Image prepare failed: 'layers'
Traceback (most recent call last):
  File "/usr/bin/tripleo-container-image-prepare", line 131, in <module>
    env, roles_data, cleanup=args.cleanup, dry_run=args.dry_run)
  File "/usr/lib/python3.6/site-packages/tripleo_common/image/", line 203, in container_images_prepare_multi
  File "/usr/lib/python3.6/site-packages/tripleo_common/image/", line 181, in upload
  File "/usr/lib/python3.6/site-packages/tripleo_common/image/", line 1570, in run_tasks
    result = uploader.upload_image(first_task)
  File "/usr/lib/python3.6/site-packages/tripleo_common/image/", line 886, in upload_image
    source_layers = [l['digest'] for l in manifest['layers']]
KeyError: 'layers'

Expected results:

  Containers deployed, overcloud deploys successfully

Additional info:

  Also notes in /var/log/containers/mistral/overcloud/ansible.log:

fatal: [controller-0]: FAILED! => {"changed": true, "cmd": "podman pull

(for each controller)

Comment 1 Artem Hrechanychenko 2019-04-04 18:26:50 UTC
Also got that issue today on titan46

Comment 7 Steve Baker 2019-04-08 20:20:55 UTC serves v1 manifests instead of the v2 manifests served by most other registries. The container prepare has support for v1 but it looks like there is a bug in that support. I'll take a look

Comment 10 John Fulton 2019-04-16 13:04:47 UTC
I hit this in ceph-ansible-4.0.0-0.beta1.95.g6d506dba.el8cp.noarch today though it was working yesterday; all I did was redeploy. I then tried via the CLI from my undercloud to podman pull it and I saw:

Error determining manifest MIME type for docker:// pinging docker registry returned: Get EOF

More details in:

Note You need to log in before you can comment on or make changes to this bug.