Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1517370 - Containerized overcloud deployment with standalone nova role fails on ControllerDeployment_Step5.0
Summary: Containerized overcloud deployment with standalone nova role fails on Control...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: rhosp-director
Version: 12.0 (Pike)
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: ---
Assignee: Angus Thomas
QA Contact: Amit Ugol
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-11-24 16:57 UTC by Gabriel Szasz
Modified: 2017-11-30 12:52 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-11-28 20:53:01 UTC


Attachments (Terms of Use)

Description Gabriel Szasz 2017-11-24 16:57:39 UTC
Created attachment 1358712 [details]
/var/log/containers/ceilometer/ceilometer-upgrade.log on controller node

Description of problem:
Containerized overcloud deployment fails in ControllerDeployment_Step5.0, when 'novacontrol' composable role profile [1] is being used.

[1] https://github.com/redhat-openstack/infrared/tree/master/plugins/tripleo-overcloud/files/roles/novacontrol

Version-Release number of selected component (if applicable):
OSP-12 (rhosp-director-images-12.0-20171122.1.el7ost, InfraRed branch: stable)

How reproducible:
100%

Steps to Reproduce:
1. Deploy OSP-12 using OSP-d-Customized-Deployment-virt Jenkins job with following set of parameters:
   TOPOLOGY: undercloud:1,controller:1,compute:1,novacontrol:1
   UNDERCLOUD_DISK: 100 [GB]
   CONTROLLER_RAM: 32768 [MB]
   CONTROLLER_CPU: 8
   CONTAINERS: True
   COMPOSABLE_ROLES: novacontrol
   USE_THT_ROLES: False


Actual results:

The deployment will fail on:

stdout: overcloud.AllNodesDeploySteps.ControllerDeployment_Step5.0:
  resource_type: OS::Heat::StructuredDeployment
  physical_resource_id: ea772795-5a50-4060-b9a1-c9860d432b83
  status: CREATE_FAILED
  status_reason: |
    Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 2
  deploy_stdout: |
    ...
            "Error running ['docker', 'run', '--name', 'ceilometer_gnocchi_upgrade', '--label', 'config_id=tripleo_step5', '--label', 'container_name=ceilometer_gnocchi_upgrade', '--label', 'managed_by=paunch', '--label', 'config_data={\"start_order\": 1, \"image\": \"192.168.24.1:8787/rhosp12/openstack-ceilometer-central-docker:20171122.1\", \"command\": [\"/usr/bin/bootstrap_host_exec\", \"ceilometer_agent_central\", \"su ceilometer -s /bin/bash -c \\'for n in {1..10}; do /usr/bin/ceilometer-upgrade --skip-metering-database && exit 0 || sleep 5; done; exit 1\\'\"], \"user\": \"root\", \"volumes\": [\"/etc/hosts:/etc/hosts:ro\", \"/etc/localtime:/etc/localtime:ro\", \"/etc/puppet:/etc/puppet:ro\", \"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\", \"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\", \"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\", \"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\", \"/dev/log:/dev/log\", \"/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro\", \"/var/lib/config-data/ceilometer/etc/ceilometer/:/etc/ceilometer/:ro\", \"/var/log/containers/ceilometer:/var/log/ceilometer\"], \"net\": \"host\", \"detach\": false, \"privileged\": false}', '--net=host', '--privileged=false', '--user=root', '--volume=/etc/hosts:/etc/hosts:ro', '--volume=/etc/localtime:/etc/localtime:ro', '--volume=/etc/puppet:/etc/puppet:ro', '--volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro', '--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro', '--volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro', '--volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro', '--volume=/dev/log:/dev/log', '--volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro', '--volume=/var/lib/config-data/ceilometer/etc/ceilometer/:/etc/ceilometer/:ro', '--volume=/var/log/containers/ceilometer:/var/log/ceilometer', '192.168.24.1:8787/rhosp12/openstack-ceilometer-central-docker:20171122.1', '/usr/bin/bootstrap_host_exec', 'ceilometer_agent_central', \"su ceilometer -s /bin/bash -c 'for n in {1..10}; do /usr/bin/ceilometer-upgrade --skip-metering-database && exit 0 || sleep 5; done; exit 1'\"]. [1]", 
            "stdout: "
        ], 
        "failed_when_result": true
    }
    	to retry, use: --limit @/var/lib/heat-config/heat-config-ansible/ca466324-f4d2-4633-8358-caad6f7758d7_playbook.retry
    
    PLAY RECAP *********************************************************************
    localhost                  : ok=7    changed=2    unreachable=0    failed=1   
    
    (truncated, view all with --long)
  deploy_stderr: |

overcloud.AllNodesDeploySteps.NovacontrolDeployment_Step5.0:
  resource_type: OS::Heat::StructuredDeployment
  physical_resource_id: 3072df8f-d061-4561-b0a1-0693ed87088f
  status: CREATE_FAILED
  status_reason: |
    CREATE aborted
  deploy_stdout: |
None
  deploy_stderr: |
None


Expected results:

The deployment will pass without errors.


Additional info:

The /var/log/containers/ceilometer/ceilometer-upgrade.log on the controller node reveals following errors:

2017-11-24 15:15:09.665 12 INFO ceilometer.cmd.storage [-] Skipping metering database upgrade
2017-11-24 15:15:09.700 12 CRITICAL ceilometer [-] Unhandled error: Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-824f02c4-09d6-4b02-9b86-8f5e6b4577e7)
2017-11-24 15:15:09.700 12 ERROR ceilometer Traceback (most recent call last):
2017-11-24 15:15:09.700 12 ERROR ceilometer   File "/usr/bin/ceilometer-upgrade", line 10, in <module>
2017-11-24 15:15:09.700 12 ERROR ceilometer     sys.exit(upgrade())
2017-11-24 15:15:09.700 12 ERROR ceilometer   File "/usr/lib/python2.7/site-packages/ceilometer/cmd/storage.py", line 59, in upgrade
2017-11-24 15:15:09.700 12 ERROR ceilometer     gnocchi_client.upgrade_resource_types(conf)
2017-11-24 15:15:09.700 12 ERROR ceilometer   File "/usr/lib/python2.7/site-packages/ceilometer/gnocchi_client.py", line 194, in upgrade_resource_types
2017-11-24 15:15:09.664 12 WARNING oslo_reports.guru_meditation_report [-] Guru meditation now registers SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 will no longer be registered in a future release, so please use SIGUSR2 to generate reports.

See attachment for complete log.


Note You need to log in before you can comment on or make changes to this bug.