Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1516021 - OSP11 -> OSP12 upgrade: controller-0 is missing the following containers running: nova_metadata, heat_api_cfn, heat_api_cron
Summary: OSP11 -> OSP12 upgrade: controller-0 is missing the following containers runn...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: rhosp-director
Version: 12.0 (Pike)
Hardware: x86_64
OS: Linux
unspecified
urgent
Target Milestone: ---
: ---
Assignee: Marios Andreou
QA Contact: Amit Ugol
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-11-21 19:49 UTC by Ziv
Modified: 2017-11-29 19:52 UTC (History)
12 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-11-29 19:52:57 UTC


Attachments (Terms of Use)
osp11 composable roles templates (deleted)
2017-11-21 19:49 UTC, Ziv
no flags Details
errors from attached controller logs (deleted)
2017-11-22 13:05 UTC, Marios Andreou
no flags Details

Description Ziv 2017-11-21 19:49:54 UTC
Created attachment 1356969 [details]
osp11 composable roles templates

Description of problem:
Upgrade failed to complete, following error was received with an infrared tool:
http://pastebin.test.redhat.com/534044

Stack update status looks completed, although it failed to finish it properly:
(undercloud) [stack@undercloud-0 ~]$ openstack stack list
+--------------------------------------+------------+----------------------------------+-----------------+----------------------+----------------------+
| ID                                   | Stack Name | Project                          | Stack Status    | Creation Time        | Updated Time         |
+--------------------------------------+------------+----------------------------------+-----------------+----------------------+----------------------+
| 57b1f9e9-e191-4ad5-82a6-b56c349220e8 | overcloud  | 75fb9f571f4c44948b12b2ec9394adf1 | UPDATE_COMPLETE | 2017-11-21T08:00:23Z | 2017-11-21T12:05:40Z |
+--------------------------------------+------------+----------------------------------+-----------------+----------------------+----------------------+

(undercloud) [stack@undercloud-0 ~]$ openstack server list
+--------------------------------------+------------------+--------+----------------------+----------------+----------------+
| ID                                   | Name             | Status | Networks             | Image          | Flavor         |
+--------------------------------------+------------------+--------+----------------------+----------------+----------------+
| 739b8aa9-be7f-4be6-8062-6e2bb9968ea3 | controller-2     | ACTIVE | ctlplane=192.0.20.32 | overcloud-full | controller     |
| eab16063-2af1-4f92-a22b-8b71b1ed99a7 | computeovsdpdk-0 | ERROR  | ctlplane=192.0.20.31 | overcloud-full | computeovsdpdk |
| 3f4a01d4-0cbe-4e6b-bbfb-c49113d83f79 | controller-0     | ERROR  | ctlplane=192.0.20.27 | overcloud-full | controller     |
| 2015f225-933f-432f-a634-b29e3de1c7c7 | controller-1     | ERROR  | ctlplane=192.0.20.30 | overcloud-full | controller     |
+--------------------------------------+------------------+--------+----------------------+----------------+----------------+


Additional checks on the controller-0 have revealed that some of the containers have missing:

[root@controller-0 heat-admin]# docker ps | egrep 'nova_metadata|heat_api_cfn|heat_api_cron|nova_api'
728f828af3d7  docker-registry.engineering.redhat.com/rhosp12/openstack-nova-api-docker:20171116.1  "kolla_start"  8 hours ago  Up 7 hours (healthy) nova_api_cron
[root@controller-0 heat-admin]#

How reproducible:
100%

Steps to Reproduce:
1. Deploy an OSP11 composable roles env by using the attached templates
2. Upgrade to OSP12.


Actual results:
The upgrade has failed.

Expected results:
The upgrade should pass.

Comment 1 Ziv 2017-11-21 20:17:14 UTC
An SOS report from the controller-0 could be found here:
https://drive.google.com/a/redhat.com/file/d/1P-8QjbHatsLSkgIp67Y3OoI42a3DRsv3/view?usp=sharing

Thanks.

Comment 2 Marios Andreou 2017-11-22 13:05:54 UTC
Created attachment 1357476 [details]
errors from attached controller logs

Hi Ziv, spent some time looking at the logs.. I don't see any issue with the upgrade_tasks there (i.e. there is no failure). The biggest issue I see is with the container images and it may explain why some of the containers didn't come up.

I don't know why you are seeing these "No such container" errors (maybe something to do with the image prepare command?) and I also don't understand why given these errors the deployment didn't fail.

I will reach out to containers dfg see if they can help out here.

Comment 3 Marios Andreou 2017-11-22 13:14:37 UTC
proactively ping DFG:Containers Martin grateful if someone can check here and help with triage. I just spoke to amitu who thinks it may be puddle related so we may try to reproduce first but reaching out anyway given the time to release :)

Comment 4 Marios Andreou 2017-11-22 13:15:10 UTC
martin please see comment #3 thanks

Comment 5 Marios Andreou 2017-11-23 09:56:53 UTC
thanks mandre for irc ping just now, clearing needinfo and waiting on amitu and ziv to see if this reproduces on latest puddle

Comment 6 Marios Andreou 2017-11-28 12:26:17 UTC
adding needinfo so it isn't forgotten can we please verify if this reproduces in latest builds thanks

Comment 7 Ziv 2017-11-29 19:52:57 UTC
Hi,

While using puddle 2017-11-28.3, an upgrade has finished successfully.

Thanks,
Ziv


Note You need to log in before you can comment on or make changes to this bug.