Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.

Bug 1602051

Summary: [Deployment] OpenDaylight container unhealthy on fresh deployment
Product: Red Hat OpenStack Reporter: Sai Sindhur Malleni <smalleni>
Component: openstack-tripleo-commonAssignee: Janki <jchhatba>
Status: CLOSED ERRATA QA Contact: Noam Manos <nmanos>
Severity: high Docs Contact:
Priority: high    
Version: 13.0 (Queens)CC: aadam, jchhatba, lmarsh, mburns, mkolesni, nyechiel, slinaber, smalleni
Target Milestone: z3Keywords: TestOnly, Triaged, ZStream
Target Release: 13.0 (Queens)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: Deployment
Fixed In Version: openstack-tripleo-common-8.6.3-13.el7ost.noarch Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-11-13 22:27:09 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Sai Sindhur Malleni 2018-07-17 17:07:09 UTC
Description of problem:
opendaylight_api container is unhealthy on a fresh deployment.

25532225dc1b        192.168.24.1:8787/rhosp13/openstack-opendaylight:2018-07-13.1                  "kolla_start"            20 hours ago        Up 20 hours (unhealthy)                       opendaylight_api

Version-Release number of selected component (if applicable):
OSP 13 

Pudddle: 2018-07-13.1

How reproducible:
100%

Steps to Reproduce:
1. Deploy OSP13 with ODL
2.
3.

Actual results:
opendaylight_api container is unhealthy

Expected results:
Container should b e healthy

Additional info:

Comment 1 Janki 2018-07-18 05:13:27 UTC
I do see this occasionally. Restarting it helps.

Comment 2 Sai Sindhur Malleni 2018-07-18 12:11:30 UTC
I think restarting helps too. I did not manually restart but due to load the containers on each controller got restarted at some point and then they showed status as healthy.

Comment 4 Janki 2018-08-06 05:53:16 UTC
@Sai, please post the output of "docker inspect opendaylight_api" showing the healthcheck "Log" section like below:

$ sudo docker inspect opendaylight_api
[
    {
        "Id": "49b7939c04180a5dc5ec0634d9fd2f51f6101cc82a0f4c248ad56760a7860c4d",
        .....
        "State": {
            "Status": "running",
        ......
            "Health": {
                "Status": "unhealthy",
                "FailingStreak": 7428,
                "Log": [
                    {
                        "Start": "2018-08-06T05:43:25.167736014Z",
                        "End": "2018-08-06T05:43:25.424213779Z",
                        "ExitCode": 1,
                        "Output": "\n000 :0 0.000 seconds\ncurl: (3) IPv6 numerical address used in URL without brackets\n"
                    },

<snippet>

Comment 5 Janki 2018-09-19 01:15:29 UTC
Sai, did you get a chance to collect above mentioned information from the setup?

Meanwhile, can you check with latest puddle. Healthcheck recently moved to using Diagstatus to get the ODL status and this issue should not be seen with the recent changes.

Comment 7 Janki 2018-10-08 07:37:38 UTC
I ran a fresh deployment based on the latest puddle and ODL container is healthy.

28100a419265        192.168.24.1:8787/rhosp13/openstack-opendaylight:2018-10-02.1                  "kolla_start"            3 days ago          Up 3 days (healthy)                         opendaylight_api

Healthchecks based on diagstatus (merged some time back) are known to be OK and should resolve this issue of unhealthy containers.

This commit https://github.com/openstack/tripleo-common/commit/bb98f5aa7db3c4b05e63af48e593841f2a71913a should solve the issue.

Comment 8 Lon Hohberger 2018-10-08 10:35:29 UTC
According to our records, this should be resolved by openstack-tripleo-common-8.6.3-13.el7ost.  This build is available now.

Comment 10 Noam Manos 2018-11-06 09:30:09 UTC
On puddle 2018-10-18.1 (with rpm openstack-tripleo-common-8.6.3-13.el7ost), I'm seeing this unhealthy state of openstack-neutron-server-opendaylight:
 
[root@titan13 ~]# ssh stack@undercloud-0

[stack@undercloud-0 ~]$ cat /etc/yum.repos.d/latest-installed 
13   -p 2018-10-18.1

(overcloud) [stack@undercloud-0 ~]$ ssh heat-admin@controller-0 "sudo docker ps | grep opendaylight"
ebe8c4165108        192.168.24.1:8787/rhosp13/openstack-neutron-server-opendaylight:2018-10-18.1   "kolla_start"            12 days ago         Up 11 days (healthy)                         neutron_api
dd7632c20c8c        192.168.24.1:8787/rhosp13/openstack-opendaylight:2018-10-18.1                  "kolla_start"            12 days ago         Up 11 days (unhealthy)                       opendaylight_api

(overcloud) [stack@undercloud-0 ~]$ ssh heat-admin@controller-1 "sudo docker ps | grep opendaylight"
7e6e0a864744        192.168.24.1:8787/rhosp13/openstack-neutron-server-opendaylight:2018-10-18.1   "kolla_start"            12 days ago         Up 11 days (healthy)                         neutron_api
44579ff7cf35        192.168.24.1:8787/rhosp13/openstack-opendaylight:2018-10-18.1                  "kolla_start"            12 days ago         Up 11 days (unhealthy)                       opendaylight_api

(overcloud) [stack@undercloud-0 ~]$ ssh heat-admin@controller-2 "sudo docker ps | grep opendaylight"
41804c0acae4        192.168.24.1:8787/rhosp13/openstack-neutron-server-opendaylight:2018-10-18.1   "kolla_start"            12 days ago         Up 11 days (healthy)                         neutron_api
696921a7060b        192.168.24.1:8787/rhosp13/openstack-opendaylight:2018-10-18.1                  "kolla_start"            12 days ago         Up 11 days (unhealthy)                       opendaylight_api

Comment 12 Noam Manos 2018-11-07 12:26:19 UTC
On OSP version: 13   2018-11-05.3


+--------------------------------------+-----------------+--------+-------------------------------------------------------------+--------+-------------+
| ID                                   | Name            | Status | Networks                                                    | Image  | Flavor      |
+--------------------------------------+-----------------+--------+-------------------------------------------------------------+--------+-------------+
| 9e741ac8-84ce-46b3-8005-3d507dde68c4 | rhel75_vm1_net2 | ACTIVE | net_ipv64_2=2002::f816:3eff:fea6:a0b5, 10.0.2.5, 10.0.0.215 | rhel75 | rhel_flavor |
| ac5e5d20-b13a-45f7-bfba-1082be31b47d | rhel75_vm1_net1 | ACTIVE | net_ipv64_1=10.0.1.5, 2001::f816:3eff:fe3e:a2d8, 10.0.0.233 | rhel75 | rhel_flavor |
+--------------------------------------+-----------------+--------+-------------------------------------------------------------+--------+-------------+

----------------------------------------------------------------------------------------------------
Creating and testing multiple VMs and Networks completed. Please verify output contains no failures.

To SSH into VM:
ssh -i tester_key.pem cloud-user@SERVER_FIP
[stack@undercloud-0 ~]$ ssh heat-admin@controller-0 "docker ps | grep opendaylight"
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.26/containers/json: dial unix /var/run/docker.sock: connect: permission denied
[stack@undercloud-0 ~]$ ssh heat-admin@controller-0 "sudo docker ps | grep opendaylight"
b9c5aa578f6f        192.168.24.1:8787/rhosp13/openstack-neutron-server-opendaylight:2018-11-05.3   "kolla_start"            14 hours ago        Up 14 hours (healthy)                         neutron_api
21b3de2359d9        192.168.24.1:8787/rhosp13/openstack-opendaylight:2018-11-05.3                  "kolla_start"            14 hours ago        Up 14 hours (healthy)                         opendaylight_api
[stack@undercloud-0 ~]$ ssh heat-admin@controller-1 "sudo docker ps | grep opendaylight"
f68bed578be5        192.168.24.1:8787/rhosp13/openstack-neutron-server-opendaylight:2018-11-05.3   "kolla_start"            14 hours ago        Up 14 hours (healthy)                         neutron_api
645d872e77dc        192.168.24.1:8787/rhosp13/openstack-opendaylight:2018-11-05.3                  "kolla_start"            14 hours ago        Up 14 hours (healthy)                         opendaylight_api
[stack@undercloud-0 ~]$ ssh heat-admin@controller-2 "sudo docker ps | grep opendaylight"
3129cef6b9dc        192.168.24.1:8787/rhosp13/openstack-neutron-server-opendaylight:2018-11-05.3   "kolla_start"            14 hours ago        Up 14 hours (healthy)                         neutron_api
0556276cb3e8        192.168.24.1:8787/rhosp13/openstack-opendaylight:2018-11-05.3                  "kolla_start"            14 hours ago        Up 14 hours (healthy)                         opendaylight_api

Comment 18 errata-xmlrpc 2018-11-13 22:27:09 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:3587