Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1695101 - [OSP15][Overcloud] pcs cluster is not currently running on Overcloud with 1 controller + 1 compute
Summary: [OSP15][Overcloud] pcs cluster is not currently running on Overcloud with 1 c...
Keywords:
Status: CLOSED WORKSFORME
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: rabbitmq-server
Version: 15.0 (Stein)
Hardware: x86_64
OS: Linux
high
high
Target Milestone: ga
: ---
Assignee: Peter Lemenkov
QA Contact: pkomarov
URL:
Whiteboard: DFG:PIDONE
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-04-02 13:25 UTC by Artem Hrechanychenko
Modified: 2019-04-08 11:13 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-04-08 11:13:55 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Artem Hrechanychenko 2019-04-02 13:25:54 UTC
Description of problem:

OSP15 deployment with 1 controller + 1compute

(overcloud) [stack@undercloud-0 ~]$ openstack endpoint list
Failed to discover available identity versions when contacting https://10.0.0.101:13000. Attempting to parse version from URL.
Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Unable to establish connection to https://10.0.0.101:13000: HTTPSConnectionPool(host='10.0.0.101', port=13000): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f8b8404def0>: Failed to establish a new connection: [Errno 113] No route to host',))


[heat-admin@controller-0 ~]$ sudo pcs status
Error: cluster is not currently running on this node

b738ccc8f3e5  192.168.24.1:8787/rhosp15/openstack-rabbitmq:20190327.1                 kolla_start           23 hours ago  Up 23 hours ago         rabbitmq

cat overcloud_deploy.sh 
#!/bin/bash

openstack overcloud deploy \
--timeout 100 \
--templates /usr/share/openstack-tripleo-heat-templates \
--stack overcloud \
--libvirt-type kvm \
--ntp-server clock.redhat.com \
-e /home/stack/virt/config_lvm.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/virt/network/network-environment.yaml \
-e /home/stack/virt/enable-tls.yaml \
-e /home/stack/virt/inject-trust-anchor.yaml \
-e /home/stack/virt/public_vip.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/ssl/tls-endpoints-public-ip.yaml \
-e /home/stack/virt/hostnames.yml \
-e /home/stack/virt/nodes_data.yaml \
-e ~/containers-prepare-parameter.yaml \
-e /home/stack/virt/extra.yaml \
--log-file overcloud_deployment_85.log


Version-Release number of selected component (if applicable):
RHOS_TRUNK-15.0-RHEL-8-20190329.n.0

How reproducible:


Steps to Reproduce:
1.Deploy OSP15 with 1 controller + 1compute + LVM
2. Login to controller
3. Check pcs output

Actual results:
Error: cluster is not currently running on this node

Expected results:
cluster status

Additional info:

Comment 2 Artem Hrechanychenko 2019-04-02 13:34:11 UTC
The reports should be available here: http://rhos-release.virt.bos.redhat.com/log/bz1695101

Comment 3 Michele Baldessari 2019-04-03 12:34:31 UTC
Can you provide a full sosreport? The ones lined in c#2 have only /var/log/containers and few other things. But nothing about the host itself.
Or just ping me when you reproduce and I will look at the system

Thanks,
Michele

Comment 4 Artem Hrechanychenko 2019-04-08 11:13:55 UTC
RHOS_TRUNK-15.0-RHEL-8-20190403.n.3


(undercloud) [stack@undercloud-0 ~]$ openstack endpoint list
+----------------------------------+-----------+------------------+-------------------------+---------+-----------+--------------------------------------------------+
| ID                               | Region    | Service Name     | Service Type            | Enabled | Interface | URL                                              |
+----------------------------------+-----------+------------------+-------------------------+---------+-----------+--------------------------------------------------+
| 062840aed3f24be4a2f0b96e86d18ce3 | regionOne | mistral          | workflowv2              | True    | admin     | http://192.168.24.3:8989/v2                      |
| 0dfb41d35d644620ad4d63037d1ec8d5 | regionOne | heat-cfn         | cloudformation          | True    | public    | https://192.168.24.2:13005/v1                    |
| 12f27b6a287b4f59be18601d50ec682a | regionOne | ironic-inspector | baremetal-introspection | True    | admin     | http://192.168.24.3:5050                         |
| 2f8e4580ff8d4b959ce6297020576cbc | regionOne | ironic           | baremetal               | True    | public    | https://192.168.24.2:13385                       |
| 30e20eb076604a47afd60d817af035aa | regionOne | ironic           | baremetal               | True    | admin     | http://192.168.24.3:6385                         |
| 33a1861e97d5439bab4a196e7664bf02 | regionOne | swift            | object-store            | True    | admin     | http://192.168.24.3:8080                         |
| 40ebef74742d4ed18374742c767b8eef | regionOne | ironic           | baremetal               | True    | internal  | http://192.168.24.3:6385                         |
| 420292bfbdc24495a43b1d59150acf6c | regionOne | swift            | object-store            | True    | internal  | http://192.168.24.3:8080/v1/AUTH_%(tenant_id)s   |
| 49182e17990546afa87ee1dfcf4e363a | regionOne | nova             | compute                 | True    | public    | https://192.168.24.2:13774/v2.1                  |
| 56b4c4d5770f48cc847cc1ae50f9cb6d | regionOne | mistral          | workflowv2              | True    | public    | https://192.168.24.2:13989/v2                    |
| 59dcf0cf632948e49eb5c2267bc1c228 | regionOne | neutron          | network                 | True    | public    | https://192.168.24.2:13696                       |
| 5b03787e321048a98de3758e9a9518dc | regionOne | glance           | image                   | True    | public    | https://192.168.24.2:13292                       |
| 5ba1e5f45b054e99a2c11c062e5fae4c | regionOne | zaqar-websocket  | messaging-websocket     | True    | internal  | ws://192.168.24.3:9000                           |
| 62d783a78e6c4c5da0ae10825bf4d7f0 | regionOne | heat-cfn         | cloudformation          | True    | admin     | http://192.168.24.3:8000/v1                      |
| 663fc87369b749cab7e6bd6edc64a4d0 | regionOne | placement        | placement               | True    | internal  | http://192.168.24.3:8778/placement               |
| 668e5bebc1614c1bb29a89fd0f31f4ec | regionOne | nova             | compute                 | True    | admin     | http://192.168.24.3:8774/v2.1                    |
| 6ac113248cfb4352860959bbd44d2658 | regionOne | glance           | image                   | True    | admin     | http://192.168.24.3:9292                         |
| 7c31e835085c41f1b4708a462327a0ad | regionOne | nova             | compute                 | True    | internal  | http://192.168.24.3:8774/v2.1                    |
| 7ee05dfeac4c4a5d9f075a8b1066fce2 | regionOne | heat-cfn         | cloudformation          | True    | internal  | http://192.168.24.3:8000/v1                      |
| 7fb96e44d584449a9700680201e4a267 | regionOne | placement        | placement               | True    | public    | https://192.168.24.2:13778/placement             |
| 8b80b1b5a0a14437a7118db20266c938 | regionOne | keystone         | identity                | True    | public    | https://192.168.24.2:13000                       |
| 8eda3081f6684f71a0d7d96d4d3353cb | regionOne | neutron          | network                 | True    | internal  | http://192.168.24.3:9696                         |
| 94e9989c3e934409899aae5c22dbe897 | regionOne | keystone         | identity                | True    | internal  | http://192.168.24.3:5000                         |
| 9d7a638a4b434251bfc733f1be1b98c7 | regionOne | keystone         | identity                | True    | admin     | http://192.168.24.3:35357                        |
| 9f6d20e63d06487a9fecb5d7c21726d3 | regionOne | heat             | orchestration           | True    | public    | https://192.168.24.2:13004/v1/%(tenant_id)s      |
| a245525df4854d7e99debdf338421637 | regionOne | placement        | placement               | True    | admin     | http://192.168.24.3:8778/placement               |
| b5b44ac3c7d44a619c58c459b3929550 | regionOne | mistral          | workflowv2              | True    | internal  | http://192.168.24.3:8989/v2                      |
| c70bfeb5549b413eb21f97affce6cfec | regionOne | ironic-inspector | baremetal-introspection | True    | internal  | http://192.168.24.3:5050                         |
| cad1caf9ed264f98b42d5f1fcf22be0a | regionOne | heat             | orchestration           | True    | internal  | http://192.168.24.3:8004/v1/%(tenant_id)s        |
| cd233cf715f34f7a8849be9e0c78dbc3 | regionOne | swift            | object-store            | True    | public    | https://192.168.24.2:13808/v1/AUTH_%(tenant_id)s |
| cdba0a7dc0e844ab8d9c9eee1e45adb6 | regionOne | zaqar-websocket  | messaging-websocket     | True    | admin     | ws://192.168.24.3:9000                           |
| d7414c8bcf1a49e7b0df34921bff85a5 | regionOne | zaqar-websocket  | messaging-websocket     | True    | public    | wss://192.168.24.2:9000                          |
| d7837b061f5143d6938bdbad9f98e8c1 | regionOne | zaqar            | messaging               | True    | internal  | http://192.168.24.3:8888                         |
| dde5256ea0914c949a7e94f85e17e090 | regionOne | heat             | orchestration           | True    | admin     | http://192.168.24.3:8004/v1/%(tenant_id)s        |
| deb066587481465d873d52f4c04ef883 | regionOne | neutron          | network                 | True    | admin     | http://192.168.24.3:9696                         |
| e80f9929abb444d6b0cd3e92a388abf2 | regionOne | ironic-inspector | baremetal-introspection | True    | public    | https://192.168.24.2:13050                       |
| f08067b4b35e45c4abc36803f656a2a2 | regionOne | glance           | image                   | True    | internal  | http://192.168.24.3:9292                         |
| f8615b34435f45e6bf1665fd1826352f | regionOne | zaqar            | messaging               | True    | public    | https://192.168.24.2:13888                       |
| f87ffe5ba33e4d6f886728dc89ab9fe6 | regionOne | zaqar            | messaging               | True    | admin     | http://192.168.24.3:8888                         |

sudo pcs status
Cluster name: tripleo_cluster
Stack: corosync
Current DC: controller-0 (version 2.0.1-4.el8-0eb7991564) - partition with quorum
Last updated: Mon Apr  8 10:37:52 2019
Last change: Mon Apr  8 10:34:01 2019 by root via cibadmin on controller-0

5 nodes configured
20 resources configured

Online: [ controller-0 ]
GuestOnline: [ galera-bundle-0@controller-0 ovn-dbs-bundle-0@controller-0 rabbitmq-bundle-0@controller-0 redis-bundle-0@controller-0 ]

Full list of resources:

 podman container: galera-bundle [192.168.24.1:8787/rhosp15/openstack-mariadb:pcmklatest]
   galera-bundle-0	(ocf::heartbeat:galera):	Master controller-0
 podman container: rabbitmq-bundle [192.168.24.1:8787/rhosp15/openstack-rabbitmq:pcmklatest]
   rabbitmq-bundle-0	(ocf::heartbeat:rabbitmq-cluster):	Started controller-0
 podman container: redis-bundle [192.168.24.1:8787/rhosp15/openstack-redis:pcmklatest]
   redis-bundle-0	(ocf::heartbeat:redis):	Master controller-0
 ip-192.168.24.7	(ocf::heartbeat:IPaddr2):	Started controller-0
 ip-10.0.0.101	(ocf::heartbeat:IPaddr2):	Started controller-0
 ip-172.17.1.26	(ocf::heartbeat:IPaddr2):	Started controller-0
 ip-172.17.1.123	(ocf::heartbeat:IPaddr2):	Started controller-0
 ip-172.17.3.106	(ocf::heartbeat:IPaddr2):	Started controller-0
 ip-172.17.4.44	(ocf::heartbeat:IPaddr2):	Started controller-0
 podman container: haproxy-bundle [192.168.24.1:8787/rhosp15/openstack-haproxy:pcmklatest]
   haproxy-bundle-podman-0	(ocf::heartbeat:podman):	Started controller-0
 podman container: ovn-dbs-bundle [192.168.24.1:8787/rhosp15/openstack-ovn-northd:pcmklatest]
   ovn-dbs-bundle-0	(ocf::ovn:ovndb-servers):	Master controller-0
 podman container: openstack-cinder-volume [192.168.24.1:8787/rhosp15/openstack-cinder-volume:pcmklatest]
   openstack-cinder-volume-podman-0	(ocf::heartbeat:podman):	Started controller-0

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled


Note You need to log in before you can comment on or make changes to this bug.