Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1361613 - Deployment of RDO M2, Gnocchi statsd gets configured only on first controller in HA environment and it fails to start on the rest of 2 controllers
Summary: Deployment of RDO M2, Gnocchi statsd gets configured only on first controlle...
Keywords:
Status: CLOSED WORKSFORME
Alias: None
Product: RDO
Classification: Community
Component: openstack-tripleo
Version: trunk
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
: trunk
Assignee: James Slagle
QA Contact: Shai Revivo
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-07-29 14:07 UTC by mlammon
Modified: 2017-06-18 12:13 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-06-18 12:13:24 UTC


Attachments (Terms of Use)

Description mlammon 2016-07-29 14:07:55 UTC
RDO M2:Deployment of RDO M2,  Gnocchi statsd gets configured only on first controller in HA environment and it fails to start on the rest of 2 controllers

Environment:
openstack-gnocchi-statsd-2.2.0-0.20160725220237.1611398.el7.centos.noarch


Description:
Deployment of RDO M2,  Gnocchi statsd gets configured only on first controller in HA environment and it fails to start on the rest of 2 controllers:

Repo used (Newton M2):
sudo wget http://trunk.rdoproject.org/centos7/delorean-deps.repo
sudo wget http://trunk.rdoproject.org/centos7/current-passed-ci/delorean.repo

Images used:
https://ci.centos.org/artifacts/rdo/images/master/delorean/stable/

Steps to reproduce:

Deploy overcloud and will reproduce issue.
openstack overcloud deploy --templates -e /usr/share/openstack-tripleo-heat-templates/environments/puppet-pacemaker.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /home/stack/network-environment.yaml --control-scale 3 --compute-scale 1 --ceph-storage-scale 1 --neutron-network-type vxlan --neutron-tunnel-types vxlan --ntp-server clock.redhat.com --libvirt-type qemu

[root@overcloud-controller-1 ~]# grep -v ^# /etc/gnocchi/gnocchi.conf | grep -v ^$
[DEFAULT]
log_dir = /var/log/gnocchi
[api]
max_limit = 1000
host=192.168.100.20
workers=4
port=8041
[archive_policy]
[cors]
[cors.subdomain]
[database]
[indexer]
url = mysql+pymysql://gnocchi:jM6RnRFYcVzqQ8eKtVnH9yveb@192.168.100.14/gnocchi
[metricd]
[oslo_middleware]
[oslo_policy]
policy_file = /etc/gnocchi/policy.json
[statsd]
[storage]
driver = file
file_basepath = /var/lib/gnocchi
[keystone_authtoken]
auth_uri=http://192.168.100.14:5000/v2.0
identity_uri=http://192.0.2.8:35357
admin_user=gnocchi
admin_password=jM6RnRFYcVzqQ8eKtVnH9yveb
admin_tenant_name=service
 
[root@overcloud-controller-2 ~]# grep -v ^# /etc/gnocchi/gnocchi.conf | grep -v ^$
[DEFAULT]
log_dir = /var/log/gnocchi
[api]
max_limit = 1000
host=192.168.100.15
workers=4
port=8041
[archive_policy]
[cors]
[cors.subdomain]
[database]
[indexer]
url = mysql+pymysql://gnocchi:jM6RnRFYcVzqQ8eKtVnH9yveb@192.168.100.14/gnocchi
[metricd]
[oslo_middleware]
[oslo_policy]
policy_file = /etc/gnocchi/policy.json
[statsd]
[storage]
driver = file
file_basepath = /var/lib/gnocchi
[keystone_authtoken]
auth_uri=http://192.168.100.14:5000/v2.0
identity_uri=http://192.0.2.8:35357
admin_user=gnocchi
admin_password=jM6RnRFYcVzqQ8eKtVnH9yveb
admin_tenant_name=service


[root@overcloud-controller-1 ~]# cat /var/log/gnocchi/statsd.log
2016-07-28 17:49:17.113 11490 CRITICAL gnocchi [-] RequiredOptError: value required for option: statsd.resource_id
2016-07-28 17:49:17.113 11490 ERROR gnocchi Traceback (most recent call last):
2016-07-28 17:49:17.113 11490 ERROR gnocchi   File "/usr/bin/gnocchi-statsd", line 10, in <module>
2016-07-28 17:49:17.113 11490 ERROR gnocchi     sys.exit(statsd())
2016-07-28 17:49:17.113 11490 ERROR gnocchi   File "/usr/lib/python2.7/site-packages/gnocchi/cli.py", line 71, in statsd
2016-07-28 17:49:17.113 11490 ERROR gnocchi     statsd_service.start()
2016-07-28 17:49:17.113 11490 ERROR gnocchi   File "/usr/lib/python2.7/site-packages/gnocchi/statsd.py", line 172, in start
2016-07-28 17:49:17.113 11490 ERROR gnocchi     raise cfg.RequiredOptError(field, cfg.OptGroup("statsd"))
2016-07-28 17:49:17.113 11490 ERROR gnocchi RequiredOptError: value required for option: statsd.resource_id
2016-07-28 17:49:17.113 11490 ERROR gnocchi



[root@overcloud-controller-2 ~]# pcs status
Cluster name: tripleo_cluster
Last updated: Fri Jul 29 13:06:23 2016		Last change: Thu Jul 28 17:59:48 2016 by root via cibadmin on overcloud-controller-1
Stack: corosync
Current DC: overcloud-controller-1 (version 1.1.13-10.el7_2.2-44eb2dd) - partition with quorum
3 nodes and 118 resources configured

Online: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]

Full list of resources:

 ip-192.168.120.13	(ocf::heartbeat:IPaddr2):	Started overcloud-controller-1
 Clone Set: haproxy-clone [haproxy]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 ip-192.0.2.8	(ocf::heartbeat:IPaddr2):	Started overcloud-controller-2
 Master/Slave Set: galera-master [galera]
     Masters: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: memcached-clone [memcached]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 ip-192.168.200.181	(ocf::heartbeat:IPaddr2):	Started overcloud-controller-0
 ip-192.168.100.10	(ocf::heartbeat:IPaddr2):	Started overcloud-controller-0
 Clone Set: rabbitmq-clone [rabbitmq]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-core-clone [openstack-core]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 ip-192.168.110.10	(ocf::heartbeat:IPaddr2):	Started overcloud-controller-2
 Master/Slave Set: redis-master [redis]
     Masters: [ overcloud-controller-2 ]
     Slaves: [ overcloud-controller-0 overcloud-controller-1 ]
 Clone Set: mongod-clone [mongod]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 ip-192.168.100.14	(ocf::heartbeat:IPaddr2):	Started overcloud-controller-1
 Clone Set: openstack-gnocchi-metricd-clone [openstack-gnocchi-metricd]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-gnocchi-statsd-clone [openstack-gnocchi-statsd]
     Started: [ overcloud-controller-0 ]
     Stopped: [ overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-aodh-evaluator-clone [openstack-aodh-evaluator]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-nova-scheduler-clone [openstack-nova-scheduler]
     Started: [ overcloud-controller-1 overcloud-controller-2 ]
     Stopped: [ overcloud-controller-0 ]
 Clone Set: neutron-l3-agent-clone [neutron-l3-agent]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-netns-cleanup-clone [neutron-netns-cleanup]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-ovs-cleanup-clone [neutron-ovs-cleanup]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 openstack-cinder-volume	(systemd:openstack-cinder-volume):	Started overcloud-controller-1
 Clone Set: openstack-nova-consoleauth-clone [openstack-nova-consoleauth]
     Started: [ overcloud-controller-1 overcloud-controller-2 ]
     Stopped: [ overcloud-controller-0 ]
 Clone Set: openstack-heat-engine-clone [openstack-heat-engine]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-ceilometer-api-clone [openstack-ceilometer-api]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-aodh-listener-clone [openstack-aodh-listener]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-metadata-agent-clone [neutron-metadata-agent]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-aodh-notifier-clone [openstack-aodh-notifier]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-heat-api-clone [openstack-heat-api]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-ceilometer-collector-clone [openstack-ceilometer-collector]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-glance-api-clone [openstack-glance-api]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-cinder-scheduler-clone [openstack-cinder-scheduler]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-nova-api-clone [openstack-nova-api]
     Started: [ overcloud-controller-1 overcloud-controller-2 ]
     Stopped: [ overcloud-controller-0 ]
 Clone Set: openstack-heat-api-cloudwatch-clone [openstack-heat-api-cloudwatch]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-glance-registry-clone [openstack-glance-registry]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-ceilometer-notification-clone [openstack-ceilometer-notification]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-cinder-api-clone [openstack-cinder-api]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-dhcp-agent-clone [neutron-dhcp-agent]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-openvswitch-agent-clone [neutron-openvswitch-agent]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-nova-novncproxy-clone [openstack-nova-novncproxy]
     Started: [ overcloud-controller-1 overcloud-controller-2 ]
     Stopped: [ overcloud-controller-0 ]
 Clone Set: neutron-server-clone [neutron-server]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-ceilometer-central-clone [openstack-ceilometer-central]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: httpd-clone [httpd]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-heat-api-cfn-clone [openstack-heat-api-cfn]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-nova-conductor-clone [openstack-nova-conductor]
     Started: [ overcloud-controller-1 overcloud-controller-2 ]
     Stopped: [ overcloud-controller-0 ]

Failed Actions:
* ip-192.0.2.8_monitor_10000 on overcloud-controller-2 'not running' (7): call=22, status=complete, exitreason='none',
    last-rc-change='Thu Jul 28 17:36:30 2016', queued=0ms, exec=0ms
* ip-192.168.110.10_monitor_10000 on overcloud-controller-2 'not running' (7): call=55, status=complete, exitreason='none',
    last-rc-change='Thu Jul 28 17:36:30 2016', queued=0ms, exec=0ms
* openstack-gnocchi-statsd_start_0 on overcloud-controller-2 'not running' (7): call=98, status=complete, exitreason='none',
    last-rc-change='Thu Jul 28 17:50:18 2016', queued=0ms, exec=2092ms
* ip-192.168.120.13_monitor_10000 on overcloud-controller-1 'not running' (7): call=6, status=complete, exitreason='none',
    last-rc-change='Thu Jul 28 17:34:59 2016', queued=0ms, exec=0ms
* openstack-gnocchi-statsd_start_0 on overcloud-controller-1 'not running' (7): call=109, status=complete, exitreason='none',
    last-rc-change='Thu Jul 28 17:50:18 2016', queued=0ms, exec=2094ms
* galera_monitor_10000 on overcloud-controller-0 'unknown error' (1): call=28, status=complete, exitreason='local node <overcloud-controller-0> is started, but not in primary mode. Unknown state.',
    last-rc-change='Thu Jul 28 17:36:56 2016', queued=0ms, exec=0ms
* openstack-nova-consoleauth_start_0 on overcloud-controller-0 'OCF_TIMEOUT' (198): call=181, status=Timed Out, exitreason='none',
    last-rc-change='Thu Jul 28 17:55:18 2016', queued=0ms, exec=19990ms


PCSD Status:
  overcloud-controller-0: Online
  overcloud-controller-1: Online
  overcloud-controller-2: Online

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled
[root@overcloud-controller-2 ~]# pcs status | grep -i stopped -B2
 Clone Set: openstack-gnocchi-statsd-clone [openstack-gnocchi-statsd]
     Started: [ overcloud-controller-0 ]
     Stopped: [ overcloud-controller-1 overcloud-controller-2 ]
--
 Clone Set: openstack-nova-scheduler-clone [openstack-nova-scheduler]
     Started: [ overcloud-controller-1 overcloud-controller-2 ]
     Stopped: [ overcloud-controller-0 ]
--
 Clone Set: openstack-nova-consoleauth-clone [openstack-nova-consoleauth]
     Started: [ overcloud-controller-1 overcloud-controller-2 ]
     Stopped: [ overcloud-controller-0 ]
--
 Clone Set: openstack-nova-api-clone [openstack-nova-api]
     Started: [ overcloud-controller-1 overcloud-controller-2 ]
     Stopped: [ overcloud-controller-0 ]
--
 Clone Set: openstack-nova-novncproxy-clone [openstack-nova-novncproxy]
     Started: [ overcloud-controller-1 overcloud-controller-2 ]
     Stopped: [ overcloud-controller-0 ]
--
 Clone Set: openstack-nova-conductor-clone [openstack-nova-conductor]
     Started: [ overcloud-controller-1 overcloud-controller-2 ]
     Stopped: [ overcloud-controller-0 ]

Comment 1 Christopher Brown 2017-06-18 12:13:24 UTC
Sorry you didn't get a response. I can't reproduce this now. Please re-open if still a problem.


Note You need to log in before you can comment on or make changes to this bug.