Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1516429 - openstack-nova: failed to launch instance on OC: "Host 'overcloud-compute-1.localdomain' is not mapped to any cell" after Scale-UP
Summary: openstack-nova: failed to launch instance on OC: "Host 'overcloud-compute-1.l...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 12.0 (Pike)
Hardware: x86_64
OS: Linux
urgent
urgent
Target Milestone: rc
: 12.0 (Pike)
Assignee: Dan Prince
QA Contact: Artem Hrechanychenko
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-11-22 15:38 UTC by Artem Hrechanychenko
Modified: 2018-02-05 19:18 UTC (History)
11 users (show)

Fixed In Version: openstack-tripleo-heat-templates-7.0.3-13.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-12-13 22:22:16 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2017:3462 normal SHIPPED_LIVE Red Hat OpenStack Platform 12.0 Enhancement Advisory 2018-02-16 01:43:25 UTC
OpenStack gerrit 522397 None master: MERGED tripleo-heat-templates: Add DeployIdentifier to Nova discover hosts container 2017-11-27 18:21:26 UTC

Description Artem Hrechanychenko 2017-11-22 15:38:04 UTC
Description of problem:

Failed to launch new instance after scale-up compute node on cluster with 3ctrl+1comp 

| fault                                | {"message": "Host 'compute-1.localdomain' is not mapped to any cell", "code": 400, "created": "2017-11-22T15:02:58Z"} |

But in the same time I can perform live-migration of instance from compute-0 to compute-1 

Version-Release number of selected component (if applicable):
openstack-tripleo-heat-templates-7.0.3-10.el7ost.noarch


How reproducible:

always
Steps to Reproduce:
1.Deploy 3ctrl+1comp
2.Scale compute
3.Launch instance on new compute

Actual results:
{"message": "Host 'compute-1.localdomain' is not mapped to any cell", "code": 400, "created": "2017-11-22T15:02:58Z"}

Expected results:
instance was launch

Additional info:

Comment 3 Dan Prince 2017-11-22 18:45:19 UTC
One idea on how we might fix this might be to append either the DeployIdentifier to the environment of the discovory hosts "init container" thus forcing it to re-run during update.

The more concerning issue here is this likely isn't the only container we need to do this to. Scaling out and adding a net-new OpenStack service or role for example is likely also broken as it would require us to re-run some of the DB and Keystone init tasks as well. Similar fixes would apply there I think.

Comment 5 Ollie Walsh 2017-11-22 19:09:32 UTC
(In reply to Dan Prince from comment #3)

> The more concerning issue here is this likely isn't the only container we
> need to do this to. Scaling out and adding a net-new OpenStack service or
> role for example is likely also broken as it would require us to re-run some
> of the DB and Keystone init tasks as well. Similar fixes would apply there I
> think.

Yea, that's my concern. dbsync tasks need to be run for all services during a minor updates for example.

Comment 6 Ollie Walsh 2017-11-22 19:13:20 UTC
Maybe should be the default for docker config steps with detach: false, since most will be based on puppet exec resources

Comment 7 Dan Prince 2017-11-22 22:08:20 UTC
I've posted a potential upstream fix here for the Nova discover hosts issue:

https://review.openstack.org/522397 Add DeployIdentifier to Nova discover hosts container

Going to try to test a scale out locally. Given the timing would be good to have early QE feedback on this as well. Simply patching it into your t-h-t directory on the Undercloud would be all you'd need to do.

Artem: does it work for you?

Comment 8 Artem Hrechanychenko 2017-11-23 18:06:09 UTC
I applied your patch[0] to my local t-h-t after deploy before scale-up

after scale-up using infrared :
infrared virsh -v --host-address $HOST --host-key ~/.ssh/id_rsa  --topology-nodes compute:1 --image-url http://download-node-02.eng.bos.redhat.com/brewroot/packages/rhel-guest-image/7.4/176/images/rhel-guest-image-7.4-191.x86_64.qcow2  --topology-extend True

infrared cloud-config --tasks scale_up --scale-nodes compute-1

infrared cloud-config --tasks add_overcloud_hosts


undercloud) [stack@undercloud-0 ~]$ heat stack-list
WARNING (shell) "heat stack-list" is deprecated, please use "openstack stack list" instead
+--------------------------------------+------------+-----------------+----------------------+----------------------+----------------------------------+
| id                                   | stack_name | stack_status    | creation_time        | updated_time         | project                          |
+--------------------------------------+------------+-----------------+----------------------+----------------------+----------------------------------+
| fa1f28ce-4ec1-4f6a-b285-b237aa49af2e | overcloud  | UPDATE_COMPLETE | 2017-11-23T16:35:38Z | 2017-11-23T17:25:25Z | ada9b3af540e42bd84fce45e9972d958 |
+--------------------------------------+------------+-----------------+----------------------+----------------------+----------------------------------+
(undercloud) [stack@undercloud-0 ~]$ nova list
+--------------------------------------+--------------+--------+------------+-------------+------------------------+
| ID                                   | Name         | Status | Task State | Power State | Networks               |
+--------------------------------------+--------------+--------+------------+-------------+------------------------+
| 6f3ea5a4-3d5c-4270-bb29-c755319811c8 | compute-0    | ACTIVE | -          | Running     | ctlplane=192.168.24.9  |
| 59bea50d-aebb-4f42-8f05-b4e6a046f2d3 | compute-1    | ACTIVE | -          | Running     | ctlplane=192.168.24.13 |
| 9a1b15fd-95bc-470a-a559-aedca2b742e9 | controller-0 | ACTIVE | -          | Running     | ctlplane=192.168.24.15 |
| 3c101b5d-3e9a-4326-a4b2-07c3451479a8 | controller-1 | ACTIVE | -          | Running     | ctlplane=192.168.24.16 |
| ea210cbe-be51-49d6-80a5-e13168d6637b | controller-2 | ACTIVE | -          | Running     | ctlplane=192.168.24.11 |
+--------------------------------------+--------------+--------+------------+-------------+------------------------+


(overcloud) [stack@undercloud-0 ~]$ openstack hypervisor list
+----+-----------------------+-----------------+-------------+-------+
| ID | Hypervisor Hostname   | Hypervisor Type | Host IP     | State |
+----+-----------------------+-----------------+-------------+-------+
|  2 | compute-0.localdomain | QEMU            | 172.17.1.13 | up    |
|  5 | compute-1.localdomain | QEMU            | 172.17.1.12 | up    |
+----+-----------------------+-----------------+-------------+-------+
(overcloud) [stack@undercloud-0 ~]$ nova show new_instance |grep hyp
| OS-EXT-SRV-ATTR:hypervisor_hostname  | compute-1.localdomain



live migration also working as expected 
(overcloud) [stack@undercloud-0 ~]$ nova live-migration after_deploy compute-1.localdomain
(overcloud) [stack@undercloud-0 ~]$ nova list
+--------------------------------------+--------------+--------+------------+-------------+---------------------------------------+
| ID                                   | Name         | Status | Task State | Power State | Networks                              |
+--------------------------------------+--------------+--------+------------+-------------+---------------------------------------+
| e5058bab-335e-47a2-b644-fb8c653be3cf | after_deploy | ACTIVE | -          | Running     | tenantvxlan=192.168.32.11, 10.0.0.182 |
| 3d67dfb8-302e-498a-a092-9927d5a97299 | new_instance | ACTIVE | -          | Running     | tenantvxlan=192.168.32.10, 10.0.0.185 |
+--------------------------------------+--------------+--------+------------+-------------+---------------------------------------+
(overcloud) [stack@undercloud-0 ~]$ nova show after_deploy |grep hyper
| OS-EXT-SRV-ATTR:hypervisor_hostname  | compute-1.localdomain

Comment 10 Artem Hrechanychenko 2017-11-28 12:07:32 UTC
Instance was launch but cannot create and attach cinder volume. 

/var/log/cinder/cinder-api.log:2017-11-28 10:50:01.786 5936 DEBUG cinder.volume.api [req-10c1b98e-66ad-4dca-b990-e79cf5a3abf3 2cb5d5cc90e644cb965649f5ce42a637 1a2a0f8c85224dfeb5180fddca4062aa - default default] Task 'cinder.volume.flows.api.create_volume.EntryCreateTask;volume:create' (3d62a65d-a47b-4548-99c8-6b55d5f9c2a2) transitioned into state 'SUCCESS' from state 'RUNNING' with result '{'volume': Volume(_name_id=None,admin_metadata=<?>,attach_status='detached',availability_zone='nova',bootable=False,cluster=<?>,cluster_name=None,consistencygroup=<?>,consistencygroup_id=None,created_at=2017-11-28T10:50:01Z,deleted=False,deleted_at=None,display_description=None,display_name=None,ec2_id=None,encryption_key_id=None,glance_metadata=<?>,group=<?>,group_id=None,host=None,id=5345b78f-423c-4a79-bfb0-d3f2f4cdaa97,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id='1a2a0f8c85224dfeb5180fddca4062aa',provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status=None,scheduled_at=None,size=1,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=None,user_id='2cb5d5cc90e644cb965649f5ce42a637',volume_attachment=<?>,volume_type=<?>,volume_type_id=None), 'volume_properties': VolumeProperties(attach_status='detached',availability_zone='nova',cgsnapshot_id=None,consistencygroup_id=None,display_description=None,display_name=None,encryption_key_id=None,group_id=None,group_type_id=<?>,metadata={},multiattach=False,project_id='1a2a0f8c85224dfeb5180fddca4062aa',qos_specs=None,replication_status=<?>,reservations=['36e23726-3f60-4649-98cd-df242c4bc8b9','81a5f77d-43d5-4c58-b719-ac700f96270e'],size=1,snapshot_id=None,source_replicaid=None,source_volid=None,status='creating',user_id='2cb5d5cc90e644cb965649f5ce42a637',volume_type_id=None), 'volume_id': '5345b78f-423c-4a79-bfb0-d3f2f4cdaa97'}' _task_receiver /usr/lib/python2.7/site-packages/taskflow/listeners/logging.py:183


(overcloud) [stack@undercloud-0 ~]$ openstack volume show 5345b78f-423c-4a79-bfb0-d3f2f4cdaa97  
+--------------------------------+--------------------------------------+
| Field                          | Value                                |
+--------------------------------+--------------------------------------+
| attachments                    | []                                   |
| availability_zone              | nova                                 |
| bootable                       | false                                |
| consistencygroup_id            | None                                 |
| created_at                     | 2017-11-28T10:50:01.000000           |
| description                    | None                                 |
| encrypted                      | False                                |
| id                             | 5345b78f-423c-4a79-bfb0-d3f2f4cdaa97 |
| migration_status               | None                                 |
| multiattach                    | False                                |
| name                           | None                                 |
| os-vol-host-attr:host          | None                                 |
| os-vol-mig-status-attr:migstat | None                                 |
| os-vol-mig-status-attr:name_id | None                                 |
| os-vol-tenant-attr:tenant_id   | 1a2a0f8c85224dfeb5180fddca4062aa     |
| properties                     |                                      |
| replication_status             | None                                 |
| size                           | 1                                    |
| snapshot_id                    | None                                 |
| source_volid                   | None                                 |
| status                         | error                                |
| type                           | None                                 |
| updated_at                     | 2017-11-28T10:50:01.000000           |
| user_id                        | 2cb5d5cc90e644cb965649f5ce42a637     |
+--------------------------------+--------------------------------------+

(overcloud) [stack@undercloud-0 ~]$ openstack volume list
+--------------------------------------+------+--------+------+---------------------------------------+
| ID                                   | Name | Status | Size | Attached to                           |
+--------------------------------------+------+--------+------+---------------------------------------+
| 5345b78f-423c-4a79-bfb0-d3f2f4cdaa97 | None | error  |    1 |                                       |
| f08b4a96-fb66-4f4a-a0d8-7c034bd20b29 | None | error  |    1 |                                       |
| 22463cd4-a97e-480b-8b2b-b4bab85df973 | None | in-use |    1 | Attached to after_deploy on /dev/vdb  |
+--------------------------------------+------+--------+------+---------------------------------------+


[stack@undercloud-0 ~]$ source stackrc 
(undercloud) [stack@undercloud-0 ~]$ heat stack-list
WARNING (shell) "heat stack-list" is deprecated, please use "openstack stack list" instead
+--------------------------------------+------------+-----------------+----------------------+----------------------+----------------------------------+
| id                                   | stack_name | stack_status    | creation_time        | updated_time         | project                          |
+--------------------------------------+------------+-----------------+----------------------+----------------------+----------------------------------+
| 66c61d2e-79ca-4b7c-9888-d595f3d5c9d7 | overcloud  | UPDATE_COMPLETE | 2017-11-27T15:38:10Z | 2017-11-27T16:33:39Z | ba1fdf4e130746ab897197a48257a565 |
+--------------------------------------+------------+-----------------+----------------------+----------------------+----------------------------------+
(undercloud) [stack@undercloud-0 ~]$ nova list
+--------------------------------------+--------------+--------+------------+-------------+------------------------+
| ID                                   | Name         | Status | Task State | Power State | Networks               |
+--------------------------------------+--------------+--------+------------+-------------+------------------------+
| d6c3464d-2935-4ed0-ba5a-3fccf29f4620 | compute-0    | ACTIVE | -          | Running     | ctlplane=192.168.24.6  |
| f5c15a80-2663-4ddf-a0a4-484ea37cf0ec | compute-1    | ACTIVE | -          | Running     | ctlplane=192.168.24.15 |
| 2fceacc9-e004-4b5d-8d30-d0758fb771a9 | controller-0 | ACTIVE | -          | Running     | ctlplane=192.168.24.18 |
| 23047b29-0b57-4d44-a70b-dba0fa87d146 | controller-1 | ACTIVE | -          | Running     | ctlplane=192.168.24.14 |
| 4041ba13-fa00-49cb-b2b2-dbb075573038 | controller-2 | ACTIVE | -          | Running     | ctlplane=192.168.24.12 |
+--------------------------------------+--------------+--------+------------+-------------+------------------------+
(overcloud) [stack@undercloud-0 ~]$ nova list
+--------------------------------------+----------------+---------+------------+-------------+---------------------------------------+
| ID                                   | Name           | Status  | Task State | Power State | Networks                              |
+--------------------------------------+----------------+---------+------------+-------------+---------------------------------------+
| 3fde8ce0-c8c2-438c-81ba-6089ea55fc07 | after_deploy   | SHUTOFF | -          | NOSTATE     | tenantvxlan=192.168.32.14, 10.0.0.177 |
| 38029aac-ee44-4ffe-b765-b2ae6891eedc | after_reboot   | ACTIVE  | -          | Running     | tenantvxlan=192.168.32.6, 10.0.0.180  |
| 41e3c029-e5ac-4ba7-859b-701ae6688505 | after_reboot_2 | ACTIVE  | -          | Running     | tenantvxlan=192.168.32.12, 10.0.0.183 |
+--------------------------------------+----------------+---------+------------+-------------+---------------------------------------+
(overcloud) [stack@undercloud-0 ~]$ nova show after_reboot|grep hyp
| OS-EXT-SRV-ATTR:hypervisor_hostname  | compute-1.localdomain                                    |
(overcloud) [stack@undercloud-0 ~]$ nova show after_reboot_2|grep hyp
| OS-EXT-SRV-ATTR:hypervisor_hostname  | compute-0.localdomain                                    |
(overcloud) [stack@undercloud-0 ~]$ ping 10.0.0.180
PING 10.0.0.180 (10.0.0.180) 56(84) bytes of data.
64 bytes from 10.0.0.180: icmp_seq=1 ttl=63 time=3.08 ms
^C
--- 10.0.0.180 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 3.087/3.087/3.087/0.000 ms
(overcloud) [stack@undercloud-0 ~]$ ping 10.0.0.183
PING 10.0.0.183 (10.0.0.183) 56(84) bytes of data.
64 bytes from 10.0.0.183: icmp_seq=1 ttl=63 time=2.03 ms
^C
--- 10.0.0.183 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 2.031/2.031/2.031/0.000 ms

Comment 11 Artem Hrechanychenko 2017-11-28 13:09:14 UTC
VERIFIED

volume wasn't created because - https://bugzilla.redhat.com/show_bug.cgi?id=1412661

openstack-tripleo-heat-templates-7.0.3-13.el7ost.noarch

Comment 15 errata-xmlrpc 2017-12-13 22:22:16 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:3462


Note You need to log in before you can comment on or make changes to this bug.