Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.

Bug 1600806

Summary: Prelive Migration failing with 'DestinationDiskExists: The supplied disk path exists'
Product: Red Hat OpenStack Reporter: Robin Cernin <rcernin>
Component: openstack-novaAssignee: nova-maint
Status: NEW --- QA Contact: nova-maint
Severity: medium Docs Contact:
Priority: medium    
Version: 10.0 (Newton)CC: dansmith, eglynn, jhakimra, kchamart, mschuppe, rbryant, sbauza, sgordon, srevivo, vromanso
Target Milestone: ---Keywords: Triaged, ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Robin Cernin 2018-07-13 05:22:25 UTC
-
This problem was already tracked upstream in https://bugs.launchpad.net/nova/+bug/1414895 but was never merged.
-

Description of problem:

Migration fails with detach failure:

[instance: b8759a7e-ed1c-446e-a57d-b3f3703dd5d9] Failed to detach volume ec1224d0-ef44-4c57-8d2c-8df1953b7ce1 from /dev/vdp

That leaves on the destination host:

[root@overcloud-compute-3 nova]# ls -la /var/lib/nova/instances/b8759a7e-ed1c-446e-a57d-b3f3703dd5d9
total 8
drwxr-xr-x.  2 nova nova   25 Jul  9 12:39 .
drwxr-xr-x. 55 nova nova 4096 Jul 13 11:58 ..
-rw-r--r--.  1 nova nova    0 Jul  9 12:39 console.log


Then all subsequent live-migrations fails with:

2018-07-13 11:58:42.577 140899 ERROR root [req-e91bd72e-885e-43f5-b3e9-b1af39b56a41 ada71b12fe64480b8fb565554a7a669a 0a9dbb2a6a7a41b1ab7aaea445ea9853 - - -] Original exception being dropped: ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 5284, in _do_live_migration\n    block_migration, disk, dest, migrate_data)\n', '  File "/usr/lib/python2.7/site-packages/nova/compute/rpcapi.py", line 716, in pre_live_migration\n    disk=disk, migrate_data=migrate_data)\n', '  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call\n    retry=self.retry)\n', '  File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 97, in _send\n    timeout=timeout, retry=retry)\n', '  File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send\n    retry=retry)\n', '  File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 575, in _send\n    raise result\n', 'DestinationDiskExists_Remote: The supplied disk path (/var/lib/nova/instances/b8759a7e-ed1c-446e-a57d-b3f3703dd5d9) already exists, it is expected not to exist.\nTraceback (most recent call last):\n\n  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming\n    res = self.dispatcher.dispatch(message)\n\n  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch\n    return self._do_dispatch(endpoint, method, ctxt, args)\n\n  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch\n    result = func(ctxt, **new_args)\n\n  File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 75, in wrapped\n    function_name, call_dict, binary)\n\n  File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__\n    self.force_reraise()\n\n  File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise\n    six.reraise(self.type_, self.value, self.tb)\n\n  File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 66, in wrapped\n    return f(self, context, *args, **kw)\n\n  File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 613, in decorated_function\n    return function(self, context, *args, **kwargs)\n\n  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 216, in decorated_function\n    kwargs[\'instance\'], e, sys.exc_info())\n\n  File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__\n    self.force_reraise()\n\n  File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise\n    six.reraise(self.type_, self.value, self.tb)\n\n  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 204, in decorated_function\n    return function(self, context, *args, **kwargs)\n\n  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 5229, in pre_live_migration\n    migrate_data)\n\n  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6603, in pre_live_migration\n    raise exception.DestinationDiskExists(path=instance_dir)\n\nDestinationDiskExists: The supplied disk path (/var/lib/nova/instances/b8759a7e-ed1c-446e-a57d-b3f3703dd5d9) already exists, it is expected not to exist.\n\n']

Version-Release number of selected component (if applicable):

openstack-nova-api-14.0.8-5.el7ost.noarch                   Thu Nov  9 01:12:52 2017
openstack-nova-cert-14.0.8-5.el7ost.noarch                  Thu Nov  9 01:12:53 2017
openstack-nova-common-14.0.8-5.el7ost.noarch                Thu Nov  9 01:09:57 2017
openstack-nova-compute-14.0.8-5.el7ost.noarch               Thu Nov  9 01:10:15 2017
openstack-nova-conductor-14.0.8-5.el7ost.noarch             Thu Nov  9 01:12:51 2017
openstack-nova-console-14.0.8-5.el7ost.noarch               Thu Nov  9 01:12:52 2017
openstack-nova-migration-14.0.8-5.el7ost.noarch             Thu Nov  9 01:12:45 2017
openstack-nova-novncproxy-14.0.8-5.el7ost.noarch            Thu Nov  9 01:12:52 2017
openstack-nova-scheduler-14.0.8-5.el7ost.noarch             Thu Nov  9 01:12:52 2017
puppet-nova-9.6.0-2.el7ost.noarch                           Thu Nov  9 01:19:32 2017
python-nova-14.0.8-5.el7ost.noarch                          Thu Nov  9 01:09:56 2017
python-novaclient-6.0.2-1.el7ost.noarch                     Thu Nov  9 00:52:12 2017

How reproducible:

live-migration with failed detach of the volume.

Actual results:

live-migration fails with time out waiting for message.

Expected results:

live-migration is successful