Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1687278 - Swap volumes of vm failed: pivot of disk 'vdb' requires an active copy job
Summary: Swap volumes of vm failed: pivot of disk 'vdb' requires an active copy job
Keywords:
Status: NEW
Alias: None
Product: Virtualization Tools
Classification: Community
Component: libvirt
Version: unspecified
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Libvirt Maintainers
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-03-11 07:45 UTC by Ryan Liang
Modified: 2019-03-11 10:02 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:


Attachments (Terms of Use)
libvirtd.log (deleted)
2019-03-11 10:02 UTC, Ryan Liang
no flags Details

Description Ryan Liang 2019-03-11 07:45:59 UTC
Description of problem:
Failed to swap volumes of vm.
Nova reported below error:

Mar 08 01:34:42.984286 UnityiSCSISlave nova-compute[19437]: ERROR nova.virt.libvirt.driver [None req-1033d1c1-bd3b-4990-a8da-e503d25a041b tempest-TestMultiAttachVolumeSwap-890516006 tempest-TestMultiAttachVolumeSwap-890516006] Failure rebasing volume /dev/sdj on vdb.: libvirt.libvirtError: Requested operation is not valid: pivot of disk 'vdb' requires an active copy job
Mar 08 01:34:42.984521 UnityiSCSISlave nova-compute[19437]: ERROR nova.virt.libvirt.driver Traceback (most recent call last):
Mar 08 01:34:42.984846 UnityiSCSISlave nova-compute[19437]: ERROR nova.virt.libvirt.driver   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1557, in _swap_volume
Mar 08 01:34:42.985032 UnityiSCSISlave nova-compute[19437]: ERROR nova.virt.libvirt.driver     dev.abort_job(pivot=True)
Mar 08 01:34:42.985214 UnityiSCSISlave nova-compute[19437]: ERROR nova.virt.libvirt.driver   File "/opt/stack/new/nova/nova/virt/libvirt/guest.py", line 754, in abort_job
Mar 08 01:34:42.986537 UnityiSCSISlave nova-compute[19437]: ERROR nova.virt.libvirt.driver     self._guest._domain.blockJobAbort(self._disk, flags=flags)
Mar 08 01:34:42.986914 UnityiSCSISlave nova-compute[19437]: ERROR nova.virt.libvirt.driver   File "/usr/local/lib/python3.5/dist-packages/eventlet/tpool.py", line 190, in doit
Mar 08 01:34:42.987246 UnityiSCSISlave nova-compute[19437]: ERROR nova.virt.libvirt.driver     result = proxy_call(self._autowrap, f, *args, **kwargs)
Mar 08 01:34:42.987608 UnityiSCSISlave nova-compute[19437]: ERROR nova.virt.libvirt.driver   File "/usr/local/lib/python3.5/dist-packages/eventlet/tpool.py", line 148, in proxy_call
Mar 08 01:34:42.987958 UnityiSCSISlave nova-compute[19437]: ERROR nova.virt.libvirt.driver     rv = execute(f, *args, **kwargs)
Mar 08 01:34:42.988276 UnityiSCSISlave nova-compute[19437]: ERROR nova.virt.libvirt.driver   File "/usr/local/lib/python3.5/dist-packages/eventlet/tpool.py", line 129, in execute
Mar 08 01:34:42.988434 UnityiSCSISlave nova-compute[19437]: ERROR nova.virt.libvirt.driver     six.reraise(c, e, tb)
Mar 08 01:34:42.988591 UnityiSCSISlave nova-compute[19437]: ERROR nova.virt.libvirt.driver   File "/usr/local/lib/python3.5/dist-packages/six.py", line 693, in reraise
Mar 08 01:34:42.988746 UnityiSCSISlave nova-compute[19437]: ERROR nova.virt.libvirt.driver     raise value
Mar 08 01:34:42.988895 UnityiSCSISlave nova-compute[19437]: ERROR nova.virt.libvirt.driver   File "/usr/local/lib/python3.5/dist-packages/eventlet/tpool.py", line 83, in tworker
Mar 08 01:34:42.989045 UnityiSCSISlave nova-compute[19437]: ERROR nova.virt.libvirt.driver     rv = meth(*args, **kwargs)
Mar 08 01:34:42.989194 UnityiSCSISlave nova-compute[19437]: ERROR nova.virt.libvirt.driver   File "/usr/local/lib/python3.5/dist-packages/libvirt.py", line 772, in blockJobAbort
Mar 08 01:34:42.989366 UnityiSCSISlave nova-compute[19437]: ERROR nova.virt.libvirt.driver     if ret == -1: raise libvirtError ('virDomainBlockJobAbort() failed', dom=self)
Mar 08 01:34:42.989516 UnityiSCSISlave nova-compute[19437]: ERROR nova.virt.libvirt.driver libvirt.libvirtError: Requested operation is not valid: pivot of disk 'vdb' requires an active copy job


Version-Release number of selected component (if applicable):
OpenStack: master (Stein) devstack
libvirt-python: 5.1.0
libvirt-bin: 4.0.0-1ubuntu8.5~cloud0

How reproducible:

Steps to Reproduce:
1. In OpenStack, attach a volume, vol-1 to two vms, vm-1 and vm-2. Volume vol-1 is provisioned on external storage system.
2. Swap vol-1 of vm-1 to another volume, vol-2.


Actual results:
You can see above failure: Failure rebasing volume /dev/sdj (vol-2) on vdb.

Expected results:
vm-1's volume updated to vol-2.

Additional info:
If you need more info, please let me know.

Full n-cpu log locates: http://publiclogs.emc.com/51/633451/5/check/EMC_Unity_iSCSI/a2d0e05/EMC_Unity_iSCSI/1218/logs/screen-n-cpu.txt.gz
Search via request id: req-1033d1c1-bd3b-4990-a8da-e503d25a041b

Comment 1 Peter Krempa 2019-03-11 08:07:18 UTC
It is possible that the job has failed prior to being able to pivot it. Please attach libvirtd debug log from the time the copy job was started.

https://wiki.libvirt.org/page/DebugLogs

Comment 2 Ryan Liang 2019-03-11 10:02:31 UTC
Created attachment 1542824 [details]
libvirtd.log


Note You need to log in before you can comment on or make changes to this bug.