Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1510471 - tls migration failed
Summary: tls migration failed
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt
Version: 7.5
Hardware: x86_64
OS: Linux
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Libvirt Maintainers
QA Contact: yanqzhan@redhat.com
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-11-07 13:59 UTC by yanqzhan@redhat.com
Modified: 2017-11-09 07:44 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-11-09 07:44:06 UTC
Target Upstream Version:


Attachments (Terms of Use)
3.9_no_verify_libvirt.log (deleted)
2017-11-07 14:42 UTC, yanqzhan@redhat.com
no flags Details

Description yanqzhan@redhat.com 2017-11-07 13:59:48 UTC
Description of problem:
tls migration failed with "invalid argument" error after configured tls env.

Version-Release number of selected component (if applicable):
libvirt-3.9.0-1.el7.x86_64
qemu-kvm-rhev-2.10.0-4.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1.Config tls env:
server: ca-key.pem,ca-cert.pem,server-key.pem,server-cert.pem into /etc/pki/qemu, scp ca* to client
client: ca-key.pem, ca-cert.pem, client-key.pem, client-cert.pem

2.tls migrate a running domain:  (situation: Not set migrate_tls_x509_verify in qemu.conf)
#  virsh migrate V qemu+ssh://$server_host/system --live  --verbose --tls  
error: internal error: qemu unexpectedly closed the monitor: 2017-11-07T12:31:46.419193Z qemu-kvm: -chardev pty,id=charserial0: char device redirected to /dev/pts/3 (label charserial0)
2017-11-07T12:31:47.072760Z qemu-kvm: Not a migration stream
2017-11-07T12:31:47.073026Z qemu-kvm: load of migration failed: Invalid argument


Actual results:
tls migration failed

Expected results:
tls migration should succeed.

Additional info:
Not reproduced with libvirt-3.8.0-1.el7.x86_64.

Comment 2 yanqzhan@redhat.com 2017-11-07 14:42:12 UTC
Created attachment 1348997 [details]
3.9_no_verify_libvirt.log

In attachment "3.9_no_verify_libvirt.log":
Server is: dell-***
Client is: 10.66.*.*

From the log, seems different here:
# cat 3.9_no_verify_libvirt.log |grep verify-peer
2017-11-07 14:21:02.424+0000: 19270: debug : virJSONValueToString:1914 : result={"execute":"object-add","arguments":{"qom-type":"tls-creds-x509","id":"objlibvirt_migrate_tls0","props":{"dir":"/etc/pki/qemu","endpoint":"client","verify-peer":true}},"id":"libvirt-25"}
2017-11-07 14:21:02.424+0000: 19270: debug : qemuMonitorJSONCommandWithFd:298 : Send command '{"execute":"object-add","arguments":{"qom-type":"tls-creds-x509","id":"objlibvirt_migrate_tls0","props":{"dir":"/etc/pki/qemu","endpoint":"client","verify-peer":true}},"id":"libvirt-25"}' for write with FD -1
2017-11-07 14:21:02.424+0000: 19270: info : qemuMonitorSend:1064 : QEMU_MONITOR_SEND_MSG: mon=0x7f9dfc01e750 msg={"execute":"object-add","arguments":{"qom-type":"tls-creds-x509","id":"objlibvirt_migrate_tls0","props":{"dir":"/etc/pki/qemu","endpoint":"client","verify-peer":true}},"id":"libvirt-25"}
2017-11-07 14:21:02.424+0000: 19269: info : qemuMonitorIOWrite:544 : QEMU_MONITOR_IO_WRITE: mon=0x7f9dfc01e750 buf={"execute":"object-add","arguments":{"qom-type":"tls-creds-x509","id":"objlibvirt_migrate_tls0","props":{"dir":"/etc/pki/qemu","endpoint":"client","verify-peer":true}},"id":"libvirt-25"}

But for libvirt-3.8.0, this part is:
#  cat 3.8_no_verify_libvirt.log |grep verify-peer
2017-11-07 14:09:58.829+0000: 17581: debug : virJSONValueToString:1914 : result={"execute":"object-add","arguments":{"qom-type":"tls-creds-x509","id":"objlibvirt_migrate_tls0","props":{"dir":"/etc/pki/qemu","endpoint":"client","verify-peer":false}},"id":"libvirt-24"}
2017-11-07 14:09:58.829+0000: 17581: debug : qemuMonitorJSONCommandWithFd:298 : Send command '{"execute":"object-add","arguments":{"qom-type":"tls-creds-x509","id":"objlibvirt_migrate_tls0","props":{"dir":"/etc/pki/qemu","endpoint":"client","verify-peer":false}},"id":"libvirt-24"}' for write with FD -1
2017-11-07 14:09:58.829+0000: 17581: info : qemuMonitorSend:1063 : QEMU_MONITOR_SEND_MSG: mon=0x7f5f8c0120d0 msg={"execute":"object-add","arguments":{"qom-type":"tls-creds-x509","id":"objlibvirt_migrate_tls0","props":{"dir":"/etc/pki/qemu","endpoint":"client","verify-peer":false}},"id":"libvirt-24"}
2017-11-07 14:09:58.829+0000: 17579: info : qemuMonitorIOWrite:543 : QEMU_MONITOR_IO_WRITE: mon=0x7f5f8c0120d0 buf={"execute":"object-add","arguments":{"qom-type":"tls-creds-x509","id":"objlibvirt_migrate_tls0","props":{"dir":"/etc/pki/qemu","endpoint":"client","verify-peer":false}},"id":"libvirt-24"}

Comment 4 Peter Krempa 2017-11-07 15:11:37 UTC
Looks like this is the intended result of https://bugzilla.redhat.com/show_bug.cgi?id=1503658

Are you sure your TLS env is setup correctly? Especially that the client can be verified?

Is there anything interresting in the VM log file?

Comment 5 yanqzhan@redhat.com 2017-11-08 02:16:13 UTC
Hi Peter,

Yes, because when I downgrade client's libvirt to 3.8.0, the tls migration can succeed.

Comment 6 Jiri Denemark 2017-11-08 11:20:27 UTC
(In reply to yanqzhan@redhat.com from comment #5)
> Hi Peter,
> 
> Yes, because when I downgrade client's libvirt to 3.8.0, the tls migration
> can succeed.

No, that's not a sign you have the TLS environment set up correctly. There was a security bug in libvirt < 3.9.0 which caused libvirt to no verify the server's certificate (you can see it as "verify-peer":false) even though only the verification of a client's certificate was supposed to be turned off by default, the server certificate should always be verified (see CVE-2017-1000256).

Comment 7 Daniel Berrange 2017-11-08 12:41:26 UTC
Please attach the CA, server & client PEM certificates to this bug.

Comment 8 Daniel Berrange 2017-11-08 12:43:21 UTC
My guess is that the hostname used in the URI for virsh migrate doesn't match the URI used in the server-cert.pem file.  This mistake would never have been visible  in previous libvirt due to the CVE bug that turned off all verification.  Most other problems would have been detected by QEMU during object-add.

Comment 9 yanqzhan@redhat.com 2017-11-09 03:23:30 UTC
Hi, 

Yes, I re-configured the certificates and tested again, tls migration passed without "invalid argument" error. 

So I think the description in comment 0 indeed reflects the intended result of  BUG1503658(should always verify server), as my tests:

a) on libvirt-3.9.0, "invalid argument" server-cert will always make tls migration fail whether set client's ”migrate_x509_verify=1“ or not. It's verify-peer is always:"verify-peer":true.

b) on libvirt-3.8.0, "invalid argument" server-cert will make tls migration pass if client's migrate_x509_verify=1 was not set("verify-peer":false), and will fail when it was set("verify-peer":true).

And for comment 8, I guess similar to my following tests:

c) on libvirt-3.8.0, other errors can also be caught even when "verify-peer":false, such as "The certificate is not trusted", "Base64 unexpected header error" or "ASN1 parser: Error in DER parsing".
e.g.
error: internal error: unable to execute QEMU command 'object-add': Unable to import server certificate /etc/pki/qemu/server-cert.pem: Base64 unexpected header error.


Thanks for your introduction, we will extend our testing to cover these scenarios. And pls let me know if any of my understanding is not correct.


Note You need to log in before you can comment on or make changes to this bug.