Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.

Bug 1510471

Summary: tls migration failed
Product: Red Hat Enterprise Linux 7 Reporter: yanqzhan <yanqzhan>
Component: libvirtAssignee: Libvirt Maintainers <libvirt-maint>
Status: CLOSED NOTABUG QA Contact: yanqzhan <yanqzhan>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 7.5CC: berrange, dyuan, fjin, jdenemar, mzhan, pkrempa, rbalakri, xuzhang, yanqzhan, zpeng
Target Milestone: rcKeywords: Regression
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-11-09 07:44:06 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
Description Flags
3.9_no_verify_libvirt.log none

Description 2017-11-07 13:59:48 UTC
Description of problem:
tls migration failed with "invalid argument" error after configured tls env.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1.Config tls env:
server: ca-key.pem,ca-cert.pem,server-key.pem,server-cert.pem into /etc/pki/qemu, scp ca* to client
client: ca-key.pem, ca-cert.pem, client-key.pem, client-cert.pem

2.tls migrate a running domain:  (situation: Not set migrate_tls_x509_verify in qemu.conf)
#  virsh migrate V qemu+ssh://$server_host/system --live  --verbose --tls  
error: internal error: qemu unexpectedly closed the monitor: 2017-11-07T12:31:46.419193Z qemu-kvm: -chardev pty,id=charserial0: char device redirected to /dev/pts/3 (label charserial0)
2017-11-07T12:31:47.072760Z qemu-kvm: Not a migration stream
2017-11-07T12:31:47.073026Z qemu-kvm: load of migration failed: Invalid argument

Actual results:
tls migration failed

Expected results:
tls migration should succeed.

Additional info:
Not reproduced with libvirt-3.8.0-1.el7.x86_64.

Comment 2 2017-11-07 14:42:12 UTC
Created attachment 1348997 [details]

In attachment "3.9_no_verify_libvirt.log":
Server is: dell-***
Client is: 10.66.*.*

From the log, seems different here:
# cat 3.9_no_verify_libvirt.log |grep verify-peer
2017-11-07 14:21:02.424+0000: 19270: debug : virJSONValueToString:1914 : result={"execute":"object-add","arguments":{"qom-type":"tls-creds-x509","id":"objlibvirt_migrate_tls0","props":{"dir":"/etc/pki/qemu","endpoint":"client","verify-peer":true}},"id":"libvirt-25"}
2017-11-07 14:21:02.424+0000: 19270: debug : qemuMonitorJSONCommandWithFd:298 : Send command '{"execute":"object-add","arguments":{"qom-type":"tls-creds-x509","id":"objlibvirt_migrate_tls0","props":{"dir":"/etc/pki/qemu","endpoint":"client","verify-peer":true}},"id":"libvirt-25"}' for write with FD -1
2017-11-07 14:21:02.424+0000: 19270: info : qemuMonitorSend:1064 : QEMU_MONITOR_SEND_MSG: mon=0x7f9dfc01e750 msg={"execute":"object-add","arguments":{"qom-type":"tls-creds-x509","id":"objlibvirt_migrate_tls0","props":{"dir":"/etc/pki/qemu","endpoint":"client","verify-peer":true}},"id":"libvirt-25"}
2017-11-07 14:21:02.424+0000: 19269: info : qemuMonitorIOWrite:544 : QEMU_MONITOR_IO_WRITE: mon=0x7f9dfc01e750 buf={"execute":"object-add","arguments":{"qom-type":"tls-creds-x509","id":"objlibvirt_migrate_tls0","props":{"dir":"/etc/pki/qemu","endpoint":"client","verify-peer":true}},"id":"libvirt-25"}

But for libvirt-3.8.0, this part is:
#  cat 3.8_no_verify_libvirt.log |grep verify-peer
2017-11-07 14:09:58.829+0000: 17581: debug : virJSONValueToString:1914 : result={"execute":"object-add","arguments":{"qom-type":"tls-creds-x509","id":"objlibvirt_migrate_tls0","props":{"dir":"/etc/pki/qemu","endpoint":"client","verify-peer":false}},"id":"libvirt-24"}
2017-11-07 14:09:58.829+0000: 17581: debug : qemuMonitorJSONCommandWithFd:298 : Send command '{"execute":"object-add","arguments":{"qom-type":"tls-creds-x509","id":"objlibvirt_migrate_tls0","props":{"dir":"/etc/pki/qemu","endpoint":"client","verify-peer":false}},"id":"libvirt-24"}' for write with FD -1
2017-11-07 14:09:58.829+0000: 17581: info : qemuMonitorSend:1063 : QEMU_MONITOR_SEND_MSG: mon=0x7f5f8c0120d0 msg={"execute":"object-add","arguments":{"qom-type":"tls-creds-x509","id":"objlibvirt_migrate_tls0","props":{"dir":"/etc/pki/qemu","endpoint":"client","verify-peer":false}},"id":"libvirt-24"}
2017-11-07 14:09:58.829+0000: 17579: info : qemuMonitorIOWrite:543 : QEMU_MONITOR_IO_WRITE: mon=0x7f5f8c0120d0 buf={"execute":"object-add","arguments":{"qom-type":"tls-creds-x509","id":"objlibvirt_migrate_tls0","props":{"dir":"/etc/pki/qemu","endpoint":"client","verify-peer":false}},"id":"libvirt-24"}

Comment 4 Peter Krempa 2017-11-07 15:11:37 UTC
Looks like this is the intended result of

Are you sure your TLS env is setup correctly? Especially that the client can be verified?

Is there anything interresting in the VM log file?

Comment 5 2017-11-08 02:16:13 UTC
Hi Peter,

Yes, because when I downgrade client's libvirt to 3.8.0, the tls migration can succeed.

Comment 6 Jiri Denemark 2017-11-08 11:20:27 UTC
(In reply to from comment #5)
> Hi Peter,
> Yes, because when I downgrade client's libvirt to 3.8.0, the tls migration
> can succeed.

No, that's not a sign you have the TLS environment set up correctly. There was a security bug in libvirt < 3.9.0 which caused libvirt to no verify the server's certificate (you can see it as "verify-peer":false) even though only the verification of a client's certificate was supposed to be turned off by default, the server certificate should always be verified (see CVE-2017-1000256).

Comment 7 Daniel Berrange 2017-11-08 12:41:26 UTC
Please attach the CA, server & client PEM certificates to this bug.

Comment 8 Daniel Berrange 2017-11-08 12:43:21 UTC
My guess is that the hostname used in the URI for virsh migrate doesn't match the URI used in the server-cert.pem file.  This mistake would never have been visible  in previous libvirt due to the CVE bug that turned off all verification.  Most other problems would have been detected by QEMU during object-add.

Comment 9 2017-11-09 03:23:30 UTC

Yes, I re-configured the certificates and tested again, tls migration passed without "invalid argument" error. 

So I think the description in comment 0 indeed reflects the intended result of  BUG1503658(should always verify server), as my tests:

a) on libvirt-3.9.0, "invalid argument" server-cert will always make tls migration fail whether set client's ”migrate_x509_verify=1“ or not. It's verify-peer is always:"verify-peer":true.

b) on libvirt-3.8.0, "invalid argument" server-cert will make tls migration pass if client's migrate_x509_verify=1 was not set("verify-peer":false), and will fail when it was set("verify-peer":true).

And for comment 8, I guess similar to my following tests:

c) on libvirt-3.8.0, other errors can also be caught even when "verify-peer":false, such as "The certificate is not trusted", "Base64 unexpected header error" or "ASN1 parser: Error in DER parsing".
error: internal error: unable to execute QEMU command 'object-add': Unable to import server certificate /etc/pki/qemu/server-cert.pem: Base64 unexpected header error.

Thanks for your introduction, we will extend our testing to cover these scenarios. And pls let me know if any of my understanding is not correct.