Note: This is a beta release of Red Hat Bugzilla 5.0. The data contained within is a snapshot of the live data so any changes you make will not be reflected in the production Bugzilla. Also email is disabled so feel free to test any aspect of the site that you want. File any problems you find or give feedback here.
Bug 1518738 - Add 'copy-on-read' filter driver for use with blockdev-add
Summary: Add 'copy-on-read' filter driver for use with blockdev-add
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm-rhev
Version: 7.4
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: rc
: ---
Assignee: Max Reitz
QA Contact: CongLi
URL:
Whiteboard:
Depends On: 1572856
Blocks: 760547 1519617 1527085 1558125
TreeView+ depends on / blocked
 
Reported: 2017-11-29 14:25 UTC by Peter Krempa
Modified: 2018-11-01 11:01 UTC (History)
14 users (show)

Fixed In Version: qemu-kvm-rhev-2.12.0-6.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-11-01 11:01:10 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Peter Krempa 2017-11-29 14:25:28 UTC
Description of problem:
Libvirt allows users to use the 'copy-on-read' feature, which currently does not have a suitable replacement when -blockdev is used. Neither -device nor any attributes of -blockdev accept it.

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Kevin Wolf 2017-11-30 16:57:21 UTC
This is going to be implemented as a filter driver rather than a blockdev-add option. But thanks for creating the BZ, this does need to be tracked.

Comment 4 Miroslav Rezanina 2018-06-25 14:03:20 UTC
Fix included in qemu-kvm-rhev-2.12.0-5.el7

Comment 11 Miroslav Rezanina 2018-07-01 03:11:26 UTC
Fix included in qemu-kvm-rhev-2.12.0-6.el7

Comment 12 CongLi 2018-07-04 06:02:49 UTC
Verified this bug on: qemu-kvm-rhev-2.12.0-6.el7.x86_64

Scenario 1:
1. copying data (a guest) from a remote address to a local image.
$ qemu-img create -f qcow2 local.qcow2 -b 'json: {"file.driver":"ssh", "file.host":"10.73.**.**","file.path":"/home/coli/rhel76-64-virtio-scsi.qcow2","file.host_key_check":"no" }'

2. $ qemu-img  map local.qcow2 (only have backing file info, no about local.qcow2)
0x4ffde0000     0x20000         0xd30000        json:{"host": "10.73.**.**", "host_key_check": "no", "driver": "ssh", "path": "/home/coli/rhel76-64-virtio-scsi.qcow2"}
0x4ffff0000     0x10000         0x70000         json:{"host": "10.73.**.**", "host_key_check": "no", "driver": "ssh", "path": "/home/coli/rhel76-64-virtio-scsi.qcow2"}

3. boot up the local.qcow2 which has a OS.
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=0x4 \
    -drive id=drive_image1,if=none,snapshot=off,aio=threads,cache=none,format=qcow2,file=/home/local.qcow2 \
    -device scsi-hd,id=image1,drive=drive_image1 \

4. $ qemu-img map local.qcow2 (have local.qcow2 info via copy-on-read)
0x100000        0x10000         0x26a0000       local.qcow2
0x9b50000       0x10000         0x26b0000       local.qcow2
0x202d0000      0x90000         0x280000        local.qcow2
0x20360000      0x60000         0x3c0000        local.qcow2


Scenario 2:
1. copying data from a remote address to a local image.
$ qemu-img create -f qcow2 local.qcow2 -b 'json: {"file.driver":"ssh", "file.host":"10.66.10.121","file.path":"/root/remote.img","file.host_key_check":"no" }'

2. $ qemu-img map local.qcow2 --output=json
[{ "start": 0, "length": 10737418240, "depth": 1, "zero": true, "data": false}]

3. read image for the 1st time
$  qemu-io --image-opts driver=copy-on-read,file.driver=qcow2,file.file.driver=file,file.file.filename=local.qcow2  -c 'read 0 1M'

4. $ qemu-img map local.qcow2 --output=json
[{ "start": 0, "length": 1048576, "depth": 0, "zero": true, "data": false},
{ "start": 1048576, "length": 10736369664, "depth": 1, "zero": true, "data": false}]

5. read image for the 2nd time
$ qemu-io --image-opts driver=copy-on-read,file.driver=qcow2,file.file.driver=file,file.file.filename=local.qcow2  -c 'read 0 1M'
read 1048576/1048576 bytes at offset 0
1 MiB, 1 ops; 0.0001 sec (5.034 GiB/sec and 5154.6392 ops/sec)


Scenario 3:
qemu-iotests 216:
./check 216 -qcow2
QEMU          -- "/usr/libexec/qemu-kvm" -nodefaults -machine accel=qtest
QEMU_IMG      -- "/usr/bin/qemu-img" 
QEMU_IO       -- "/usr/bin/qemu-io"  --cache writeback -f qcow2
QEMU_NBD      -- "/usr/bin/qemu-nbd" 
IMGFMT        -- qcow2 (compat=1.1)
IMGPROTO      -- file
PLATFORM      -- Linux/x86_64 hp-dl385pg8-04 3.10.0-862.6.1.el7.x86_64
TEST_DIR      -- /root/rpmbuild/BUILD/qemu-2.12.0/tests/qemu-iotests/scratch
SOCKET_SCM_HELPER -- 

216        
Passed all 1 tests


Thanks.

Comment 13 CongLi 2018-07-04 06:28:26 UTC
(In reply to CongLi from comment #12)
> Verified this bug on: qemu-kvm-rhev-2.12.0-6.el7.x86_64
>
> Scenario 2:
> 1. copying data from a remote address to a local image.
> $ qemu-img create -f qcow2 local.qcow2 -b 'json: {"file.driver":"ssh",
> "file.host":"10.66.10.121","file.path":"/root/remote.img","file.
> host_key_check":"no" }'
> 
> 2. $ qemu-img map local.qcow2 --output=json
> [{ "start": 0, "length": 10737418240, "depth": 1, "zero": true, "data":
> false}]
> 
> 3. read image for the 1st time
> $  qemu-io --image-opts
> driver=copy-on-read,file.driver=qcow2,file.file.driver=file,file.file.
> filename=local.qcow2  -c 'read 0 1M'

read 1048576/1048576 bytes at offset 0
1 MiB, 1 ops; 0.0029 sec (339.789 MiB/sec and 339.7893 ops/sec)

> 
> 4. $ qemu-img map local.qcow2 --output=json
> [{ "start": 0, "length": 1048576, "depth": 0, "zero": true, "data": false},
> { "start": 1048576, "length": 10736369664, "depth": 1, "zero": true, "data":
> false}]
> 
> 5. read image for the 2nd time
> $ qemu-io --image-opts
> driver=copy-on-read,file.driver=qcow2,file.file.driver=file,file.file.
> filename=local.qcow2  -c 'read 0 1M'
> read 1048576/1048576 bytes at offset 0
> 1 MiB, 1 ops; 0.0001 sec (5.034 GiB/sec and 5154.6392 ops/sec)

The 2nd time(0.0001 sec) read is faster than the 1st time(0.0029sec) which is as expected.

Comment 14 CongLi 2018-07-04 07:28:50 UTC
(In reply to CongLi from comment #12)
> Verified this bug on: qemu-kvm-rhev-2.12.0-6.el7.x86_64
> 
> Scenario 1:
> 1. copying data (a guest) from a remote address to a local image.
> $ qemu-img create -f qcow2 local.qcow2 -b 'json: {"file.driver":"ssh",
> "file.host":"10.73.**.**","file.path":"/home/coli/rhel76-64-virtio-scsi.
> qcow2","file.host_key_check":"no" }'
> 
> 2. $ qemu-img  map local.qcow2 (only have backing file info, no about
> local.qcow2)
> 0x4ffde0000     0x20000         0xd30000        json:{"host": "10.73.**.**",
> "host_key_check": "no", "driver": "ssh", "path":
> "/home/coli/rhel76-64-virtio-scsi.qcow2"}
> 0x4ffff0000     0x10000         0x70000         json:{"host": "10.73.**.**",
> "host_key_check": "no", "driver": "ssh", "path":
> "/home/coli/rhel76-64-virtio-scsi.qcow2"}
> 
> 3. boot up the local.qcow2 which has a OS.
>     -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=0x4 \
>     -drive
> id=drive_image1,if=none,snapshot=off,aio=threads,cache=none,format=qcow2,
> file=/home/local.qcow2 \
>     -device scsi-hd,id=image1,drive=drive_image1 \
> 
> 4. $ qemu-img map local.qcow2 (have local.qcow2 info via copy-on-read)
> 0x100000        0x10000         0x26a0000       local.qcow2
> 0x9b50000       0x10000         0x26b0000       local.qcow2
> 0x202d0000      0x90000         0x280000        local.qcow2
> 0x20360000      0x60000         0x3c0000        local.qcow2


Sorry, please ignore the invalid test scenario.

Comment 15 CongLi 2018-07-04 08:36:25 UTC
Hi Max,

QE would like to confirm if 'copy-on-read' is ready in -blockdev and blockdev-add now.

QE met some errors when test 'copy-on-read', could you please help confirm if the usage is wrong ?

1. -blockdev:
qemu-kvm: -blockdev driver=raw,cache.direct=off,cache.no-flush=on,file.filename=/root/test.raw,node-name=my_file,file.driver=file,copy-on-read=on: Parameter 'copy-on-read' is unexpected

2. blockdev-add:
{"execute": "blockdev-add","arguments": {"node-name": "drive2","driver": "raw", "file":    {"driver":"file", "filename": "/root/test.raw","copy-on-read":"on"}}}
{"error": {"class": "GenericError", "desc": "Parameter 'file.copy-on-read' is unexpected"}}


Thanks.

Comment 16 Max Reitz 2018-07-04 13:47:01 UTC
Hi,

The copy-on-read filter driver basically supersedes the copy-on-read option we had for -drive (which is not available for -blockdev or blockdev-add, as you can see).  It is instead a block driver, so you would use it like so:

1. -blockdev:
-blockdev node-name=my_file,driver=copy-on-read,file.driver=raw,file.file.driver=file,file.file.filename=/root/test.raw

2. blockdev-add:
{ "execute": "blockdev-add",
  "arguments": {
      "node-name": "drive2",
      "driver": "copy-on-read",
      "file": {
          "driver": "raw",
          "file": {
              "driver": "file",
              "filename": "/root/test.raw"
          }
      }
  } }

In practice, those test cases are not very useful, however, since copy-on-read is only useful with a backing file (which raw does not provide).

So comparable useful test cases would look like this:

Image preparation:
$ qemu-img create -f qcow2 backing.qcow2 64M
Formatting 'backing.qcow2', fmt=qcow2 size=67108864 cluster_size=65536 lazy_refcounts=off refcount_bits=16
$ qemu-img create -f qcow2 -b backing.qcow2 overlay.qcow2 64M
Formatting 'overlay.qcow2', fmt=qcow2 size=67108864 backing_file=backing.qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16
$ qemu-io -c 'write 0 64M' backing.qcow2

Check the overlay's allocation state:
$ qemu-img map overlay.qcow2 
Offset          Length          Mapped to       File
0               0x4000000       0x50000         backing.qcow2

(Note the filename, so the data is still in the backing file.)

1. -blockdev:
-blockdev "{'node-name':'node0','driver':'copy-on-read','file':{'driver':'qcow2','file':{'driver':'file','filename':'overlay.qcow2'}}}"

2. blockdev-add:
{'execute':'blockdev-add','arguments':{'node-name':'node0','driver':'copy-on-read','file':{'driver':'qcow2','file':{'driver':'file','filename':'overlay.qcow2'}}}}

In both cases, execute the following over QMP:
{'execute':'human-monitor-command','arguments':{'command-line':'qemu-io node0 "read 0 64M"'}}

Close the VM and check the overlay's allocation status again:
$ qemu-img map overlay.qcow2                  
Offset          Length          Mapped to       File
0               0x4000000       0x50000         overlay.qcow2

(The filename has changed, so the data is now allocated in the overlay.)

Max

Comment 17 CongLi 2018-07-05 12:41:51 UTC
(In reply to Max Reitz from comment #16)
> Hi,
> 
> The copy-on-read filter driver basically supersedes the copy-on-read option
> we had for -drive (which is not available for -blockdev or blockdev-add, as
> you can see).  It is instead a block driver, so you would use it like so:
> 
> 1. -blockdev:
> -blockdev
> node-name=my_file,driver=copy-on-read,file.driver=raw,file.file.driver=file,
> file.file.filename=/root/test.raw
> 
> 2. blockdev-add:
> { "execute": "blockdev-add",
>   "arguments": {
>       "node-name": "drive2",
>       "driver": "copy-on-read",
>       "file": {
>           "driver": "raw",
>           "file": {
>               "driver": "file",
>               "filename": "/root/test.raw"
>           }
>       }
>   } }

Thanks Max, it works.

> 
> In practice, those test cases are not very useful, however, since
> copy-on-read is only useful with a backing file (which raw does not provide).
> So comparable useful test cases would look like this:
> 
> Image preparation:
> $ qemu-img create -f qcow2 backing.qcow2 64M
> Formatting 'backing.qcow2', fmt=qcow2 size=67108864 cluster_size=65536
> lazy_refcounts=off refcount_bits=16
> $ qemu-img create -f qcow2 -b backing.qcow2 overlay.qcow2 64M
> Formatting 'overlay.qcow2', fmt=qcow2 size=67108864
> backing_file=backing.qcow2 cluster_size=65536 lazy_refcounts=off
> refcount_bits=16
> $ qemu-io -c 'write 0 64M' backing.qcow2
> 
> Check the overlay's allocation state:
> $ qemu-img map overlay.qcow2 
> Offset          Length          Mapped to       File
> 0               0x4000000       0x50000         backing.qcow2
> 
> (Note the filename, so the data is still in the backing file.)
> 
> 1. -blockdev:
> -blockdev
> "{'node-name':'node0','driver':'copy-on-read','file':{'driver':'qcow2',
> 'file':{'driver':'file','filename':'overlay.qcow2'}}}"
> 
> 2. blockdev-add:
> {'execute':'blockdev-add','arguments':{'node-name':'node0','driver':'copy-on-
> read','file':{'driver':'qcow2','file':{'driver':'file','filename':'overlay.
> qcow2'}}}}
> 
> In both cases, execute the following over QMP:
> {'execute':'human-monitor-command','arguments':{'command-line':'qemu-io
> node0 "read 0 64M"'}}
> 
> Close the VM and check the overlay's allocation status again:
> $ qemu-img map overlay.qcow2                  
> Offset          Length          Mapped to       File
> 0               0x4000000       0x50000         overlay.qcow2
> 
> (The filename has changed, so the data is now allocated in the overlay.)
> 
> Max


This scenario is useful and reasonable for the feature, it's helpful in the testing, I have added this scenario as a test case.


Tested the scenario above on qemu-kvm-rhev-2.12.0-7.el7.x86_64:
1. Image preparation:
$ qemu-img create -f qcow2 backing.qcow2 64
Formatting 'backing.qcow2', fmt=qcow2 size=67108864 cluster_size=65536 lazy_refcounts=off refcount_bits=16
$ qemu-img create -f qcow2 -b backing.qcow2 overlay.qcow2 64
Formatting 'overlay.qcow2', fmt=qcow2 size=67108864 backing_file=backing.qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16

2. Write sth into the backing file (0~64M)
$ qemu-io -c 'write 0 64M' backing.qcow2
wrote 67108864/67108864 bytes at offset 0
64 MiB, 1 ops; 0.1065 sec (600.392 MiB/sec and 9.3811 ops/sec)

3. Check the overlay's allocation state:
$ qemu-img map overlay.qcow2
Offset          Length          Mapped to       File
0               0x4000000       0x50000         backing.qcow2   --> here is 'File' name is backing.qcow2

4. Boot up guest with the overlay file with copy-on-read.
-blockdev driver=copy-on-read,file.file.driver=file,file.file.filename=/home/overlay.qcow2,node-name=node0,file.driver=qcow2 \
    -device scsi-hd,id=image2,drive=node0 \

5. Read the info written in 0~64M from overlay file via QMP:
{'execute':'human-monitor-command','arguments':{'command-line':'qemu-io node0 "read 0 64M"'}}
{"return": ""}

6. Shutdown the vm.
 
7. Check the overlay's allocation state again:
$ qemu-img map overlay.qcow2
Offset          Length          Mapped to       File
0               0x4000000       0x50000         overlay.qcow2   --> The filename has changed, so the data is now allocated in the overlay.
...

It also works well with blockdev-add.
{"execute":"blockdev-add","arguments":{"node-name":"node0","driver":"copy-on-read","file":{"driver":"qcow2","file":{"driver":"file","filename":"/home/overlay.qcow2"}}}}


Max, thank you again.

Comment 21 errata-xmlrpc 2018-11-01 11:01:10 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:3443


Note You need to log in before you can comment on or make changes to this bug.