mbox series

[v3,0/5] qcow2: advanced compression options

Message ID 1571163625-642312-1-git-send-email-andrey.shinkevich@virtuozzo.com
Headers show
Series qcow2: advanced compression options | expand

Message

Andrey Shinkevich Oct. 15, 2019, 6:20 p.m. UTC
New enhancements for writing compressed data to QCOW2 image.

The preceding patches have been queued in the Max's block branch:

Based-on: <20190916175324.18478-1-vsementsov@virtuozzo.com>

v2:
    Instead of introducing multiple key options for many drivers, the
    'compression' option has been introduced on generic block layer
    as suggested by Roman Kagan. Discussed on the thread ID
    <1570026166-748566-1-git-send-email-andrey.shinkevich@virtuozzo.com>

Andrey Shinkevich (5):
  qcow2: Allow writing compressed data of multiple clusters
  tests/qemu-iotests: add case to write compressed data of multiple
    clusters
  block: support compressed write for copy-on-read
  block-stream: add compress option
  tests/qemu-iotests: add case for block-stream compress

 block.c                    |  12 ++++-
 block/io.c                 |  23 +++++++---
 block/qcow2.c              | 106 +++++++++++++++++++++++++++++++++------------
 block/qcow2.h              |   1 +
 block/stream.c             |  10 ++++-
 block/trace-events         |   2 +-
 blockdev.c                 |  16 ++++++-
 include/block/block.h      |   1 +
 include/block/block_int.h  |   2 +
 qapi/block-core.json       |   6 ++-
 qemu-options.hx            |   6 ++-
 tests/qemu-iotests/030     |  51 +++++++++++++++++++++-
 tests/qemu-iotests/030.out |   4 +-
 tests/qemu-iotests/214     |  35 +++++++++++++++
 tests/qemu-iotests/214.out |  15 +++++++
 15 files changed, 246 insertions(+), 44 deletions(-)

Comments

no-reply@patchew.org Oct. 16, 2019, 5:52 a.m. UTC | #1
Patchew URL: https://patchew.org/QEMU/1571163625-642312-1-git-send-email-andrey.shinkevich@virtuozzo.com/



Hi,

This series failed the docker-quick@centos7 build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

=== TEST SCRIPT BEGIN ===
#!/bin/bash
make docker-image-centos7 V=1 NETWORK=1
time make docker-test-quick@centos7 SHOW_ENV=1 J=14 NETWORK=1
=== TEST SCRIPT END ===

--> Processing Dependency: perl(File::Copy) for package: git-1.8.3.1-20.el7.x86_64
--> Processing Dependency: perl(File::Basename) for package: git-1.8.3.1-20.el7.x86_64
--> Processing Dependency: perl(Exporter) for package: git-1.8.3.1-20.el7.x86_64
--> Processing Dependency: perl(Error) for package: git-1.8.3.1-20.el7.x86_64
--> Processing Dependency: openssh-clients for package: git-1.8.3.1-20.el7.x86_64
--> Processing Dependency: less for package: git-1.8.3.1-20.el7.x86_64
--> Processing Dependency: /usr/bin/perl for package: git-1.8.3.1-20.el7.x86_64
---
--> Processing Dependency: perl(Filter::Util::Call) for package: 4:perl-5.16.3-294.el7_6.x86_64
--> Processing Dependency: libperl.so()(64bit) for package: 4:perl-5.16.3-294.el7_6.x86_64
---> Package perl-Carp.noarch 0:1.26-244.el7 will be installed
---> Package perl-Error.noarch 1:0.17020-2.el7 will be installed
---> Package perl-Exporter.noarch 0:5.68-3.el7 will be installed
---> Package perl-File-Path.noarch 0:2.09-2.el7 will be installed
---> Package perl-File-Temp.noarch 0:0.23.01-3.el7 will be installed
---
 perl-Carp                noarch 1.26-244.el7               base           19 k
 perl-Data-Dumper         x86_64 2.145-3.el7                base           47 k
 perl-Encode              x86_64 2.51-7.el7                 base          1.5 M
 perl-Error               noarch 1:0.17020-2.el7            base           32 k
 perl-Exporter            noarch 5.68-3.el7                 base           28 k
 perl-File-Path           noarch 2.09-2.el7                 base           26 k
 perl-File-Temp           noarch 0.23.01-3.el7              base           56 k
---
Total download size: 146 M
Downloading packages:
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
warning: /var/cache/yum/x86_64/7/epel/packages/ccache-3.3.4-1.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 352c64e5: NOKEY
Public key for ccache-3.3.4-1.el7.x86_64.rpm is not installed
warning: /var/cache/yum/x86_64/7/centos-virt-xen-48/packages/kernel-headers-4.9.188-35.el7.x86_64.rpm: Header V4 RSA/SHA1 Signature, key ID 61e8806c: NOKEY
Public key for kernel-headers-4.9.188-35.el7.x86_64.rpm is not installed
--------------------------------------------------------------------------------
Total                                              7.5 MB/s | 146 MB  00:19     
---
  Updating   : glibc-common-2.17-292.el7.x86_64                           2/315 
  Updating   : nss-softokn-freebl-3.44.0-5.el7.x86_64                     3/315 
  Updating   : glibc-2.17-292.el7.x86_64                                  4/315 
warning: /etc/nsswitch.conf created as /etc/nsswitch.conf.rpmnew
  Updating   : nspr-4.21.0-1.el7.x86_64                                   5/315 
  Updating   : nss-util-3.44.0-3.el7.x86_64                               6/315 
  Updating   : libstdc++-4.8.5-39.el7.x86_64                              7/315 
---
  Installing : 1:perl-Pod-Simple-3.28-4.el7.noarch                      118/315 
  Installing : perl-Getopt-Long-2.40-3.el7.noarch                       119/315 
  Installing : 4:perl-5.16.3-294.el7_6.x86_64                           120/315 
  Installing : 1:perl-Error-0.17020-2.el7.noarch                        121/315 
  Installing : perl-TermReadKey-2.30-20.el7.x86_64                      122/315 
  Installing : perl-Data-Dumper-2.145-3.el7.x86_64                      123/315 
  Installing : autoconf-2.69-11.el7.noarch                              124/315 
---
  Installing : python-rpm-macros-3-32.el7.noarch                        193/315 
  Installing : libffi-devel-3.0.13-18.el7.x86_64                        194/315 
install-info: No such file or directory for /usr/share/info/libffi.info.gz
warning: %post(libffi-devel-3.0.13-18.el7.x86_64) scriptlet failed, exit status 1
Non-fatal POSTIN scriptlet failure in rpm package libffi-devel-3.0.13-18.el7.x86_64
  Installing : xml-common-0.6.3-39.el7.noarch                           195/315 
  Installing : iso-codes-3.46-2.el7.noarch                              196/315 
---
  Verifying  : flac-libs-1.3.0-5.el7_1.x86_64                           261/315 
  Verifying  : opus-1.0.2-6.el7.x86_64                                  262/315 
  Verifying  : libuuid-2.23.2-61.el7.x86_64                             263/315 
  Verifying  : 1:perl-Error-0.17020-2.el7.noarch                        264/315 
  Verifying  : libXfixes-5.0.3-1.el7.x86_64                             265/315 
  Verifying  : xml-common-0.6.3-39.el7.noarch                           266/315 
  Verifying  : perl-Scalar-List-Utils-1.27-248.el7.x86_64               267/315 
---
  perl-Carp.noarch 0:1.26-244.el7                                               
  perl-Data-Dumper.x86_64 0:2.145-3.el7                                         
  perl-Encode.x86_64 0:2.51-7.el7                                               
  perl-Error.noarch 1:0.17020-2.el7                                             
  perl-Exporter.noarch 0:5.68-3.el7                                             
  perl-File-Path.noarch 0:2.09-2.el7                                            
  perl-File-Temp.noarch 0:0.23.01-3.el7                                         
---
libudev           no
default devices   yes

warning: Python 2 support is deprecated
warning: Python 3 will be required for building future versions of QEMU
cross containers  no

NOTE: guest cross-compilers enabled: cc
---
  CC      block/qed-check.o
  CC      block/vhdx.o
/tmp/qemu-test/src/block/qcow2.c: In function 'qcow2_co_pwritev_compressed_part':
/tmp/qemu-test/src/block/qcow2.c:4244:9: error: 'ret' may be used uninitialized in this function [-Werror=maybe-uninitialized]
     int ret;
         ^
cc1: all warnings being treated as errors
make: *** [block/qcow2.o] Error 1
make: *** Waiting for unfinished jobs....
Traceback (most recent call last):
  File "./tests/docker/docker.py", line 664, in <module>
---
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/subprocess.py", line 291, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', '--label', 'com.qemu.instance.uuid=dc4bae98efd811e99e7f68b59973b7d0', '-u', '1003', '--security-opt', 'seccomp=unconfined', '--rm', '-e', 'TARGET_LIST=', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 'SHOW_ENV=1', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/home/patchew2/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-hb38s10j/src/docker-src.2019-10-16-01.50.14.25013:/var/tmp/qemu:z,ro', 'qemu:centos7', '/var/tmp/qemu/run', 'test-quick']' returned non-zero exit status 2.
make[1]: *** [docker-run] Error 1
make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-hb38s10j/src'
make: *** [docker-run-test-quick@centos7] Error 2

real    2m26.033s
user    0m8.345s


The full log is available at
http://patchew.org/logs/1571163625-642312-1-git-send-email-andrey.shinkevich@virtuozzo.com/testing.docker-quick@centos7/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com
no-reply@patchew.org Oct. 16, 2019, 5:55 a.m. UTC | #2
Patchew URL: https://patchew.org/QEMU/1571163625-642312-1-git-send-email-andrey.shinkevich@virtuozzo.com/



Hi,

This series failed the docker-mingw@fedora build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

=== TEST SCRIPT BEGIN ===
#! /bin/bash
export ARCH=x86_64
make docker-image-fedora V=1 NETWORK=1
time make docker-test-mingw@fedora J=14 NETWORK=1
=== TEST SCRIPT END ===

  CC      block/blklogwrites.o
  CC      block/block-backend.o
/tmp/qemu-test/src/block/qcow2.c: In function 'qcow2_co_pwritev_compressed_part':
/tmp/qemu-test/src/block/qcow2.c:4244:9: error: 'ret' may be used uninitialized in this function [-Werror=maybe-uninitialized]
     int ret;
         ^~~
cc1: all warnings being treated as errors
make: *** [/tmp/qemu-test/src/rules.mak:69: block/qcow2.o] Error 1
make: *** Waiting for unfinished jobs....
Traceback (most recent call last):
  File "./tests/docker/docker.py", line 664, in <module>
---
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/subprocess.py", line 291, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', '--label', 'com.qemu.instance.uuid=4299392cefd911e9addb68b59973b7d0', '-u', '1003', '--security-opt', 'seccomp=unconfined', '--rm', '-e', 'TARGET_LIST=', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 'SHOW_ENV=', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/home/patchew2/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-r2c14at8/src/docker-src.2019-10-16-01.53.08.3890:/var/tmp/qemu:z,ro', 'qemu:fedora', '/var/tmp/qemu/run', 'test-mingw']' returned non-zero exit status 2.
make[1]: *** [docker-run] Error 1
make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-r2c14at8/src'
make: *** [docker-run-test-mingw@fedora] Error 2

real    2m50.343s
user    0m8.261s


The full log is available at
http://patchew.org/logs/1571163625-642312-1-git-send-email-andrey.shinkevich@virtuozzo.com/testing.docker-mingw@fedora/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com
no-reply@patchew.org Oct. 16, 2019, 11 a.m. UTC | #3
Patchew URL: https://patchew.org/QEMU/1571163625-642312-1-git-send-email-andrey.shinkevich@virtuozzo.com/



Hi,

This series failed the docker-quick@centos7 build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

=== TEST SCRIPT BEGIN ===
#!/bin/bash
make docker-image-centos7 V=1 NETWORK=1
time make docker-test-quick@centos7 SHOW_ENV=1 J=14 NETWORK=1
=== TEST SCRIPT END ===

--> Processing Dependency: perl(File::Copy) for package: git-1.8.3.1-20.el7.x86_64
--> Processing Dependency: perl(File::Basename) for package: git-1.8.3.1-20.el7.x86_64
--> Processing Dependency: perl(Exporter) for package: git-1.8.3.1-20.el7.x86_64
--> Processing Dependency: perl(Error) for package: git-1.8.3.1-20.el7.x86_64
--> Processing Dependency: openssh-clients for package: git-1.8.3.1-20.el7.x86_64
--> Processing Dependency: less for package: git-1.8.3.1-20.el7.x86_64
--> Processing Dependency: /usr/bin/perl for package: git-1.8.3.1-20.el7.x86_64
---
--> Processing Dependency: perl(Filter::Util::Call) for package: 4:perl-5.16.3-294.el7_6.x86_64
--> Processing Dependency: libperl.so()(64bit) for package: 4:perl-5.16.3-294.el7_6.x86_64
---> Package perl-Carp.noarch 0:1.26-244.el7 will be installed
---> Package perl-Error.noarch 1:0.17020-2.el7 will be installed
---> Package perl-Exporter.noarch 0:5.68-3.el7 will be installed
---> Package perl-File-Path.noarch 0:2.09-2.el7 will be installed
---> Package perl-File-Temp.noarch 0:0.23.01-3.el7 will be installed
---
 perl-Carp                noarch 1.26-244.el7               base           19 k
 perl-Data-Dumper         x86_64 2.145-3.el7                base           47 k
 perl-Encode              x86_64 2.51-7.el7                 base          1.5 M
 perl-Error               noarch 1:0.17020-2.el7            base           32 k
 perl-Exporter            noarch 5.68-3.el7                 base           28 k
 perl-File-Path           noarch 2.09-2.el7                 base           26 k
 perl-File-Temp           noarch 0.23.01-3.el7              base           56 k
---
Total download size: 146 M
Downloading packages:
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
warning: /var/cache/yum/x86_64/7/epel/packages/ccache-3.3.4-1.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 352c64e5: NOKEY
Public key for ccache-3.3.4-1.el7.x86_64.rpm is not installed
warning: /var/cache/yum/x86_64/7/centos-virt-xen-48/packages/kernel-headers-4.9.188-35.el7.x86_64.rpm: Header V4 RSA/SHA1 Signature, key ID 61e8806c: NOKEY
Public key for kernel-headers-4.9.188-35.el7.x86_64.rpm is not installed
--------------------------------------------------------------------------------
Total                                              8.3 MB/s | 146 MB  00:17     
---
  Updating   : glibc-common-2.17-292.el7.x86_64                           2/315 
  Updating   : nss-softokn-freebl-3.44.0-5.el7.x86_64                     3/315 
  Updating   : glibc-2.17-292.el7.x86_64                                  4/315 
warning: /etc/nsswitch.conf created as /etc/nsswitch.conf.rpmnew
  Updating   : nspr-4.21.0-1.el7.x86_64                                   5/315 
  Updating   : nss-util-3.44.0-3.el7.x86_64                               6/315 
  Updating   : libstdc++-4.8.5-39.el7.x86_64                              7/315 
---
  Installing : 1:perl-Pod-Simple-3.28-4.el7.noarch                      118/315 
  Installing : perl-Getopt-Long-2.40-3.el7.noarch                       119/315 
  Installing : 4:perl-5.16.3-294.el7_6.x86_64                           120/315 
  Installing : 1:perl-Error-0.17020-2.el7.noarch                        121/315 
  Installing : perl-TermReadKey-2.30-20.el7.x86_64                      122/315 
  Installing : perl-Data-Dumper-2.145-3.el7.x86_64                      123/315 
  Installing : autoconf-2.69-11.el7.noarch                              124/315 
---
  Installing : python-rpm-macros-3-32.el7.noarch                        193/315 
  Installing : libffi-devel-3.0.13-18.el7.x86_64                        194/315 
install-info: No such file or directory for /usr/share/info/libffi.info.gz
warning: %post(libffi-devel-3.0.13-18.el7.x86_64) scriptlet failed, exit status 1
Non-fatal POSTIN scriptlet failure in rpm package libffi-devel-3.0.13-18.el7.x86_64
  Installing : xml-common-0.6.3-39.el7.noarch                           195/315 
  Installing : iso-codes-3.46-2.el7.noarch                              196/315 
---
  Verifying  : flac-libs-1.3.0-5.el7_1.x86_64                           261/315 
  Verifying  : opus-1.0.2-6.el7.x86_64                                  262/315 
  Verifying  : libuuid-2.23.2-61.el7.x86_64                             263/315 
  Verifying  : 1:perl-Error-0.17020-2.el7.noarch                        264/315 
  Verifying  : libXfixes-5.0.3-1.el7.x86_64                             265/315 
  Verifying  : xml-common-0.6.3-39.el7.noarch                           266/315 
  Verifying  : perl-Scalar-List-Utils-1.27-248.el7.x86_64               267/315 
---
  perl-Carp.noarch 0:1.26-244.el7                                               
  perl-Data-Dumper.x86_64 0:2.145-3.el7                                         
  perl-Encode.x86_64 0:2.51-7.el7                                               
  perl-Error.noarch 1:0.17020-2.el7                                             
  perl-Exporter.noarch 0:5.68-3.el7                                             
  perl-File-Path.noarch 0:2.09-2.el7                                            
  perl-File-Temp.noarch 0:0.23.01-3.el7                                         
---
libudev           no
default devices   yes

warning: Python 2 support is deprecated
warning: Python 3 will be required for building future versions of QEMU
cross containers  no

NOTE: guest cross-compilers enabled: cc
---
  CC      block/accounting.o
  CC      block/dirty-bitmap.o
/tmp/qemu-test/src/block/qcow2.c: In function 'qcow2_co_pwritev_compressed_part':
/tmp/qemu-test/src/block/qcow2.c:4244:9: error: 'ret' may be used uninitialized in this function [-Werror=maybe-uninitialized]
     int ret;
         ^
cc1: all warnings being treated as errors
make: *** [block/qcow2.o] Error 1
make: *** Waiting for unfinished jobs....
Traceback (most recent call last):
  File "./tests/docker/docker.py", line 664, in <module>
---
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/subprocess.py", line 291, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', '--label', 'com.qemu.instance.uuid=f18e34eef00311e98c0768b59973b7d0', '-u', '1003', '--security-opt', 'seccomp=unconfined', '--rm', '-e', 'TARGET_LIST=', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 'SHOW_ENV=1', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/home/patchew2/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-9ddg5_qn/src/docker-src.2019-10-16-06.58.38.3939:/var/tmp/qemu:z,ro', 'qemu:centos7', '/var/tmp/qemu/run', 'test-quick']' returned non-zero exit status 2.
make[1]: *** [docker-run] Error 1
make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-9ddg5_qn/src'
make: *** [docker-run-test-quick@centos7] Error 2

real    2m4.544s
user    0m7.752s


The full log is available at
http://patchew.org/logs/1571163625-642312-1-git-send-email-andrey.shinkevich@virtuozzo.com/testing.docker-quick@centos7/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com
no-reply@patchew.org Oct. 16, 2019, 11:04 a.m. UTC | #4
Patchew URL: https://patchew.org/QEMU/1571163625-642312-1-git-send-email-andrey.shinkevich@virtuozzo.com/



Hi,

This series failed the docker-mingw@fedora build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

=== TEST SCRIPT BEGIN ===
#! /bin/bash
export ARCH=x86_64
make docker-image-fedora V=1 NETWORK=1
time make docker-test-mingw@fedora J=14 NETWORK=1
=== TEST SCRIPT END ===

  CC      block/snapshot.o
  CC      block/qapi.o
/tmp/qemu-test/src/block/qcow2.c: In function 'qcow2_co_pwritev_compressed_part':
/tmp/qemu-test/src/block/qcow2.c:4244:9: error: 'ret' may be used uninitialized in this function [-Werror=maybe-uninitialized]
     int ret;
         ^~~
cc1: all warnings being treated as errors
make: *** [/tmp/qemu-test/src/rules.mak:69: block/qcow2.o] Error 1
make: *** Waiting for unfinished jobs....
Traceback (most recent call last):
  File "./tests/docker/docker.py", line 664, in <module>
---
  File "/opt/rh/rh-python36/root/usr/lib64/python3.6/subprocess.py", line 291, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', '--label', 'com.qemu.instance.uuid=4c58e220f00411e9be7868b59973b7d0', '-u', '1001', '--security-opt', 'seccomp=unconfined', '--rm', '-e', 'TARGET_LIST=', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 'SHOW_ENV=', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/home/patchew/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-eeoq4u_0/src/docker-src.2019-10-16-07.01.11.11056:/var/tmp/qemu:z,ro', 'qemu:fedora', '/var/tmp/qemu/run', 'test-mingw']' returned non-zero exit status 2.
make[1]: *** [docker-run] Error 1
make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-eeoq4u_0/src'
make: *** [docker-run-test-mingw@fedora] Error 2

real    2m57.644s
user    0m8.518s


The full log is available at
http://patchew.org/logs/1571163625-642312-1-git-send-email-andrey.shinkevich@virtuozzo.com/testing.docker-mingw@fedora/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com