mbox series

[v4,00/30] Add subcluster allocation to qcow2

Message ID cover.1584468723.git.berto@igalia.com
Headers show
Series Add subcluster allocation to qcow2 | expand

Message

Alberto Garcia March 17, 2020, 6:15 p.m. UTC 
Hi,

here's the new version of the patches to add subcluster allocation
support to qcow2.

Please refer to the cover letter of the first version for a full
description of the patches:

   https://lists.gnu.org/archive/html/qemu-block/2019-10/msg00983.html

I think that this version fixes all the problems pointed out by Max
and Eric during the review a couple of weeks ago. I also dropped the
RFC tag.

Berto

v4:
- Patch 01: New patch
- Patch 02: New patch
- Patch 05: Documentation updates [Eric]
- Patch 06: Fix rebase conflicts
- Patch 07: Change bit order in the subcluster allocation bitmap.
            Change incompatible bit number. [Max, Eric]
- Patch 09: Rename QCOW_MAX_SUBCLUSTERS_PER_CLUSTER to
            QCOW_EXTL2_SUBCLUSTERS_PER_CLUSTER [Eric]
- Patch 13: Change bit order in the subcluster allocation bitmap [Max, Eric]
            Add more documentation.
            Ignore the subcluster bitmap in the L2 entries of
            compressed clusters.
- Patch 14: New patch
- Patch 15: Update to work with the changes from patches 02 and 14.
- Patch 16: Update to work with the changes from patches 02 and 14.
- Patch 18: Update to work with the changes from patches 02 and 14.
            Update documentation.
            Fix return value on early exit.
- Patch 20: Make sure to clear the subcluster allocation bitmap when a
            cluster is unallocated.
- Patch 26: Update to work with the changes from patch 14.
- Patch 27: New patch [Max]
- Patch 28: Update version number, incompatible bit number and test
            expectations.
- Patch 30: Add new tests.
            Make the test verify its own results. [Max]

v3: https://lists.gnu.org/archive/html/qemu-block/2019-12/msg00587.html
- Patch 01: Rename host_offset to host_cluster_offset and make 'bytes'
            an unsigned int [Max]
- Patch 03: Rename cluster_needs_cow to cluster_needs_new_alloc and
            count_cow_clusters to count_single_write_clusters. Update
            documentation and add more assertions and checks [Max]
- Patch 09: Update qcow2_co_truncate() to properly support extended L2
            entries [Max]
- Patch 10: Forbid calling set_l2_bitmap() if the image does not have
            extended L2 entries [Max]
- Patch 11 (new): Add QCow2SubclusterType [Max]
- Patch 12 (new): Replace QCOW2_CLUSTER_* with QCOW2_SUBCLUSTER_*
- Patch 13 (new): Handle QCOW2_SUBCLUSTER_UNALLOCATED_ALLOC
- Patch 14: Use QCow2SubclusterType instead of QCow2ClusterType [Max]
- Patch 15: Use QCow2SubclusterType instead of QCow2ClusterType [Max]
- Patch 19: Don't call set_l2_bitmap() if the image does not have
            extended L2 entries [Max]
- Patch 21: Use smaller data types.
- Patch 22: Don't call set_l2_bitmap() if the image does not have
            extended L2 entries [Max]
- Patch 23: Use smaller data types.
- Patch 25: Update test results and documentation. Move the check for
            the minimum subcluster size to validate_cluster_size().
- Patch 26 (new): Add subcluster support to qcow2_measure()
- Patch 27: Add more tests

v2: https://lists.gnu.org/archive/html/qemu-block/2019-10/msg01642.html
- Patch 12: Update after the changes in 88f468e546.
- Patch 21 (new): Clear the L2 bitmap when allocating a compressed
  cluster. Compressed clusters should have the bitmap all set to 0.
- Patch 24: Document the new fields in the QAPI documentation [Eric].
- Patch 25: Allow qcow2 preallocation with backing files.
- Patch 26: Add some tests for qcow2 images with extended L2 entries.

v1: https://lists.gnu.org/archive/html/qemu-block/2019-10/msg00983.html

Output of git backport-diff against v3:

Key:
[----] : patches are identical
[####] : number of functional differences between upstream/downstream patch
[down] : patch is downstream-only
The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively

001/30:[down] 'qcow2: Make Qcow2AioTask store the full host offset'
002/30:[down] 'qcow2: Convert qcow2_get_cluster_offset() into qcow2_get_host_offset()'
003/30:[----] [-C] 'qcow2: Add calculate_l2_meta()'
004/30:[----] [--] 'qcow2: Split cluster_needs_cow() out of count_cow_clusters()'
005/30:[0020] [FC] 'qcow2: Process QCOW2_CLUSTER_ZERO_ALLOC clusters in handle_copied()'
006/30:[0010] [FC] 'qcow2: Add get_l2_entry() and set_l2_entry()'
007/30:[0020] [FC] 'qcow2: Document the Extended L2 Entries feature'
008/30:[----] [--] 'qcow2: Add dummy has_subclusters() function'
009/30:[0004] [FC] 'qcow2: Add subcluster-related fields to BDRVQcow2State'
010/30:[----] [--] 'qcow2: Add offset_to_sc_index()'
011/30:[----] [-C] 'qcow2: Add l2_entry_size()'
012/30:[----] [--] 'qcow2: Update get/set_l2_entry() and add get/set_l2_bitmap()'
013/30:[0046] [FC] 'qcow2: Add QCow2SubclusterType and qcow2_get_subcluster_type()'
014/30:[down] 'qcow2: Add cluster type parameter to qcow2_get_host_offset()'
015/30:[0082] [FC] 'qcow2: Replace QCOW2_CLUSTER_* with QCOW2_SUBCLUSTER_*'
016/30:[0002] [FC] 'qcow2: Handle QCOW2_SUBCLUSTER_UNALLOCATED_ALLOC'
017/30:[----] [-C] 'qcow2: Add subcluster support to calculate_l2_meta()'
018/30:[down] 'qcow2: Add subcluster support to qcow2_get_host_offset()'
019/30:[----] [--] 'qcow2: Add subcluster support to zero_in_l2_slice()'
020/30:[0012] [FC] 'qcow2: Add subcluster support to discard_in_l2_slice()'
021/30:[----] [--] 'qcow2: Add subcluster support to check_refcounts_l2()'
022/30:[----] [--] 'qcow2: Fix offset calculation in handle_dependencies()'
023/30:[----] [-C] 'qcow2: Update L2 bitmap in qcow2_alloc_cluster_link_l2()'
024/30:[----] [--] 'qcow2: Clear the L2 bitmap when allocating a compressed cluster'
025/30:[----] [--] 'qcow2: Add subcluster support to handle_alloc_space()'
026/30:[0006] [FC] 'qcow2: Restrict qcow2_co_pwrite_zeroes() to full clusters only'
027/30:[down] 'qcow2: Assert that expand_zero_clusters_in_l1() does not support subclusters'
028/30:[0019] [FC] 'qcow2: Add the 'extended_l2' option and the QCOW2_INCOMPAT_EXTL2 bit'
029/30:[----] [--] 'qcow2: Add subcluster support to qcow2_measure()'
030/30:[0313] [FC] 'iotests: Add tests for qcow2 images with extended L2 entries'

Alberto Garcia (30):
  qcow2: Make Qcow2AioTask store the full host offset
  qcow2: Convert qcow2_get_cluster_offset() into qcow2_get_host_offset()
  qcow2: Add calculate_l2_meta()
  qcow2: Split cluster_needs_cow() out of count_cow_clusters()
  qcow2: Process QCOW2_CLUSTER_ZERO_ALLOC clusters in handle_copied()
  qcow2: Add get_l2_entry() and set_l2_entry()
  qcow2: Document the Extended L2 Entries feature
  qcow2: Add dummy has_subclusters() function
  qcow2: Add subcluster-related fields to BDRVQcow2State
  qcow2: Add offset_to_sc_index()
  qcow2: Add l2_entry_size()
  qcow2: Update get/set_l2_entry() and add get/set_l2_bitmap()
  qcow2: Add QCow2SubclusterType and qcow2_get_subcluster_type()
  qcow2: Add cluster type parameter to qcow2_get_host_offset()
  qcow2: Replace QCOW2_CLUSTER_* with QCOW2_SUBCLUSTER_*
  qcow2: Handle QCOW2_SUBCLUSTER_UNALLOCATED_ALLOC
  qcow2: Add subcluster support to calculate_l2_meta()
  qcow2: Add subcluster support to qcow2_get_host_offset()
  qcow2: Add subcluster support to zero_in_l2_slice()
  qcow2: Add subcluster support to discard_in_l2_slice()
  qcow2: Add subcluster support to check_refcounts_l2()
  qcow2: Fix offset calculation in handle_dependencies()
  qcow2: Update L2 bitmap in qcow2_alloc_cluster_link_l2()
  qcow2: Clear the L2 bitmap when allocating a compressed cluster
  qcow2: Add subcluster support to handle_alloc_space()
  qcow2: Restrict qcow2_co_pwrite_zeroes() to full clusters only
  qcow2: Assert that expand_zero_clusters_in_l1() does not support
    subclusters
  qcow2: Add the 'extended_l2' option and the QCOW2_INCOMPAT_EXTL2 bit
  qcow2: Add subcluster support to qcow2_measure()
  iotests: Add tests for qcow2 images with extended L2 entries

 docs/interop/qcow2.txt           |  68 ++-
 docs/qcow2-cache.txt             |  19 +-
 qapi/block-core.json             |   7 +
 block/qcow2.h                    | 178 +++++++-
 include/block/block_int.h        |   1 +
 block/qcow2-cluster.c            | 696 ++++++++++++++++++++-----------
 block/qcow2-refcount.c           |  38 +-
 block/qcow2.c                    | 257 +++++++-----
 tests/qemu-iotests/031.out       |   8 +-
 tests/qemu-iotests/036.out       |   4 +-
 tests/qemu-iotests/049.out       | 102 ++---
 tests/qemu-iotests/060.out       |   1 +
 tests/qemu-iotests/061           |   6 +
 tests/qemu-iotests/061.out       |  25 +-
 tests/qemu-iotests/065           |  18 +-
 tests/qemu-iotests/082.out       |  48 ++-
 tests/qemu-iotests/085.out       |  38 +-
 tests/qemu-iotests/144.out       |   4 +-
 tests/qemu-iotests/182.out       |   2 +-
 tests/qemu-iotests/185.out       |   8 +-
 tests/qemu-iotests/198.out       |   2 +
 tests/qemu-iotests/206.out       |   4 +
 tests/qemu-iotests/242.out       |   5 +
 tests/qemu-iotests/255.out       |   8 +-
 tests/qemu-iotests/271           | 359 ++++++++++++++++
 tests/qemu-iotests/271.out       | 244 +++++++++++
 tests/qemu-iotests/280.out       |   2 +-
 tests/qemu-iotests/common.filter |   1 +
 tests/qemu-iotests/group         |   1 +
 29 files changed, 1682 insertions(+), 472 deletions(-)
 create mode 100755 tests/qemu-iotests/271
 create mode 100644 tests/qemu-iotests/271.out

Comments

Max Reitz April 8, 2020, 12:42 p.m. UTC | #1 
On 17.03.20 19:16, Alberto Garcia wrote:
> In order to support extended L2 entries some functions of the qcow2
> driver need to start dealing with subclusters instead of clusters.
> 
> qcow2_get_host_offset() is modified to return the subcluster type
> instead of the cluster type, and all callers are updated to replace
> all values of QCow2ClusterType with their QCow2SubclusterType
> equivalents.
> 
> This patch only changes the data types, there are no semantic changes.
> 
> Signed-off-by: Alberto Garcia <berto@igalia.com>
> ---
>  block/qcow2.h         |  2 +-
>  block/qcow2-cluster.c | 10 +++----
>  block/qcow2.c         | 70 ++++++++++++++++++++++---------------------
>  3 files changed, 42 insertions(+), 40 deletions(-)

Reviewed-by: Max Reitz <mreitz@redhat.com>
Vladimir Sementsov-Ogievskiy April 15, 2020, 7:10 a.m. UTC | #2 
17.03.2020 21:16, Alberto Garcia wrote:
> In order to support extended L2 entries some functions of the qcow2
> driver need to start dealing with subclusters instead of clusters.
> 
> qcow2_get_host_offset() is modified to return the subcluster type
> instead of the cluster type, and all callers are updated to replace
> all values of QCow2ClusterType with their QCow2SubclusterType
> equivalents.
> 
> This patch only changes the data types, there are no semantic changes.
> 
> Signed-off-by: Alberto Garcia<berto@igalia.com>

Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Derek Su April 21, 2020, 5:06 a.m. UTC | #3 
Hello,

This work is promising and interesting.
I'd like to try this new feature.
Could you please export a branch because the patches cannot be applied to current master?
Thanks.

Regards,
Derek


On 2020/3/18 上午2:15, Alberto Garcia wrote:
> Hi,
>
> here's the new version of the patches to add subcluster allocation
> support to qcow2.
>
> Please refer to the cover letter of the first version for a full
> description of the patches:
>
>     https://lists.gnu.org/archive/html/qemu-block/2019-10/msg00983.html
>
> I think that this version fixes all the problems pointed out by Max
> and Eric during the review a couple of weeks ago. I also dropped the
> RFC tag.
>
> Berto
>
> v4:
> - Patch 01: New patch
> - Patch 02: New patch
> - Patch 05: Documentation updates [Eric]
> - Patch 06: Fix rebase conflicts
> - Patch 07: Change bit order in the subcluster allocation bitmap.
>              Change incompatible bit number. [Max, Eric]
> - Patch 09: Rename QCOW_MAX_SUBCLUSTERS_PER_CLUSTER to
>              QCOW_EXTL2_SUBCLUSTERS_PER_CLUSTER [Eric]
> - Patch 13: Change bit order in the subcluster allocation bitmap [Max, Eric]
>              Add more documentation.
>              Ignore the subcluster bitmap in the L2 entries of
>              compressed clusters.
> - Patch 14: New patch
> - Patch 15: Update to work with the changes from patches 02 and 14.
> - Patch 16: Update to work with the changes from patches 02 and 14.
> - Patch 18: Update to work with the changes from patches 02 and 14.
>              Update documentation.
>              Fix return value on early exit.
> - Patch 20: Make sure to clear the subcluster allocation bitmap when a
>              cluster is unallocated.
> - Patch 26: Update to work with the changes from patch 14.
> - Patch 27: New patch [Max]
> - Patch 28: Update version number, incompatible bit number and test
>              expectations.
> - Patch 30: Add new tests.
>              Make the test verify its own results. [Max]
>
> v3: https://lists.gnu.org/archive/html/qemu-block/2019-12/msg00587.html
> - Patch 01: Rename host_offset to host_cluster_offset and make 'bytes'
>              an unsigned int [Max]
> - Patch 03: Rename cluster_needs_cow to cluster_needs_new_alloc and
>              count_cow_clusters to count_single_write_clusters. Update
>              documentation and add more assertions and checks [Max]
> - Patch 09: Update qcow2_co_truncate() to properly support extended L2
>              entries [Max]
> - Patch 10: Forbid calling set_l2_bitmap() if the image does not have
>              extended L2 entries [Max]
> - Patch 11 (new): Add QCow2SubclusterType [Max]
> - Patch 12 (new): Replace QCOW2_CLUSTER_* with QCOW2_SUBCLUSTER_*
> - Patch 13 (new): Handle QCOW2_SUBCLUSTER_UNALLOCATED_ALLOC
> - Patch 14: Use QCow2SubclusterType instead of QCow2ClusterType [Max]
> - Patch 15: Use QCow2SubclusterType instead of QCow2ClusterType [Max]
> - Patch 19: Don't call set_l2_bitmap() if the image does not have
>              extended L2 entries [Max]
> - Patch 21: Use smaller data types.
> - Patch 22: Don't call set_l2_bitmap() if the image does not have
>              extended L2 entries [Max]
> - Patch 23: Use smaller data types.
> - Patch 25: Update test results and documentation. Move the check for
>              the minimum subcluster size to validate_cluster_size().
> - Patch 26 (new): Add subcluster support to qcow2_measure()
> - Patch 27: Add more tests
>
> v2: https://lists.gnu.org/archive/html/qemu-block/2019-10/msg01642.html
> - Patch 12: Update after the changes in 88f468e546.
> - Patch 21 (new): Clear the L2 bitmap when allocating a compressed
>    cluster. Compressed clusters should have the bitmap all set to 0.
> - Patch 24: Document the new fields in the QAPI documentation [Eric].
> - Patch 25: Allow qcow2 preallocation with backing files.
> - Patch 26: Add some tests for qcow2 images with extended L2 entries.
>
> v1: https://lists.gnu.org/archive/html/qemu-block/2019-10/msg00983.html
>
> Output of git backport-diff against v3:
>
> Key:
> [----] : patches are identical
> [####] : number of functional differences between upstream/downstream patch
> [down] : patch is downstream-only
> The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively
>
> 001/30:[down] 'qcow2: Make Qcow2AioTask store the full host offset'
> 002/30:[down] 'qcow2: Convert qcow2_get_cluster_offset() into qcow2_get_host_offset()'
> 003/30:[----] [-C] 'qcow2: Add calculate_l2_meta()'
> 004/30:[----] [--] 'qcow2: Split cluster_needs_cow() out of count_cow_clusters()'
> 005/30:[0020] [FC] 'qcow2: Process QCOW2_CLUSTER_ZERO_ALLOC clusters in handle_copied()'
> 006/30:[0010] [FC] 'qcow2: Add get_l2_entry() and set_l2_entry()'
> 007/30:[0020] [FC] 'qcow2: Document the Extended L2 Entries feature'
> 008/30:[----] [--] 'qcow2: Add dummy has_subclusters() function'
> 009/30:[0004] [FC] 'qcow2: Add subcluster-related fields to BDRVQcow2State'
> 010/30:[----] [--] 'qcow2: Add offset_to_sc_index()'
> 011/30:[----] [-C] 'qcow2: Add l2_entry_size()'
> 012/30:[----] [--] 'qcow2: Update get/set_l2_entry() and add get/set_l2_bitmap()'
> 013/30:[0046] [FC] 'qcow2: Add QCow2SubclusterType and qcow2_get_subcluster_type()'
> 014/30:[down] 'qcow2: Add cluster type parameter to qcow2_get_host_offset()'
> 015/30:[0082] [FC] 'qcow2: Replace QCOW2_CLUSTER_* with QCOW2_SUBCLUSTER_*'
> 016/30:[0002] [FC] 'qcow2: Handle QCOW2_SUBCLUSTER_UNALLOCATED_ALLOC'
> 017/30:[----] [-C] 'qcow2: Add subcluster support to calculate_l2_meta()'
> 018/30:[down] 'qcow2: Add subcluster support to qcow2_get_host_offset()'
> 019/30:[----] [--] 'qcow2: Add subcluster support to zero_in_l2_slice()'
> 020/30:[0012] [FC] 'qcow2: Add subcluster support to discard_in_l2_slice()'
> 021/30:[----] [--] 'qcow2: Add subcluster support to check_refcounts_l2()'
> 022/30:[----] [--] 'qcow2: Fix offset calculation in handle_dependencies()'
> 023/30:[----] [-C] 'qcow2: Update L2 bitmap in qcow2_alloc_cluster_link_l2()'
> 024/30:[----] [--] 'qcow2: Clear the L2 bitmap when allocating a compressed cluster'
> 025/30:[----] [--] 'qcow2: Add subcluster support to handle_alloc_space()'
> 026/30:[0006] [FC] 'qcow2: Restrict qcow2_co_pwrite_zeroes() to full clusters only'
> 027/30:[down] 'qcow2: Assert that expand_zero_clusters_in_l1() does not support subclusters'
> 028/30:[0019] [FC] 'qcow2: Add the 'extended_l2' option and the QCOW2_INCOMPAT_EXTL2 bit'
> 029/30:[----] [--] 'qcow2: Add subcluster support to qcow2_measure()'
> 030/30:[0313] [FC] 'iotests: Add tests for qcow2 images with extended L2 entries'
>
> Alberto Garcia (30):
>    qcow2: Make Qcow2AioTask store the full host offset
>    qcow2: Convert qcow2_get_cluster_offset() into qcow2_get_host_offset()
>    qcow2: Add calculate_l2_meta()
>    qcow2: Split cluster_needs_cow() out of count_cow_clusters()
>    qcow2: Process QCOW2_CLUSTER_ZERO_ALLOC clusters in handle_copied()
>    qcow2: Add get_l2_entry() and set_l2_entry()
>    qcow2: Document the Extended L2 Entries feature
>    qcow2: Add dummy has_subclusters() function
>    qcow2: Add subcluster-related fields to BDRVQcow2State
>    qcow2: Add offset_to_sc_index()
>    qcow2: Add l2_entry_size()
>    qcow2: Update get/set_l2_entry() and add get/set_l2_bitmap()
>    qcow2: Add QCow2SubclusterType and qcow2_get_subcluster_type()
>    qcow2: Add cluster type parameter to qcow2_get_host_offset()
>    qcow2: Replace QCOW2_CLUSTER_* with QCOW2_SUBCLUSTER_*
>    qcow2: Handle QCOW2_SUBCLUSTER_UNALLOCATED_ALLOC
>    qcow2: Add subcluster support to calculate_l2_meta()
>    qcow2: Add subcluster support to qcow2_get_host_offset()
>    qcow2: Add subcluster support to zero_in_l2_slice()
>    qcow2: Add subcluster support to discard_in_l2_slice()
>    qcow2: Add subcluster support to check_refcounts_l2()
>    qcow2: Fix offset calculation in handle_dependencies()
>    qcow2: Update L2 bitmap in qcow2_alloc_cluster_link_l2()
>    qcow2: Clear the L2 bitmap when allocating a compressed cluster
>    qcow2: Add subcluster support to handle_alloc_space()
>    qcow2: Restrict qcow2_co_pwrite_zeroes() to full clusters only
>    qcow2: Assert that expand_zero_clusters_in_l1() does not support
>      subclusters
>    qcow2: Add the 'extended_l2' option and the QCOW2_INCOMPAT_EXTL2 bit
>    qcow2: Add subcluster support to qcow2_measure()
>    iotests: Add tests for qcow2 images with extended L2 entries
>
>   docs/interop/qcow2.txt           |  68 ++-
>   docs/qcow2-cache.txt             |  19 +-
>   qapi/block-core.json             |   7 +
>   block/qcow2.h                    | 178 +++++++-
>   include/block/block_int.h        |   1 +
>   block/qcow2-cluster.c            | 696 ++++++++++++++++++++-----------
>   block/qcow2-refcount.c           |  38 +-
>   block/qcow2.c                    | 257 +++++++-----
>   tests/qemu-iotests/031.out       |   8 +-
>   tests/qemu-iotests/036.out       |   4 +-
>   tests/qemu-iotests/049.out       | 102 ++---
>   tests/qemu-iotests/060.out       |   1 +
>   tests/qemu-iotests/061           |   6 +
>   tests/qemu-iotests/061.out       |  25 +-
>   tests/qemu-iotests/065           |  18 +-
>   tests/qemu-iotests/082.out       |  48 ++-
>   tests/qemu-iotests/085.out       |  38 +-
>   tests/qemu-iotests/144.out       |   4 +-
>   tests/qemu-iotests/182.out       |   2 +-
>   tests/qemu-iotests/185.out       |   8 +-
>   tests/qemu-iotests/198.out       |   2 +
>   tests/qemu-iotests/206.out       |   4 +
>   tests/qemu-iotests/242.out       |   5 +
>   tests/qemu-iotests/255.out       |   8 +-
>   tests/qemu-iotests/271           | 359 ++++++++++++++++
>   tests/qemu-iotests/271.out       | 244 +++++++++++
>   tests/qemu-iotests/280.out       |   2 +-
>   tests/qemu-iotests/common.filter |   1 +
>   tests/qemu-iotests/group         |   1 +
>   29 files changed, 1682 insertions(+), 472 deletions(-)
>   create mode 100755 tests/qemu-iotests/271
>   create mode 100644 tests/qemu-iotests/271.out
>
Alberto Garcia April 21, 2020, 10:35 a.m. UTC | #4 
On Tue 21 Apr 2020 07:06:42 AM CEST, Derek Su <dereksu@qnap.com> wrote:
> Hello,
>
> This work is promising and interesting.
> I'd like to try this new feature.
> Could you please export a branch because the patches cannot be applied
> to current master?

Hi,

you can apply v4 on top of 3189e9d38c82266ea5750a81255fd229c7ddf1e6

I also plan to publish v5 this week.

Any feedback is very much appreciated.

Thanks!

Berto