[0/5] Discrad blocks during block-stream operation

Hello everyone!

The given feature discards blocks with copy-on-read operation while the
streaming process runs. Adding the 'discard' argument to the QMP block-stream
command allows dropping a block in the backing chain after it has been copied
to the active layer. That will elude the block duplication in the intermediate
backing file. It saves the disk space while external snapshots are being
merged.
The method involves the filter insertion above the active layer to allow write
operation in the backing chain. The method is similar to that in the 'commit
active' command (mirror.c).
The permission to write into an inactive layer can not be obtained due to the
existing child permission mechanism. There is a commented up hack in the
callback function bdrv_stream_top_pwritev() in block/stream.c that redirects
write operations below the filter node. Being uncommented, it enables writing
into the inactive layer and passing all the iotests in the 030 file. Otherwise,
no WRITE permission is granted after the filter insertion above the target node.
Any suggestions to resolve that issue will be appreciated.

The suggestions of Dr. David Alan Gilbert and Alberto Garcia after their first
review have been applied.

Sincerely,

Andrey Shinkevich (5):
  Discard blocks while copy-on-read
  The discard flag for block stream operation
  iotests: allow resume_drive by node name
  iotests: prepare 030 for graph change
  iotests: 030 with block-stream discard

 block/stream.c                | 429 ++++++++++++++++++++++++++++++++++++++++--
 blockdev.c                    |   8 +-
 hmp-commands.hx               |   4 +-
 hmp.c                         |   4 +-
 include/block/block_int.h     |   2 +-
 qapi/block-core.json          |   5 +-
 tests/qemu-iotests/030        | 163 +++++++++++-----
 tests/qemu-iotests/030.out    |   4 +-
 tests/qemu-iotests/iotests.py |   9 +-
 9 files changed, 558 insertions(+), 70 deletions(-)

On Thu, Nov 22, 2018 at 21:48:02 +0300, Andrey Shinkevich wrote:
> Hello everyone!
> 
> The given feature discards blocks with copy-on-read operation while the
> streaming process runs. Adding the 'discard' argument to the QMP block-stream
> command allows dropping a block in the backing chain after it has been copied
> to the active layer. That will elude the block duplication in the intermediate
> backing file. It saves the disk space while external snapshots are being
> merged.

So you specifically want to merge the snapshot by pulling rather than
commiting? Do you have any specific reasons for that? I'm curious
because I plan to finally finish external snapshots in libvirt.

Allowing to pull into intermediate layers will be (or is?) very welcome
by libvirt since I plan to do external snapshot deletion/merging and
that will be greatly simplified by pulling.

On the other hand libvirt will not be able to always use 'discard' as
libvirt's API allows creating alternate histories for a VM and in such
case when merging a snapshot at a branching point we'll need to pull it
into multiple images. The 'discard' optimization can then be used only
with the last branch.

Libvirt's reasons for using 'block-stream' are mostly as it corresponds
to the operations necessary for not messing up the relationship between
the snapshot and which files on disk belong to it.

Dear Peter,

Thank you for your response.

The 'discard' option is useful when pulling a huge file. That will save 
a disk

space significantly.

The 'commit' operation would be useful as well but it is harder to implement

due to concurrent writes of a guest. Which command to use, 'pull' or 
'commit',

depends on an image to merge that can incur the performance issue.

Currently, the 'discard' option works for streaming into the active 
layer only.

There is a serious trouble with the child's WRITE permission while streaming

into an inactive layer. Once resolved, it will work right away with the 
current

version. So, the existent child permission mechanism is a headache we kindly

ask all of you to advise how to cope with it. Please refer to the 
descriptions and

comments in the patch files for details.

Kindly,

Andrey Shinkevich



On 23.11.2018 10:45, Peter Krempa wrote:
> On Thu, Nov 22, 2018 at 21:48:02 +0300, Andrey Shinkevich wrote:
>> Hello everyone!
>>
>> The given feature discards blocks with copy-on-read operation while the
>> streaming process runs. Adding the 'discard' argument to the QMP block-stream
>> command allows dropping a block in the backing chain after it has been copied
>> to the active layer. That will elude the block duplication in the intermediate
>> backing file. It saves the disk space while external snapshots are being
>> merged.
> So you specifically want to merge the snapshot by pulling rather than
> commiting? Do you have any specific reasons for that? I'm curious
> because I plan to finally finish external snapshots in libvirt.
>
> Allowing to pull into intermediate layers will be (or is?) very welcome
> by libvirt since I plan to do external snapshot deletion/merging and
> that will be greatly simplified by pulling.
>
> On the other hand libvirt will not be able to always use 'discard' as
> libvirt's API allows creating alternate histories for a VM and in such
> case when merging a snapshot at a branching point we'll need to pull it
> into multiple images. The 'discard' optimization can then be used only
> with the last branch.
>
> Libvirt's reasons for using 'block-stream' are mostly as it corresponds
> to the operations necessary for not messing up the relationship between
> the snapshot and which files on disk belong to it.