| Message ID | 20230822114504.239505-10-david@redhat.com |
|---|---|
| State | Handled Elsewhere |
| Headers | show |
| Series | memory-backend-file related improvements and VM templating support | expand |
On Tue, Aug 22, 2023 at 01:44:57PM +0200, David Hildenbrand wrote: > Let's add some details about VM templating, focusing on the VM memory > configuration only. > > There is much more to VM templating (VM state? block devices?), but I leave > that as future work. Then there's the supposedly "unique" hardware identifiers, most notably VM UUID & NIC MAC addr that don't change if you create many VMs from a "template". Or from the guest OS there are "unique" things like /etc/machine-id, SSH host keys, web server certificates, etc. The vmgenid device at least provides a way for guest OS to get notified to update its unique resources/identifiers, but doesn't solve the overall VM UUID. NIC MAC addr could be solved by hotunplug+plug either side of creating the template & instantiating the template. > > Signed-off-by: David Hildenbrand <david@redhat.com> > --- > docs/vm-templating.txt | 109 +++++++++++++++++++++++++++++++++++++++++ Can you make this doument RST from the start and link to it from somewhere appropriate in our documentation. Perhaps it should live under the docs/system/ directory ? > 1 file changed, 109 insertions(+) > create mode 100644 docs/vm-templating.txt > > diff --git a/docs/vm-templating.txt b/docs/vm-templating.txt > new file mode 100644 > index 0000000000..419362c1ea > --- /dev/null > +++ b/docs/vm-templating.txt > @@ -0,0 +1,109 @@ > +QEMU VM templating > +================== > + > +This document explains how to use VM templating in QEMU. > + > +For now, the focus is on VM memory aspects, and not about how to save and > +restore other VM state (i.e., migrate-to-file with 'x-ignore-shared'). > + > +Overview > +-------- > + > +With VM templating, a single template VM serves as the starting point for > +new VMs. This allows for fast and efficient replication of VMs, resulting > +in fast startup times and reduced memory consumption. > + > +Conceptually, the VM state is frozen, to then be used as a basis for new > +VMs. The Copy-On-Write mechanism in the operating systems makes > +sure that new VMs are able to read template VM memory; however, any > +modifications stay private and don't modify the original template VM or any > +other created VM. I feel like we should have a paragraph at the top here explicitly calling out the dangers of templating, wrt to unique data in the hardware and guest OS. Don't have to provide solutions, just more of a scarcy "here be dragons" warning to users who might be tempted to try this. With regards, Daniel
On 22.08.23 15:47, Daniel P. Berrangé wrote: > On Tue, Aug 22, 2023 at 01:44:57PM +0200, David Hildenbrand wrote: >> Let's add some details about VM templating, focusing on the VM memory >> configuration only. >> >> There is much more to VM templating (VM state? block devices?), but I leave >> that as future work. > > Then there's the supposedly "unique" hardware identifiers, most notably > VM UUID & NIC MAC addr that don't change if you create many VMs from > a "template". Or from the guest OS there are "unique" things like > /etc/machine-id, SSH host keys, web server certificates, etc. > > The vmgenid device at least provides a way for guest OS to get notified > to update its unique resources/identifiers, but doesn't solve the overall > VM UUID. NIC MAC addr could be solved by hotunplug+plug either side of > creating the template & instantiating the template. > >> >> Signed-off-by: David Hildenbrand <david@redhat.com> >> --- >> docs/vm-templating.txt | 109 +++++++++++++++++++++++++++++++++++++++++ > > Can you make this doument RST from the start and link to it from > somewhere appropriate in our documentation. Perhaps it should live > under the docs/system/ directory ? I blindly did what memory-hotplug.txt and nvdimm.txt do. I can make it a RST and move under docs/system [+ link it in the index] > >> 1 file changed, 109 insertions(+) >> create mode 100644 docs/vm-templating.txt >> >> diff --git a/docs/vm-templating.txt b/docs/vm-templating.txt >> new file mode 100644 >> index 0000000000..419362c1ea >> --- /dev/null >> +++ b/docs/vm-templating.txt >> @@ -0,0 +1,109 @@ >> +QEMU VM templating >> +================== >> + >> +This document explains how to use VM templating in QEMU. >> + >> +For now, the focus is on VM memory aspects, and not about how to save and >> +restore other VM state (i.e., migrate-to-file with 'x-ignore-shared'). >> + >> +Overview >> +-------- >> + >> +With VM templating, a single template VM serves as the starting point for >> +new VMs. This allows for fast and efficient replication of VMs, resulting >> +in fast startup times and reduced memory consumption. >> + >> +Conceptually, the VM state is frozen, to then be used as a basis for new >> +VMs. The Copy-On-Write mechanism in the operating systems makes >> +sure that new VMs are able to read template VM memory; however, any >> +modifications stay private and don't modify the original template VM or any >> +other created VM. > > I feel like we should have a paragraph at the top here explicitly calling > out the dangers of templating, wrt to unique data in the hardware and guest > OS. Don't have to provide solutions, just more of a scarcy "here be dragons" > warning to users who might be tempted to try this. Agreed, I'll use some of your information above, thanks!
On Tue, 22 Aug 2023 at 12:49, David Hildenbrand <david@redhat.com> wrote: > > Let's add some details about VM templating, focusing on the VM memory > configuration only. > > There is much more to VM templating (VM state? block devices?), but I leave > that as future work. > > Signed-off-by: David Hildenbrand <david@redhat.com> > --- > docs/vm-templating.txt | 109 +++++++++++++++++++++++++++++++++++++++++ > 1 file changed, 109 insertions(+) > create mode 100644 docs/vm-templating.txt No new .txt files in docs/, please. Use rst, and incorporate the information into the correct parts of the manual structure. thanks -- PMM
On 22.08.23 16:23, Peter Maydell wrote: > On Tue, 22 Aug 2023 at 12:49, David Hildenbrand <david@redhat.com> wrote: >> >> Let's add some details about VM templating, focusing on the VM memory >> configuration only. >> >> There is much more to VM templating (VM state? block devices?), but I leave >> that as future work. >> >> Signed-off-by: David Hildenbrand <david@redhat.com> >> --- >> docs/vm-templating.txt | 109 +++++++++++++++++++++++++++++++++++++++++ >> 1 file changed, 109 insertions(+) >> create mode 100644 docs/vm-templating.txt > > No new .txt files in docs/, please. Use rst, and incorporate > the information into the correct parts of the manual structure. Thanks, already raised by Daniel. Will be an RST and moved under docs/system.
diff --git a/docs/vm-templating.txt b/docs/vm-templating.txt new file mode 100644 index 0000000000..419362c1ea --- /dev/null +++ b/docs/vm-templating.txt @@ -0,0 +1,109 @@ +QEMU VM templating +================== + +This document explains how to use VM templating in QEMU. + +For now, the focus is on VM memory aspects, and not about how to save and +restore other VM state (i.e., migrate-to-file with 'x-ignore-shared'). + +Overview +-------- + +With VM templating, a single template VM serves as the starting point for +new VMs. This allows for fast and efficient replication of VMs, resulting +in fast startup times and reduced memory consumption. + +Conceptually, the VM state is frozen, to then be used as a basis for new +VMs. The Copy-On-Write mechanism in the operating systems makes +sure that new VMs are able to read template VM memory; however, any +modifications stay private and don't modify the original template VM or any +other created VM. + +Memory configuration +-------------------- + +In order to create the template VM, we have to make sure that VM memory +ends up in a file, from where it can be reused for the new VMs: + +Supply VM RAM via memory-backend-file, with 'share=on' (modifications go +to the file) and 'readonly=off' (open the file writable). Note that +'readonly=off' is implicit. + +In the following command-line example, a 2GB VM is created, whereby VM RAM +is to be stored in the 'template' file. + + qemu [...] -m 2g \ + -object memory-backend-file,id=pc.ram,mem-path=template,size=2g,share=on,... \ + -machine q35,memory-backend=pc.ram', + +If multiple memory backends are used (vNUMA, DIMMs), configure all +memory backends accordingly. + +Once the VM is in the desired state, stop the VM and save other VM state, +leaving the current state of VM RAM reside in the file. + +In order to have a new VM be based on a template VM, we have to +configure VM RAM to be based on a template VM RAM file; however, the VM +should not be able to modify file content. + +Supply VM RAM via memory-backend-file, with 'share=off' (modifications stay +private), 'readonly=on' (open the file readonly) and 'rom=off' (don't make +the memory readonly for the VM). Note that 'share=off' is implicit and +that other VM state has to be restored separately. + +In the following command-line example, a 2GB VM is created based on the +existing 2GB file 'template'. + + qemu [...] -m 2g \ + -object memory-backend-file,id=pc.ram,mem-path=template,size=2g,readonly=on,rom=off,... \ + -machine q35,memory-backend=pc.ram', + +If multiple memory backends are used (vNUMA, DIMMs), configure all +memory backends accordingly. + +Note that '-mem-path' cannot be used for VM templating when creating the +template VM or when starting new VMs based on a template VM. + +Incompatible features +--------------------- + +Some features are incompatible with VM templating, as the underlying file +cannot be modified to discard VM RAM, or to actually share memory with +another process. + +vhost-user and multi-process QEMU +''''''''''''''''''''''''''''''''' + +vhost-user and multi-process QEMU are incompatible with VM templating. +These technologies rely on shared memory, however, the template VMs +don't actually share memory ('share=off'), even though they are file-based. + +virtio-balloon +'''''''''''''' + +virtio-balloon inflation and "free page reporting" cannot discard VM RAM +and will repeatedly report errors. While virtio-balloon can be used +for template VMs (e.g., report VM RAM stats), "free page reporting" +should be disabled and the balloon should not be inflated. + +virtio-mem +'''''''''' + +virtio-mem cannot discard VM RAM that is managed by the virtio-mem +device. virtio-mem will fail early when realizing the device. To use +VM templating with virtio-mem, either hotplug virtio-mem devices to the new +VM, or don't supply any memory to the template VM using virtio-mem +(requested-size=0), not using a template VM file as memory backend for the +virtio-mem device. + +VM migration +'''''''''''' + +For VM migration, "x-release-ram" similarly relies on discarding of VM +RAM on the migration source to free up migrated RAM, and will +repeatedly report errors. + +Postcopy live migration fails discarding VM RAM on the migration +destination early and refuses to activate postcopy live migration. Note +that postcopy live migration usually only works on selected filesystems +(shmem/tmpfs, hugetlbfs) either way.
Let's add some details about VM templating, focusing on the VM memory configuration only. There is much more to VM templating (VM state? block devices?), but I leave that as future work. Signed-off-by: David Hildenbrand <david@redhat.com> --- docs/vm-templating.txt | 109 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 109 insertions(+) create mode 100644 docs/vm-templating.txt