qemu/scripts
Maciej S. Szmigiero 0d9e8c0b67 Add Hyper-V Dynamic Memory Protocol driver (hv-balloon) base
This driver is like virtio-balloon on steroids: it allows both changing the
guest memory allocation via ballooning and (in the next patch) inserting
pieces of extra RAM into it on demand from a provided memory backend.

The actual resizing is done via ballooning interface (for example, via
the "balloon" HMP command).
This includes resizing the guest past its boot size - that is, hot-adding
additional memory in granularity limited only by the guest alignment
requirements, as provided by the next patch.

In contrast with ACPI DIMM hotplug where one can only request to unplug a
whole DIMM stick this driver allows removing memory from guest in single
page (4k) units via ballooning.

After a VM reboot the guest is back to its original (boot) size.

In the future, the guest boot memory size might be changed on reboot
instead, taking into account the effective size that VM had before that
reboot (much like Hyper-V does).

For performance reasons, the guest-released memory is tracked in a few
range trees, as a series of (start, count) ranges.
Each time a new page range is inserted into such tree its neighbors are
checked as candidates for possible merging with it.

Besides performance reasons, the Dynamic Memory protocol itself uses page
ranges as the data structure in its messages, so relevant pages need to be
merged into such ranges anyway.

One has to be careful when tracking the guest-released pages, since the
guest can maliciously report returning pages outside its current address
space, which later clash with the address range of newly added memory.
Similarly, the guest can report freeing the same page twice.

The above design results in much better ballooning performance than when
using virtio-balloon with the same guest: 230 GB / minute with this driver
versus 70 GB / minute with virtio-balloon.

During a ballooning operation most of time is spent waiting for the guest
to come up with newly freed page ranges, processing the received ranges on
the host side (in QEMU and KVM) is nearly instantaneous.

The unballoon operation is also pretty much instantaneous:
thanks to the merging of the ballooned out page ranges 200 GB of memory can
be returned to the guest in about 1 second.
With virtio-balloon this operation takes about 2.5 minutes.

These tests were done against a Windows Server 2019 guest running on a
Xeon E5-2699, after dirtying the whole memory inside guest before each
balloon operation.

Using a range tree instead of a bitmap to track the removed memory also
means that the solution scales well with the guest size: even a 1 TB range
takes just a few bytes of such metadata.

Since the required GTree operations aren't present in every Glib version
a check for them was added to the meson build script, together with new
"--enable-hv-balloon" and "--disable-hv-balloon" configure arguments.
If these GTree operations are missing in the system's Glib version this
driver will be skipped during QEMU build.

An optional "status-report=on" device parameter requests memory status
events from the guest (typically sent every second), which allow the host
to learn both the guest memory available and the guest memory in use
counts.

Following commits will add support for their external emission as
"HV_BALLOON_STATUS_REPORT" QMP events.

The driver is named hv-balloon since the Linux kernel client driver for
the Dynamic Memory Protocol is named as such and to follow the naming
pattern established by the virtio-balloon driver.
The whole protocol runs over Hyper-V VMBus.

The driver was tested against Windows Server 2012 R2, Windows Server 2016
and Windows Server 2019 guests and obeys the guest alignment requirements
reported to the host via DM_CAPABILITIES_REPORT message.

Acked-by: David Hildenbrand <david@redhat.com>
Signed-off-by: Maciej S. Szmigiero <maciej.szmigiero@oracle.com>
2023-11-06 14:08:10 +01:00
..
ci net: add initial support for AF_XDP network backend 2023-09-18 14:36:13 +08:00
coccinelle cleanup: Tweak and re-run return_directly.cocci 2022-12-14 16:19:35 +01:00
codeconverter scripts/: spelling fixes 2023-09-08 13:08:52 +03:00
coverage scripts/coverage: initial coverage comparison script 2023-04-04 15:16:29 +01:00
coverity-scan system: Rename softmmu/ directory as system/ 2023-10-08 21:08:08 +02:00
kvm target/i386: enumerate bit 56 of MSR_IA32_VMX_BASIC 2023-09-21 10:39:51 +02:00
modules scripts/modules/module_block: Use Python 3 interpreter & add pseudo-main 2020-05-31 13:56:46 +02:00
oss-fuzz system: Rename softmmu/ directory as system/ 2023-10-08 21:08:08 +02:00
performance scripts/: spelling fixes 2023-09-08 13:08:52 +03:00
qapi qapi: provide a friendly string representation of QAPI classes 2023-10-19 07:21:37 +02:00
qemu-guest-agent qemu-guest-agent: freeze-hook to ignore dpkg files as well 2018-08-23 18:46:25 +02:00
qemugdb scripts/gdb: implement 'qemu bt' 2021-01-12 12:38:03 +01:00
qmp python: rename qemu.aqmp to qemu.qmp 2022-04-21 11:01:00 -04:00
simplebench scripts/: spelling fixes 2023-09-08 13:08:52 +03:00
tracetool tracetool: avoid invalid escape in Python string 2023-10-17 15:20:53 +02:00
analyse-9p-simpletrace.py drop "from __future__ import print_function" 2020-02-07 15:15:16 +01:00
analyse-locks-simpletrace.py scripts/analyse-locks-simpletrace.py: changed iteritems() to items() 2023-09-26 12:28:51 -04:00
analyze-inclusions scripts/analyze-inclusions: drop qemu-common.h from analysis 2022-04-21 16:56:55 +04:00
analyze-migration.py migration: Fix analyze-migration read operation signedness 2023-10-17 09:14:32 +02:00
archive-source.sh make-release: do not ship dtc sources 2023-09-29 09:33:10 +02:00
block-coroutine-wrapper.py block-coroutine-wrapper: Add no_co_wrapper_bdrv_rdlock functions 2023-10-12 16:31:33 +02:00
check_sparse.py meson: move sparse detection to Meson and rewrite check_sparse.py 2020-10-04 18:36:23 +02:00
checkpatch.pl system: Rename softmmu/ directory as system/ 2023-10-08 21:08:08 +02:00
clean-header-guards.pl disas: Remove libvixl disassembler 2022-07-05 10:15:49 +02:00
clean-includes scripts/clean-includes: Improve --git commit message 2023-02-08 07:16:23 +01:00
cleanup-trace-events.pl scripts/cleanup-trace-events: Emit files in alphabetical order 2020-09-09 17:17:00 +01:00
cocci-macro-file.h compiler.h: replace QEMU_NORETURN with G_NORETURN 2022-04-21 17:03:51 +04:00
cpu-x86-uarch-abi.py python/qemu: rename command() to cmd() 2023-10-12 14:21:43 -04:00
decodetree.py tests/decode: Suppress "error: " string for expected-failure tests 2023-07-25 10:56:52 +01:00
device-crash-test python/qemu: rename command() to cmd() 2023-10-12 14:21:43 -04:00
disas-objdump.pl scripts: Switch to more portable Perl shebang 2017-05-10 10:19:24 +03:00
dump-guest-memory.py drop "from __future__ import print_function" 2020-02-07 15:15:16 +01:00
entitlement.sh scripts/entitlement.sh: Use backward-compatible cp flags 2021-11-30 22:25:58 +01:00
extract-vsssdk-headers
feature_to_c.py scripts: Mark feature_to_c.py as non-executable to fix a build issue 2023-10-17 15:20:53 +02:00
fix-multiline-comments.sh docs: move CODING_STYLE into the developer documentation 2021-02-24 11:05:21 +00:00
gensyscalls.sh linux-user: Add LoongArch syscall support 2022-07-04 11:08:57 +05:30
get_maintainer.pl scripts/get_maintainer.pl: don't print parentheses 2023-10-17 15:20:53 +02:00
git-submodule.sh scripts/git-submodule.sh: Don't rely on non-POSIX 'read' behaviour 2023-07-25 10:56:51 +01:00
git.orderfile scripts/git.orderfile: Display MAINTAINERS changes first 2023-01-13 16:22:57 +01:00
hxtool meson: generate hxtool files 2020-08-21 06:30:14 -04:00
kernel-doc scripts/kernel-doc: strip QEMU_ from function definitions 2021-03-24 14:24:40 +00:00
make-config-poison.sh exec/poison: Do not poison CONFIG_SOFTMMU 2023-06-05 12:04:29 -07:00
make-release make-release: do not ship dtc sources 2023-09-29 09:33:10 +02:00
meson-buildoptions.py meson-buildoptions: document the data at the top 2023-10-18 10:01:02 +02:00
meson-buildoptions.sh Add Hyper-V Dynamic Memory Protocol driver (hv-balloon) base 2023-11-06 14:08:10 +01:00
meson.build scripts: add a XML preprocessor script 2023-06-27 17:08:56 +02:00
minikconf.py meson: infrastructure for building emulators 2020-08-21 06:30:17 -04:00
modinfo-collect.py scripts: make sure scripts are invoked via $(PYTHON) 2023-05-19 20:40:29 +02:00
modinfo-generate.py scripts: make sure scripts are invoked via $(PYTHON) 2023-05-19 20:40:29 +02:00
mtest2make.py mtest2make.py: teach suite name that are just "PROJECT" 2023-04-20 11:17:34 +02:00
nsis.py scripts/nsis.py: Automatically package required DLLs of QEMU executables 2022-10-31 10:06:11 +01:00
probe-gdb-support.py scripts: make sure scripts are invoked via $(PYTHON) 2023-05-19 20:40:29 +02:00
python_qmp_updater.py scripts: add python_qmp_updater.py 2023-10-12 14:21:44 -04:00
qapi-gen.py qapi: move generator entrypoint into package 2020-10-10 11:37:47 +02:00
qemu-binfmt-conf.sh scripts/qemu-binfmt-conf: Add LoongArch to qemu_get_family() 2022-07-05 16:17:53 +05:30
qemu-gdb.py scripts/gdb: implement 'qemu bt' 2021-01-12 12:38:03 +01:00
qemu-stamp.py meson, configure: move --with-pkgversion, CONFIG_STAMP to meson 2022-05-07 07:46:58 +02:00
qemu-trace-stap qemu-trace-stap: changing SYSTEMTAP_TAPSET considered harmful. 2021-07-12 17:37:06 +01:00
qemu-version.sh build-sys: fix git version from -version 2020-10-12 11:50:23 -04:00
qom-cast-macro-clean-cocci-gen.py scripts: Add qom-cast-macro-clean-cocci-gen.py 2023-06-05 20:48:34 +02:00
refresh-pxe-roms.sh
render_block_graph.py python/qemu: rename command() to cmd() 2023-10-12 14:21:43 -04:00
replay-dump.py scripts/: spelling fixes 2023-09-08 13:08:52 +03:00
shaderinclude.py build-sys: fix crlf-ending C code 2023-02-02 10:44:23 +00:00
signrom.py drop "from __future__ import print_function" 2020-02-07 15:15:16 +01:00
simpletrace.py simpletrace: added simplified Analyzer2 class 2023-09-26 12:28:51 -04:00
symlink-install-tree.py Add information how to fix common build error on Windows in symlink-install-tree 2023-05-16 09:14:18 +02:00
tracetool.py tracetool: add output filename command-line argument 2021-01-04 14:24:58 +00:00
u2f-setup-gen.py scripts: Add u2f-setup-gen script 2020-08-31 08:23:39 +02:00
undefsym.py build: fix macOS --enable-modules build 2020-10-22 11:53:52 -04:00
update-linux-headers.sh scripts/update-linux-headers: Add iommufd.h 2023-10-18 10:10:49 +02:00
update-mips-syscall-args.sh linux-user, mips: update syscall-args-o32.c.inc to Linux v5.13 2021-07-13 13:59:59 +02:00
update-syscalltbl.sh linux-user, scripts: add a script to update syscall.tbl 2020-03-20 16:02:00 +01:00
userfaultfd-wrlat.py migration: introduce 'userfaultfd-wrlat.py' script 2021-02-08 11:19:51 +00:00
vmstate-static-checker.py vmstate-static-checker: Recognize "num" field 2023-04-27 10:18:26 +02:00
xen-detect.c xen: Drop support for Xen versions below 4.7.1 2023-06-07 15:07:10 +01:00
xml-preprocess-test.py scripts: add a XML preprocessor script 2023-06-27 17:08:56 +02:00
xml-preprocess.py scripts/xml-preprocess: Make sure this script is invoked via the right Python 2023-10-12 00:37:39 +03:00