qemu/tests
Daniel P. Berrange 409437e16d tests: introduce a framework for testing migration performance
This introduces a moderately general purpose framework for
testing performance of migration.

The initial guest workload is provided by the included 'stress'
program, which is configured to spawn one thread per guest CPU
and run a maximally memory intensive workload. It will loop
over GB of memory, xor'ing each byte with data from a 4k array
of random bytes. This ensures heavy read and write load across
all of guest memory to stress the migration performance. While
running the 'stress' program will record how long it takes to
xor each GB of memory and print this data for later reporting.

The test engine will spawn a pair of QEMU processes, either on
the same host, or with the target on a remote host via ssh,
using the host kernel and a custom initrd built with 'stress'
as the /init binary. Kernel command line args are set to ensure
a fast kernel boot time (< 1 second) between launching QEMU and
the stress program starting execution.

None the less, the test engine will initially wait N seconds for
the guest workload to stablize, before starting the migration
operation. When migration is running, the engine will use pause,
post-copy, autoconverge, xbzrle compression and multithread
compression features, as well as downtime & bandwidth tuning
to encourage completion. If migration completes, the test engine
will wait N seconds again for the guest workooad to stablize on
the target host. If migration does not complete after a preset
number of iterations, it will be aborted.

While the QEMU process is running on the source host, the test
engine will sample the host CPU usage of QEMU as a whole, and
each vCPU thread. While migration is running, it will record
all the stats reported by 'query-migration'. Finally, it will
capture the output of the stress program running in the guest.

All the data produced from a single test execution is recorded
in a structured JSON file. A separate program is then able to
create interactive charts using the "plotly" python + javascript
libraries, showing the characteristics of the migration.

The data output provides visualization of the effect on guest
vCPU workloads from the migration process, the corresponding
vCPU utilization on the host, and the overall CPU hit from
QEMU on the host. This is correlated from statistics from the
migration process, such as downtime, vCPU throttling and iteration
number.

While the tests can be run individually with arbitrary parameters,
there is also a facility for producing batch reports for a number
of pre-defined scenarios / comparisons, in order to be able to
get standardized results across different hardware configurations
(eg TCP vs RDMA, or comparing different VCPU counts / memory
sizes, etc).

To use this, first you must build the initrd image

 $ make tests/migration/initrd-stress.img

To run a a one-shot test with all default parameters

 $ ./tests/migration/guestperf.py > result.json

This has many command line args for varying its behaviour.
For example, to increase the RAM size and CPU count and
bind it to specific host NUMA nodes

 $ ./tests/migration/guestperf.py \
       --mem 4 --cpus 2 \
       --src-mem-bind 0 --src-cpu-bind 0,1 \
       --dst-mem-bind 1 --dst-cpu-bind 2,3 \
       > result.json

Using mem + cpu binding is strongly recommended on NUMA
machines, otherwise the guest performance results will
vary wildly between runs of the test due to lucky/unlucky
NUMA placement, making sensible data analysis impossible.

To make it run across separate hosts:

 $ ./tests/migration/guestperf.py \
       --dst-host somehostname > result.json

To request that post-copy is enabled, with switchover
after 5 iterations

 $ ./tests/migration/guestperf.py \
       --post-copy --post-copy-iters 5 > result.json

Once a result.json file is created, a graph of the data
can be generated, showing guest workload performance per
thread and the migration iteration points:

 $ ./tests/migration/guestperf-plot.py --output result.html \
        --migration-iters --split-guest-cpu result.json

To further include host vCPU utilization and overall QEMU
utilization

 $ ./tests/migration/guestperf-plot.py --output result.html \
        --migration-iters --split-guest-cpu \
	--qemu-cpu --vcpu-cpu result.json

NB, the 'guestperf-plot.py' command requires that you have
the plotly python library installed. eg you must do

 $ pip install --user  plotly

Viewing the result.html file requires that you have the
plotly.min.js file in the same directory as the HTML
output. This js file is installed as part of the plotly
python library, so can be found in

  $HOME/.local/lib/python2.7/site-packages/plotly/offline/plotly.min.js

The guestperf-plot.py program can accept multiple json files
to plot, enabling results from different configurations to
be compared.

Finally, to run the entire standardized set of comparisons

  $ ./tests/migration/guestperf-batch.py \
       --dst-host somehost \
       --mem 4 --cpus 2 \
       --src-mem-bind 0 --src-cpu-bind 0,1 \
       --dst-mem-bind 1 --dst-cpu-bind 2,3
       --output tcp-somehost-4gb-2cpu

will store JSON files from all scenarios in the directory
named tcp-somehost-4gb-2cpu

Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
Message-Id: <1469020993-29426-7-git-send-email-berrange@redhat.com>
Signed-off-by: Amit Shah <amit.shah@redhat.com>
2016-07-22 13:23:39 +05:30
..
acpi-test-data tests: add APIC.cphp and DSDT.cphp blobs 2016-07-04 19:43:33 +03:00
docker docker: pass EXECUTABLE to build script 2016-07-20 19:19:43 +08:00
guest-debug tests/guest-debug: introduce basic gdbstub tests 2015-12-17 13:37:15 +00:00
image-fuzzer tests: Use Python 2.6 "except E as ..." syntax 2016-02-08 17:29:54 +01:00
libqos Clean up ill-advised or unusual header guards 2016-07-12 16:20:46 +02:00
migration tests: introduce a framework for testing migration performance 2016-07-22 13:23:39 +05:30
multiboot tests/multiboot: Add test for modules 2015-01-26 12:22:44 +01:00
qapi-schema qapi: Implement boxed types for commands/events 2016-07-19 13:21:08 +02:00
qemu-iotests scripts: refactor the VM class in iotests for reuse 2016-07-22 13:23:24 +05:30
rocker rocker: tests: don't need to specify master/self when setting vlans 2015-07-07 13:13:22 +01:00
tcg Use #include "..." for our own headers, <...> for others 2016-07-12 16:19:16 +02:00
vmstate-static-checker-data tests: vmstate static checker: add size mismatch inside substructure 2014-06-23 19:14:52 +02:00
.gitignore qapi: Add new clone visitor 2016-07-06 10:52:04 +02:00
ac97-test.c tests: Remove unnecessary glib.h includes 2016-06-07 18:19:24 +03:00
ahci-test.c tests: in IDE and AHCI tests perform DMA write before flushing 2016-07-18 18:19:01 -04:00
bios-tables-test.c tests: acpi: add CPU hotplug testcase 2016-07-04 16:49:34 +03:00
boot-order-test.c tests: Remove unnecessary glib.h includes 2016-06-07 18:19:24 +03:00
boot-sector.c all: Clean up includes 2016-02-23 12:43:05 +00:00
boot-sector.h Clean up ill-advised or unusual header guards 2016-07-12 16:20:46 +02:00
check-block.sh qemu-iotests: make check-block.sh work on out-of-tree builds 2016-01-07 21:30:16 +01:00
check-qdict.c tests: Remove unnecessary glib.h includes 2016-06-07 18:19:24 +03:00
check-qfloat.c tests: Remove unnecessary glib.h includes 2016-06-07 18:19:24 +03:00
check-qint.c tests: Remove unnecessary glib.h includes 2016-06-07 18:19:24 +03:00
check-qjson.c qapi: Improve use of qmp/types.h 2016-07-06 10:52:03 +02:00
check-qlist.c tests: Remove unnecessary glib.h includes 2016-06-07 18:19:24 +03:00
check-qnull.c qapi: Add new visit_complete() function 2016-07-06 10:52:04 +02:00
check-qom-interface.c tests: Remove unnecessary glib.h includes 2016-06-07 18:19:24 +03:00
check-qom-proplist.c tests: Remove unnecessary glib.h includes 2016-06-07 18:19:24 +03:00
check-qstring.c tests: Remove unnecessary glib.h includes 2016-06-07 18:19:24 +03:00
crypto-tls-x509-helpers.c tests: Clean up includes 2016-02-16 14:29:27 +00:00
crypto-tls-x509-helpers.h crypto: add sanity checking of TLS x509 credentials 2015-09-15 15:05:09 +01:00
device-introspect-test.c tests: Remove unnecessary glib.h includes 2016-06-07 18:19:24 +03:00
display-vga-test.c coccinelle: Remove unnecessary variables for function return value 2016-06-20 16:38:13 +02:00
drive_del-test.c tests: Remove unnecessary glib.h includes 2016-06-07 18:19:24 +03:00
ds1338-test.c tests: Remove unnecessary glib.h includes 2016-06-07 18:19:24 +03:00
e1000-test.c tests: Remove unnecessary glib.h includes 2016-06-07 18:19:24 +03:00
e1000e-test.c clean-includes: run it once more 2016-06-16 18:39:03 +02:00
eepro100-test.c tests: Remove unnecessary glib.h includes 2016-06-07 18:19:24 +03:00
endianness-test.c coccinelle: Remove unnecessary variables for function return value 2016-06-20 16:38:13 +02:00
es1370-test.c tests: Remove unnecessary glib.h includes 2016-06-07 18:19:24 +03:00
fdc-test.c tests: Remove unnecessary glib.h includes 2016-06-07 18:19:24 +03:00
fw_cfg-test.c tests: Remove unnecessary glib.h includes 2016-06-07 18:19:24 +03:00
hd-geo-test.c tests: Remove unnecessary glib.h includes 2016-06-07 18:19:24 +03:00
i440fx-test.c coccinelle: Remove unnecessary variables for function return value 2016-06-20 16:38:13 +02:00
i82801b11-test.c tests: Remove unnecessary glib.h includes 2016-06-07 18:19:24 +03:00
ide-test.c tests: in IDE and AHCI tests perform DMA write before flushing 2016-07-18 18:19:01 -04:00
intel-hda-test.c coccinelle: Remove unnecessary variables for function return value 2016-06-20 16:38:13 +02:00
io-channel-helpers.c include/qemu/osdep.h: Don't include qapi/error.h 2016-03-22 22:20:15 +01:00
io-channel-helpers.h io: add QIOChannelSocket class 2015-12-18 12:18:31 +00:00
ioh3420-test.c tests: Remove unnecessary glib.h includes 2016-06-07 18:19:24 +03:00
ipmi-bt-test.c tests: Remove unnecessary glib.h includes 2016-06-07 18:19:24 +03:00
ipmi-kcs-test.c tests: Remove unnecessary glib.h includes 2016-06-07 18:19:24 +03:00
ipoctal232-test.c tests: Remove unnecessary glib.h includes 2016-06-07 18:19:24 +03:00
ivshmem-test.c os-posix: include sys/mman.h 2016-06-16 18:39:03 +02:00
libqtest.c tests: fix libqtest socket timeouts 2016-06-16 09:50:07 +05:30
libqtest.h all: Clean up includes 2016-02-23 12:43:05 +00:00
m48t59-test.c tests: Remove unnecessary glib.h includes 2016-06-07 18:19:24 +03:00
Makefile.include tests: introduce a framework for testing migration performance 2016-07-22 13:23:39 +05:30
ne2000-test.c tests: Remove unnecessary glib.h includes 2016-06-07 18:19:24 +03:00
nvme-test.c tests: Remove unnecessary glib.h includes 2016-06-07 18:19:24 +03:00
pc-cpu-test.c tests: Remove unnecessary glib.h includes 2016-06-07 18:19:24 +03:00
pcnet-test.c tests: Remove unnecessary glib.h includes 2016-06-07 18:19:24 +03:00
pkix_asn1_tab.c tests: Clean up includes 2016-02-16 14:29:27 +00:00
postcopy-test.c Use #include "..." for our own headers, <...> for others 2016-07-12 16:19:16 +02:00
prom-env-test.c tests/prom-env-test: increase the test timeout 2016-07-20 19:30:26 +03:00
pvpanic-test.c tests: Remove unnecessary glib.h includes 2016-06-07 18:19:24 +03:00
pxe-test.c tests: Remove unnecessary glib.h includes 2016-06-07 18:19:24 +03:00
q35-test.c tests: Remove unnecessary glib.h includes 2016-06-07 18:19:24 +03:00
qemu-iotests-quick.sh qemu-iotests: Speed up make check-block 2015-01-13 11:47:55 +00:00
qht-bench.c clean-includes: run it once more 2016-06-16 18:39:03 +02:00
qom-test.c tests: Remove unnecessary glib.h includes 2016-06-07 18:19:24 +03:00
rcutorture.c tests: Remove unnecessary glib.h includes 2016-06-07 18:19:24 +03:00
rtc-test.c tests: Remove unnecessary glib.h includes 2016-06-07 18:19:24 +03:00
rtl8139-test.c tests: Remove unnecessary glib.h includes 2016-06-07 18:19:24 +03:00
spapr-phb-test.c tests: Remove unnecessary glib.h includes 2016-06-07 18:19:24 +03:00
tco-test.c tests: Remove unnecessary glib.h includes 2016-06-07 18:19:24 +03:00
test-aio.c tests: Remove unnecessary glib.h includes 2016-06-07 18:19:24 +03:00
test-base64.c tests: Remove unnecessary glib.h includes 2016-06-07 18:19:24 +03:00
test-bitops.c bitops.h: Implement half-shuffle and half-unshuffle ops 2016-06-17 15:23:51 +01:00
test-blockjob-txn.c coroutine: move entry argument to qemu_coroutine_create 2016-07-13 13:26:02 +02:00
test-clone-visitor.c qapi: Add new clone visitor 2016-07-06 10:52:04 +02:00
test-coroutine.c coroutine: move entry argument to qemu_coroutine_create 2016-07-13 13:26:02 +02:00
test-crypto-afsplit.c include/qemu/osdep.h: Don't include qapi/error.h 2016-03-22 22:20:15 +01:00
test-crypto-block.c include/qemu/osdep.h: Don't include qapi/error.h 2016-03-22 22:20:15 +01:00
test-crypto-cipher.c tests: Remove unnecessary glib.h includes 2016-06-07 18:19:24 +03:00
test-crypto-hash.c crypto: implement sha224, sha384, sha512 and ripemd160 hashes 2016-07-04 15:52:36 +01:00
test-crypto-ivgen.c include/qemu/osdep.h: Don't include qapi/error.h 2016-03-22 22:20:15 +01:00
test-crypto-pbkdf.c include/qemu/osdep.h: Don't include qapi/error.h 2016-03-22 22:20:15 +01:00
test-crypto-secret.c crypto: remove temp files on completion of secrets test 2016-06-13 12:41:17 +01:00
test-crypto-tlscredsx509.c include/qemu/osdep.h: Don't include qapi/error.h 2016-03-22 22:20:15 +01:00
test-crypto-tlssession.c include/qemu/osdep.h: Don't include qapi/error.h 2016-03-22 22:20:15 +01:00
test-crypto-xts.c crypto: rename OUT to out in xts test to avoid clash on MinGW 2016-07-04 10:46:59 +01:00
test-cutils.c tests: Remove unnecessary glib.h includes 2016-06-07 18:19:24 +03:00
test-filter-mirror.c tests: Remove unnecessary glib.h includes 2016-06-07 18:19:24 +03:00
test-filter-redirector.c coccinelle: Remove unnecessary variables for function return value 2016-06-20 16:38:13 +02:00
test-hbitmap.c hbitmap: Use DIV_ROUND_UP 2016-06-07 18:19:25 +03:00
test-int128.c tests: Remove unnecessary glib.h includes 2016-06-07 18:19:24 +03:00
test-io-channel-buffer.c tests: Clean up includes 2016-02-16 14:29:27 +00:00
test-io-channel-command.c include/qemu/osdep.h: Don't include qapi/error.h 2016-03-22 22:20:15 +01:00
test-io-channel-file.c include/qemu/osdep.h: Don't include qapi/error.h 2016-03-22 22:20:15 +01:00
test-io-channel-socket.c socket: unlink unix socket on remove 2016-06-29 16:49:41 +02:00
test-io-channel-tls.c tests: Clean up includes 2016-02-16 14:29:27 +00:00
test-io-task.c tests: Remove unnecessary glib.h includes 2016-06-07 18:19:24 +03:00
test-iov.c tests: Remove unnecessary glib.h includes 2016-06-07 18:19:24 +03:00
test-logging.c log: Permit -dfilter 0..0xffffffffffffffff 2016-07-04 16:49:33 +03:00
test-mul64.c tests: Remove unnecessary glib.h includes 2016-06-07 18:19:24 +03:00
test-netfilter.c tests: Remove unnecessary glib.h includes 2016-06-07 18:19:24 +03:00
test-opts-visitor.c opts-visitor: Favor new visit_free() function 2016-07-06 10:52:04 +02:00
test-qdev-global-props.c tests: Remove unnecessary glib.h includes 2016-06-07 18:19:24 +03:00
test-qdist.c clean-includes: run it once more 2016-06-16 18:39:03 +02:00
test-qemu-opts.c tests: Remove unnecessary glib.h includes 2016-06-07 18:19:24 +03:00
test-qga.c tests: start a /qga/guest-exec test 2016-06-07 11:25:06 -05:00
test-qht-par.c clean-includes: run it once more 2016-06-16 18:39:03 +02:00
test-qht.c clean-includes: run it once more 2016-06-16 18:39:03 +02:00
test-qmp-commands.c qapi: Implement boxed types for commands/events 2016-07-19 13:21:08 +02:00
test-qmp-event.c tests: Remove unnecessary glib.h includes 2016-06-07 18:19:24 +03:00
test-qmp-input-strict.c qmp-input-visitor: Favor new visit_free() function 2016-07-06 10:52:04 +02:00
test-qmp-input-visitor.c qmp-input-visitor: Favor new visit_free() function 2016-07-06 10:52:04 +02:00
test-qmp-output-visitor.c qapi: Add new visit_complete() function 2016-07-06 10:52:04 +02:00
test-rcu-list.c tests: Remove unnecessary glib.h includes 2016-06-07 18:19:24 +03:00
test-rfifolock.c tests: Remove unnecessary glib.h includes 2016-06-07 18:19:24 +03:00
test-string-input-visitor.c string-input-visitor: Favor new visit_free() function 2016-07-06 10:52:04 +02:00
test-string-output-visitor.c qapi: Add new visit_complete() function 2016-07-06 10:52:04 +02:00
test-thread-pool.c coroutine: move entry argument to qemu_coroutine_create 2016-07-13 13:26:02 +02:00
test-throttle.c trivial patches for 2016-06-07 2016-06-07 16:34:45 +01:00
test-timed-average.c tests: Remove unnecessary glib.h includes 2016-06-07 18:19:24 +03:00
test-visitor-serialization.c qapi: Add new visit_complete() function 2016-07-06 10:52:04 +02:00
test-vmstate.c tests: Remove unnecessary glib.h includes 2016-06-07 18:19:24 +03:00
test-write-threshold.c tests: Remove unnecessary glib.h includes 2016-06-07 18:19:24 +03:00
test-x86-cpuid.c tests: Remove unnecessary glib.h includes 2016-06-07 18:19:24 +03:00
test-xbzrle.c util: move declarations out of qemu-common.h 2016-03-22 22:20:17 +01:00
tmp105-test.c tests: Remove unnecessary glib.h includes 2016-06-07 18:19:24 +03:00
tpci200-test.c tests: Remove unnecessary glib.h includes 2016-06-07 18:19:24 +03:00
usb-hcd-ehci-test.c tests: Remove unnecessary glib.h includes 2016-06-07 18:19:24 +03:00
usb-hcd-ohci-test.c tests: Remove unnecessary glib.h includes 2016-06-07 18:19:24 +03:00
usb-hcd-uhci-test.c tests: Remove unnecessary glib.h includes 2016-06-07 18:19:24 +03:00
usb-hcd-xhci-test.c tests: Remove unnecessary glib.h includes 2016-06-07 18:19:24 +03:00
vhost-user-bridge.c Use #include "..." for our own headers, <...> for others 2016-07-12 16:19:16 +02:00
vhost-user-test.c Use #include "..." for our own headers, <...> for others 2016-07-12 16:19:16 +02:00
virtio-9p-test.c tests: Remove unnecessary glib.h includes 2016-06-07 18:19:24 +03:00
virtio-balloon-test.c tests: Remove unnecessary glib.h includes 2016-06-07 18:19:24 +03:00
virtio-blk-test.c Error reporting patches for 2016-06-20 2016-06-20 16:19:18 +01:00
virtio-console-test.c coccinelle: Remove unnecessary variables for function return value 2016-06-20 16:38:13 +02:00
virtio-net-test.c Error reporting patches for 2016-06-20 2016-06-20 16:19:18 +01:00
virtio-rng-test.c tests: Remove unnecessary glib.h includes 2016-06-07 18:19:24 +03:00
virtio-scsi-test.c Error reporting patches for 2016-06-20 2016-06-20 16:19:18 +01:00
virtio-serial-test.c tests: Remove unnecessary glib.h includes 2016-06-07 18:19:24 +03:00
vmxnet3-test.c tests: Remove unnecessary glib.h includes 2016-06-07 18:19:24 +03:00
wdt_ib700-test.c coccinelle: Remove unnecessary variables for function return value 2016-06-20 16:38:13 +02:00