This reverts commits eea2d14117 ("Makefile: remove $(TESTS_PYTHON)",
2023-05-26) and 9c6692db55 ("tests: Use configure-provided pyvenv for
tests", 2023-05-18).
Right now, there is a conflict between wanting a ">=" constraint when
using a distro-provided package and wanting a "==" constraint when
installing Avocado from PyPI; this would provide the best of both worlds
in terms of resiliency for both distros that have required packages and
distros that don't.
The conflict is visible also for meson, where we would like to install
the latest 0.63.x version but also accept a distro 1.1.x version.
But it is worse for avocado, for two reasons:
1) we cannot use an "==" constraint to install avocado if the venv
includes a system avocado. The distro will package plugins that have
"==" constraints on the version that is included in the distro, and, using
"pip install avocado==88.1" on a venv that includes system packages will
result in this error:
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
avocado-framework-plugin-varianter-yaml-to-mux 98.0 requires avocado-framework==98.0, but you have avocado-framework 88.1 which is incompatible.
avocado-framework-plugin-result-html 98.0 requires avocado-framework==98.0, but you have avocado-framework 88.1 which is incompatible.
make[1]: Leaving directory '/home/berrange/src/virt/qemu/build'
2) we cannot use ">=" either if the venv does _not_ include a system
avocado, because that would result in the installation of v101.0 which
is the one we've just reverted.
So the idea is to encode the dependencies as an (acceptable, locked)
tuple, like this hypothetical TOML that would be committed inside
python/ and used by mkvenv.py:
[meson]
meson = { minimum = "0.63.0", install = "0.63.3", canary = "meson" }
[docs]
# 6.0 drops support for Python 3.7
sphinx = { minimum = "1.6", install = "<6.0", canary = "sphinx-build" }
sphinx_rtd_theme = { minimum = "0.5" }
[avocado]
avocado-framework = { minimum = "88.1", install = "88.1", canary = "avocado" }
Once this is implemented, it would also be possible to install avocado in
pyvenv/ using "mkvenv.py ensure", thus using the distro package on Fedora
and CentOS Stream (the only distros where it's available). But until
this is implemented, keep avocado in a separate venv. There is still the
benefit of using a single python for meson custom_targets and for sphinx.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
This reverts commit ec5ffa0056.
Bumping avocado to version 101 has two issues. First, there are problems
where Avocado is not logging of command lines or terminal output, and not
collecting Python logs outside the avocado namespace.
Second, the recent changes to Python handling mean that there is a single
virtual environment for all the build, instead of a separate one for testing.
Requiring a too-new version of avocado causes conflicts with any avocado
plugins installed on the host:
$ make check-venv
make[1]: Entering directory '/home/berrange/src/virt/qemu/build'
GIT ui/keycodemapdb tests/fp/berkeley-testfloat-3 tests/fp/berkeley-softfloat-3 dtc
VENVPIP install -e /home/berrange/src/virt/qemu/python/
VENVPIP install -r /home/berrange/src/virt/qemu/tests/requirements.txt
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
avocado-framework-plugin-varianter-yaml-to-mux 98.0 requires avocado-framework==98.0, but you have avocado-framework 101.0 which is incompatible.
avocado-framework-plugin-result-html 98.0 requires avocado-framework==98.0, but you have avocado-framework 101.0 which is incompatible.
make[1]: Leaving directory '/home/berrange/src/virt/qemu/build'
To avoid this issue, tests/requirements.txt should use a ">=" constraint
and the version of Avocado should be limited to what distros provide
in the system packages. Only Fedora has Avocado, and more specifically
version 92.0 (though 98.0 is also available as a module). As a first
step, this patch reverts the introduction of a too-new Avocado.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
This patch changes how the avocado tests are provided, ever so
slightly. Instead of creating a new testing venv, use the
configure-provided 'pyvenv' instead, and install optional packages into
that.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20230511035435.734312-20-jsnow@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Avocado version 101.0 has a fix to re-compute the checksum
of an asset file if the algorithm used in the *-CHECKSUM
file isn't the same as the one being passed to it by the
avocado user (i.e. the avocado_qemu python module).
In the earlier avocado versions this fix wasn't there due
to which if the checksum wouldn't match the earlier
checksum (calculated by a different algorithm), the avocado
code would start downloading a fresh image from the internet
URL thus making the test-cases take longer to execute.
Bump up the avocado-framework version to 101.0.
Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com>
Tested-by: Hariharan T S <hariharan.ts@linux.vnet.ibm.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20230327115030.3418323-2-kconsul@linux.vnet.ibm.com>
Message-Id: <20230424092249.58552-2-alex.bennee@linaro.org>
This patch adds the "qemu" namespace package to the $build/tests/venv
directory. It does so in "editable" mode, which means that changes to
the source python directory will actively be reflected by the venv.
This patch also then removes any sys.path hacking from the avocado test
scripts directly. By doing this, the environment of where to find these
packages is managed entirely by the virtual environment and not by the
scripts themselves.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20220526000921.1581503-7-jsnow@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Besides some internal changes, new features, and bug fixes, on the QEMU side,
this version fixes the following message seen when running the acceptance
tests: "Error running method "pre_tests" of plugin "fetchasset": 'bytes'
object has no attribute 'encode'".
The release notes are available at
https://avocado-framework.readthedocs.io/en/latest/releases/88_0.html.
Signed-off-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20210520204747.210764-2-willianr@redhat.com>
Acked-by: Cleber Rosa <crosa@redhat.com>
Signed-off-by: John Snow <jsnow@redhat.com>
This version (and 84.0) contain improvements that address specific
QEMU use cases, including:
* Being able to download and use Fedora 31 images and thus
re-activate the "boot_linux.py" tests
* Being able to register local assets via "avocado assets register"
and use them in tests
Signed-off-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20210211232835.2608059-2-crosa@redhat.com>
Acked-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Signed-off-by: Cleber Rosa <crosa@redhat.com>
To use Avocado's testlogs plug-in on CI it is required to use
its 83.0 or greater version.
Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20201211183827.915232-2-wainersm@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
On with certain versions of "pip", package installations will attempt
to create wheels. And, on environments without a "complete" Python
installation (as described in the acceptance tests requirements docs),
that will fail.
pycdlib, starting with version 1.11.0, is now being made available
as wheels, so its instalation on those constrained environments is
now possible.
Buglink: https://bugs.launchpad.net/qemu/+bug/1897783
Reported-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20201009205513.751968-2-crosa@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
Reverse debugging test uses gdb remote client of avocado framework.
This client was fixed since the currently used version 76.
Therefore this patch bumps the version to 81 and fixes command
line version compatibility issue.
Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgalyuk@ispras.ru>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Tested-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <159903462803.28509.16851113546106095750.stgit@pasha-ThinkPad-X280>
Message-Id: <20200908202352.298506-2-philmd@redhat.com>
Message-Id: <20200909112742.25730-9-alex.bennee@linaro.org>
This acceptance test, validates that a full blown Linux guest can
successfully boot in QEMU. In this specific case, the guest chosen is
Fedora version 31.
* x86_64, pc-i440fx and pc-q35 machine types, with TCG and KVM as
accelerators
* aarch64 and virt machine type, with TCG and KVM as accelerators
* ppc64 and pseries machine type with TCG as accelerator
* s390x and s390-ccw-virtio machine type with TCG as accelerator
The Avocado vmimage utils library is used to download and cache the
Linux guest images, and from those images a snapshot image is created
and given to QEMU. If a qemu-img binary is available in the build
directory, it's used to create the snapshot image, so that matching
qemu-system-* and qemu-img are used in the same test run. If qemu-img
is not available in the build tree, one is attempted to be found
installed system-wide (in the $PATH). If qemu-img is not found in the
build dir or in the $PATH, the test is canceled.
The method for checking the successful boot is based on "cloudinit"
and its "phone home" feature. The guest is given an ISO image with
the location of the phone home server, and the information to post
(the instance ID). Upon receiving the correct information, from the
guest, the test is considered to have PASSed.
This test is currently limited to user mode networking only, and
instructs the guest to connect to the "router" address that is hard
coded in QEMU.
To create the cloudinit ISO image that will be used to configure the
guest, the pycdlib library is also required and has been added as
requirement to the virtual environment created by "check-venv".
The console output is read by a separate thread, by means of the
Avocado datadrainer utility module.
Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Tested-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20200317141654.29355-3-crosa@redhat.com>
Signed-off-by: Cleber Rosa <crosa@redhat.com>
If we want to use @skipUnless decorations on the class we need a
newer version of avocado.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20200303150622.20133-9-alex.bennee@linaro.org>
This replaces paramiko with avocado.utils.ssh module, which is based
on a (open)ssh binary, supposedly more ubiquitous.
Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Message-Id: <20190919225905.10829-1-crosa@redhat.com>
[Cleber: consolidated existing skipUnless from tests to setUp]
Signed-off-by: Cleber Rosa <crosa@redhat.com>
It's a good practice (I'd really say a must) to pin as much as
possible of the software versions used during test, so let's apply
that to paramiko.
According to https://pypi.org/project/paramiko/, 2.4.2 is the latest
released version. It's also easily obtainable on systems such as
Fedora 30.
Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20190607152223.9467-3-crosa@redhat.com>
Signed-off-by: Cleber Rosa <crosa@redhat.com>
This tests boot a full VM and check the serial console until
the SSH daemon is running, then start a SSH session and run
some commands.
This test can be run using:
$ avocado --show=ssh run -t arch:mips tests/acceptance/linux_ssh_mips_malta.py
ssh: Entering interactive session.
ssh: # uname -a
ssh: Linux debian-mips 3.2.0-4-4kc-malta #1 Debian 3.2.51-1 mips GNU/Linux
ssh: # lspci -d 11ab:4620
ssh: 00:00.0 Host bridge: Marvell Technology Group Ltd. GT-64120/64120A/64121A System Controller (rev 10)
ssh: # cat /sys/bus/i2c/devices/i2c-0/name
ssh: SMBus PIIX4 adapter at 1100
ssh: # cat /proc/mtd
ssh: dev: size erasesize name
ssh: mtd0: 00100000 00010000 "YAMON"
ssh: mtd1: 002e0000 00010000 "User FS"
ssh: mtd2: 00020000 00010000 "Board Config"
ssh: # md5sum /dev/mtd2ro
ssh: 0dfbe8aa4c20b52e1b8bf3cb6cbdf193 /dev/mtd2ro
ssh: # poweroff
Acked-by: Aleksandar Markovic <amarkovic@wavecomp.com>
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Aleksandar Markovic <amarkovic@wavecomp.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Message-Id: <20190523161832.22490-5-f4bug@amsat.org>
The Avocado test runner attemps to find its INSTRUMENTED (that is,
Python based tests) in a manner that is as safe as possible to the
user. Different from plain Python unittest, it won't load or
execute test code on an operation such as:
$ avocado list tests/acceptance/
Before version 68.0, the logic implemented to identify INSTRUMENTED
tests would require either the "🥑 enable" or "🥑
recursive" statement as a flag for tests that would not inherit
directly from "avocado.Test". This is not necessary anymore,
and because of that the boiler plate statements can now be removed.
Reference: https://avocado-framework.readthedocs.io/en/68.0/release_notes/68_0.html#users-test-writers
Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Caio Carrara <ccarrara@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Message-Id: <20190218173723.26120-1-crosa@redhat.com>
Signed-off-by: Cleber Rosa <crosa@redhat.com>
The acceptance (aka functional, aka Avocado-based) tests are
Python files located in "tests/acceptance" that need to be run
with the Avocado libs and test runner.
Let's provide a convenient way for QEMU developers to run them,
by making use of the tests-venv with the required setup.
Also, while the Avocado test runner will take care of creating a
location to save test results to, it was understood that it's better
if the results are kept within the build tree.
Signed-off-by: Cleber Rosa <crosa@redhat.com>
Acked-by: Stefan Hajnoczi <stefanha@redhat.com>
Acked-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Caio Carrara <ccarrara@redhat.com>
Message-Id: <20181018153134.8493-3-crosa@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
A number of QEMU tests are written in Python, and may benefit
from an untainted Python venv.
By using make rules, tests that depend on specific Python libs
can set that rule as a requirement, along with rules that require
the presence or installation of specific libraries.
The tests/requirements.txt is supposed to contain the Python
requirements that should be added to the venv created by check-venv.
Signed-off-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Acked-by: Stefan Hajnoczi <stefanha@redhat.com>
Acked-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Caio Carrara <ccarrara@redhat.com>
Message-Id: <20181018153134.8493-2-crosa@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
