Alpine is especially interesting because it uses musl as libc.
The build does not yet succeed. There are several issues that
need to be fixed.
However, it will be simpler to fix things, if we have tests
in place -- even if at the moment they are known to be broken.
See-also: https://git.alpinelinux.org/aports/tree/community/networkmanager?h=master
Ubuntu/Debian and CentOS/Fedora are sufficiently similar that it's
better that we have only one variant of ".gitlab-ci/*-install.sh"
and "contrib/*/REQUIRED_PACKAGES".
This was already the case, however, we used to symlink
".gitlab-ci/centos-install.sh" to "fedora-install.sh". That
worked, but it didn't scale very well. For example, if we would follow
that pattern, we would also need a symlink "contrib/centos/REQUIRED_PACKAGES"
Or should "contrib/centos" symlink to "contrib/fedora"? That seems even
more wrong.
We already had the "distro.base_type" variable for that. Make use of
that also for the install script.
It's not clear that ci-fairy has a stable API. Instead, we pin the
version by specifying the git commit sha.
In the build script, install the pinned version.
ci-templates builds and caches the test containers. When the build
scripts, the ci-template or "config.yml" changes, we need to bump
the tag so that the containers get rebuild.
Partly automate this. The tag now gets generated by the template and
contains a checksum of certain build files. Thus, if you change
any build files, then `ci-fairy generate-template` would generate a
different tag. You can not miss that, because we have tests that ensure
that our ".gitlab-ci.yml" is up to date. Also, you no longer need to
manually bump the tag when a build script changes, just regenerate
".gitlab-ci.yml" with `ci-fairy generate-template`.
See also: https://gitlab.freedesktop.org/freedesktop/ci-templates/-/merge_requests/54
The goal is to run most distros only manually. However, it would be nice
to avoid (manually) clicking twice to start the tests for one distro:
once for the container preparation, and once for the actual test.
Previously, the container prep part was set to manual and the actual
test automatic. It worked almost as desired, except that this leads
to the entire gitlab-ci pipeline be be in running state indefinitely.
To fix that, always run the container prep steps. If the container is
cached, this is supposed to be fast and cheap. Now only the actual tests
are marked as "manual".
It seems "pages" test does not get properly triggered, if only
t_fedora:33 completes. It should, because the other distros are
optional. Try to set "needs" to fix that.
Certain parts of the code are entirely generated or must follow
a certain format that can be enforced by a tool. These invariants
must never fail:
- ci-fairy generate-template (check-ci-script)
- black python formatting
- clang-format C formatting
- msgfmt -vs
On the other hand, we also have a checkpatch script that checks
the current patch for common errors. These are heuristics and
only depend on the current patch (contrary to the previous type
that depend on the entire source tree).
Refactor the gitlab-ci tests:
- split "checkpatch" into "check-patch" and "check-tree".
- merge the "check-ci-script" test into "check-tree".
On one image we do extra work, like generating documentation (gitlab-pages).
The same image is currently also used for the "checkpatch" step. That step
checks code formatting using clang-format. The formatting depends on the
clang version, and we currently choose Fedora 33 as the desired version
for formatting.
It means, the "checkpatch" step requires Fedora 33. We could choose a
different image for generating pages and run check patch. However, that
might not be best. Just also generate the pages using Fedora 33.
All the steps of "checkpatch" test (except the last) check
the current tree for consistency. Those checks must always
pass.
Only the last step calls the "checkpatch-feature-branch.sh".
That script checks for common patterns, like avoiding g_assert()
(in favor of other assertion types). That last check only checks
the current patch, and there are many cases where the test is
known to fail (because these are just heuristics). As such, the
step that may fail should be called as last.
ci-templates encourages building specific containers that can be re-used:
- containers are re-used across pipelines, producing consistent results
- containers are re-used by contributors since they will use the upstream
containers for their MR, thus guaranteeing the same results.
Containers are automatically rebuild whenever the respective
FDO_DISTRIBUTION_TAG changes. This is particularly interesting now that
Docker Hub will introduce pull limits.
This CI script consists of a config file and a jinja2 template, simply
running 'ci-fairy generate-template' produces the .gitlab-ci.yml.
ci-fairy is part of the freedesktop.org ci-templates and can be pip
installed, see the check-ci-script job.
Functional changes to the previous script:
- new job: check-ci-script, verifies that our gitlab-ci.yml is the one
generated by the sources
- Added distributions:
- Fedora 33
- The actual work is now down by a set of scripts in .gitlab-ci/,
specifically:
- .gitlab-ci/build.sh is the previous do_build job
- .gitlab-ci/{fedora|debian}-install.sh are the previous {fedora|debian}_install jobs
symlinks are in place for centos and ubuntu
Why the scripts instead of steps in the CI? Easer to reading and
reproduce. With the containers being static, it's easy to pull one
locally and re-run the CI job to reproduce an issue. Having everything in a
single script makes that trivial.
Signed-off-by: Peter Hutterer <peter.hutterer@who-t.net>
https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/merge_requests/664
There is no need spawn up two containers (and install similar packages, doing so is
resource intensive), to run two different tests. We can run them in the same
container.
At this point Fedora 33 is not yet fully released. But it will happen soon, so the
image is stable enough for these kinds of tests (and will only get more
stable in the following weeks).
Also no longer collect an artifact with the test output. The test output
is already nicely visible in the gitlab-ci UI. No need to collect it
separately.
Meson 0.54.0 requires ninja-1.7 ([1]).
On Ubuntu 16.04, we now would get meson 0.54.0 via pip3, but ninja-1.5.1 via
apt. That doesn't work anymore.
We could install ninja via pip3, but of course, doing that on other
Debian/Ubuntu versions fails due to ... I don't even want to know.
So, instead use an old meson version on Ubuntu 16.04, which is
known to still work with the ninja provided by the packaging system.
We anyway don't want to test the same meson/ninja versions on all our
Ubuntu/Debian images. The point of having different images is to build
with different software versions. If `pip3 install` gives us the same
everywhere, it isn't very useful.
https://mesonbuild.com/Release-notes-for-0-54-0.html#ninja-version-requirement-bumped-to-17
This fixes the pipeline as 'gem' will be installed by default in the
container image.
Also fix wording and run gitlab-triage in debug mode to get more output.
For the moment, we use docker images from dockerhub, which require
a lot of extra overhead to prepare and install the test environment.
This should be improved, by using more suitable container images.
Anyway, for now to alleviate the pressure on the freedesktop gitlab
infrastructure, disable most test to only run manually.
https://gitlab.freedesktop.org/NetworkManager/NetworkManager/issues/241#note_282521
When opening a merge request from a fork of NetworkManager, then the
pipeline runs with the a checkout of the fork. That means, checkpatch
would compare the branch against "master" (or "nm-x-y" stable branches)
of the fork, instead of upstream.
That doesn't seem too useful. Instead, also add upstream NetworkManager
as git remote, fetch the branches, and use the branches from there as
base for checkpatch.
https://gitlab.freedesktop.org/NetworkManager/NetworkManager/issues/255
"ubuntu:devel" ships iproute2 version "5.2.0-1ubuntu1". This has a well known
bug that prevents it from creating IP tunnels during the unit tests.
We already workaround that on Debian. Add the same workaround to match the
Ubuntu package.
REQUIRED_PACKAGES has two uses:
- to setup a system for developing NetworkManager. This installs
convenience packages like "cscope".
- to install the packages required for unit testing in gitlab-ci.
For gitlab-ci we should only install the packages that we actually
need.
Our platform unit tests try to add an IP tunnel using iproute2.
That fails with
"add tunnel "ip6tnl0" failed: File exists"
This is a bug in iproute2-5.2.0, see [1].
Workaround the issue by downgrading the package.
[1] https://www.spinics.net/lists/netdev/msg584916.html
Rolling is the latest release (regardless of whether LTS), currently
that would be 19.04.
Devel is the next release, currently that would be 19.10.
Add manual build steps to trigger those builds so we can manually verify
that they pass.
On Ubuntu 16.04 (trusty) valgrind fails due to rdrand being advertised
but not implemented.
Work around that by installing valgrind from Ubuntu 18.04 (bionic) via
the "contrib/scripts/nm-ci-install-valgrind-in-ubuntu1604.sh" script.
