- the container that is also "pages_build" should always
run automatically. This can replace the "always" tag.
- comment out the "always: 33" part, because we no longer need
it. It was also wrong, because by now we should run Fedora 34
automatically.
- the python-black version on Fedora 33 was updated, and this formats
our python code differently. The black version that is used by our
gitlab-ci is the authoritative version that should be used. Update
it by regenerating the containers.
- especially Fedora:rawhide and Debian:sid often introduce changes that
will cause a break of our build. Update the containers to test latest
versions.
Alpine is especially interesting because it uses musl as libc.
The build does not yet succeed. There are several issues that
need to be fixed.
However, it will be simpler to fix things, if we have tests
in place -- even if at the moment they are known to be broken.
See-also: https://git.alpinelinux.org/aports/tree/community/networkmanager?h=master
Ubuntu/Debian and CentOS/Fedora are sufficiently similar that it's
better that we have only one variant of ".gitlab-ci/*-install.sh"
and "contrib/*/REQUIRED_PACKAGES".
This was already the case, however, we used to symlink
".gitlab-ci/centos-install.sh" to "fedora-install.sh". That
worked, but it didn't scale very well. For example, if we would follow
that pattern, we would also need a symlink "contrib/centos/REQUIRED_PACKAGES"
Or should "contrib/centos" symlink to "contrib/fedora"? That seems even
more wrong.
We already had the "distro.base_type" variable for that. Make use of
that also for the install script.
It's not clear that ci-fairy has a stable API. Instead, we pin the
version by specifying the git commit sha.
In the build script, install the pinned version.
ci-templates builds and caches the test containers. When the build
scripts, the ci-template or "config.yml" changes, we need to bump
the tag so that the containers get rebuild.
Partly automate this. The tag now gets generated by the template and
contains a checksum of certain build files. Thus, if you change
any build files, then `ci-fairy generate-template` would generate a
different tag. You can not miss that, because we have tests that ensure
that our ".gitlab-ci.yml" is up to date. Also, you no longer need to
manually bump the tag when a build script changes, just regenerate
".gitlab-ci.yml" with `ci-fairy generate-template`.
See also: https://gitlab.freedesktop.org/freedesktop/ci-templates/-/merge_requests/54
The goal is to run most distros only manually. However, it would be nice
to avoid (manually) clicking twice to start the tests for one distro:
once for the container preparation, and once for the actual test.
Previously, the container prep part was set to manual and the actual
test automatic. It worked almost as desired, except that this leads
to the entire gitlab-ci pipeline be be in running state indefinitely.
To fix that, always run the container prep steps. If the container is
cached, this is supposed to be fast and cheap. Now only the actual tests
are marked as "manual".
It seems "pages" test does not get properly triggered, if only
t_fedora:33 completes. It should, because the other distros are
optional. Try to set "needs" to fix that.
We now install black by default via REQUIRED_PACKAGES script.
Thus, also when we build on Fedora 30, `make check` would run
python black. However, the formatting depends on the version
of python black, and the one in Fedora 30 is not right.
Skip all black tests during `make check`. We have a deicated
gitlab-ci test that runs black already (with the desired version
of black).
Certain parts of the code are entirely generated or must follow
a certain format that can be enforced by a tool. These invariants
must never fail:
- ci-fairy generate-template (check-ci-script)
- black python formatting
- clang-format C formatting
- msgfmt -vs
On the other hand, we also have a checkpatch script that checks
the current patch for common errors. These are heuristics and
only depend on the current patch (contrary to the previous type
that depend on the entire source tree).
Refactor the gitlab-ci tests:
- split "checkpatch" into "check-patch" and "check-tree".
- merge the "check-ci-script" test into "check-tree".
Now that the individual steps are no longer in .gitlab.yml but we
run a full shell script, clean it up to be better readable.
Also, we need to fail the script when any command fails.
On one image we do extra work, like generating documentation (gitlab-pages).
The same image is currently also used for the "checkpatch" step. That step
checks code formatting using clang-format. The formatting depends on the
clang version, and we currently choose Fedora 33 as the desired version
for formatting.
It means, the "checkpatch" step requires Fedora 33. We could choose a
different image for generating pages and run check patch. However, that
might not be best. Just also generate the pages using Fedora 33.
All the steps of "checkpatch" test (except the last) check
the current tree for consistency. Those checks must always
pass.
Only the last step calls the "checkpatch-feature-branch.sh".
That script checks for common patterns, like avoiding g_assert()
(in favor of other assertion types). That last check only checks
the current patch, and there are many cases where the test is
known to fail (because these are just heuristics). As such, the
step that may fail should be called as last.
ci-templates encourages building specific containers that can be re-used:
- containers are re-used across pipelines, producing consistent results
- containers are re-used by contributors since they will use the upstream
containers for their MR, thus guaranteeing the same results.
Containers are automatically rebuild whenever the respective
FDO_DISTRIBUTION_TAG changes. This is particularly interesting now that
Docker Hub will introduce pull limits.
This CI script consists of a config file and a jinja2 template, simply
running 'ci-fairy generate-template' produces the .gitlab-ci.yml.
ci-fairy is part of the freedesktop.org ci-templates and can be pip
installed, see the check-ci-script job.
Functional changes to the previous script:
- new job: check-ci-script, verifies that our gitlab-ci.yml is the one
generated by the sources
- Added distributions:
- Fedora 33
- The actual work is now down by a set of scripts in .gitlab-ci/,
specifically:
- .gitlab-ci/build.sh is the previous do_build job
- .gitlab-ci/{fedora|debian}-install.sh are the previous {fedora|debian}_install jobs
symlinks are in place for centos and ubuntu
Why the scripts instead of steps in the CI? Easer to reading and
reproduce. With the containers being static, it's easy to pull one
locally and re-run the CI job to reproduce an issue. Having everything in a
single script makes that trivial.
Signed-off-by: Peter Hutterer <peter.hutterer@who-t.net>
https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/merge_requests/664
