Debian 9 (stretch) is end of life, and the repositories are archived. We
need to patch the containers so that `apt-get update` continues to work.
A new ci-templates version brings that.
Note that at the moment, there is still another issue for debian:9
containers. Unclear whether that can be fixed. In any case, bumping to
latest ci-templates is not wrong, and works around the first issue on
debian:9, making it possible to at least look at the second issue.
https://gitlab.freedesktop.org/freedesktop/ci-templates/-/merge_requests/175
We want to follow current Fedora, so update to Fedora 38.
Also, we now use clang-format from Fedora 38 release, so the default
image in gitlab-ci must match, because that image is used for the
"check-tree" test.
While at it, add Fedora 39 and move Fedora 36 to tier 3.
These stages were not properly implemented and don't seem to work.
Drop them.
Note that we do want that our cached containers get collected eventually.
As these are just caches for performance reasons, that could be done with
little downsides (we can just regenerate the containers when we need them).
However, that's not done by our gitlab-ci stages. Instead, it should
be done on a project level. It's not clear whether that is actually done,
but if there is a need (because of the resources that this wastes), then
we should do that (on freedesktop.org's gitlab instance).
We want that the tier2+ tests are only run manually. As those tests
depend on the respective prep step, there are 3 possibilities:
1) make prep manual and the tier test automatic. That is what we would
want, because then we can just manually trigger the prep step (one
click). However, in the past this didn't work.
2) make the prep automatic and the test manual. That works, the downside
is that we often run the prep step when its not needed. This is what
we used to do to workaround 1).
3) make prep and the test manual. Then there are no unnecessary tests
run, but triggering a manual test is cumbersome. First click to start
the prep step, then wait, then click again.
Revisit this. It seems 1) is working now. Yeay.
Also rename the prep stages, so that it's clear to which tier they
belong. I guess, I could move them instead to prep1, prep2, prep3
stages, but then there are a lot of columns on the web site.
The distro.name is not just a pretty name, its the name under which we fetch
the container. It is thus a well-known name, that we can rely on.
The "base_type" only depends on the distro name, and it makes no sense
to ever choose a different name. Tracking it in the "distributions"
array is thus redundant.
Move the mapping of distro.name to the base type to a separate place.
The tag we actually use already contains a hash of the input files and
is generated (by `ci-fairy generate-templates`). There is no need for having
this fixed prefix. As also seens by having a date there, which is maintained
badly and meaningless.
Drop it.
The benefit is that instead of one long running job for fedora:37 (the
current tier1 test), we have several smaller.
A minor downside is, that if the build is broken, then usually the very
first test would already fail. Previously, that meant that the follow up
tests were skipped. Now, they run all in parallel. However, test
failures should be the exception, so the wasted resources are probably
irrelevant. The upside is, that we can see which tests fail, and we run
them much faster (in parallel).
This is only done for the tier1 test, because those tests are started
automatically. Other tiers need to be triggered manually, which already
means a lot of clicking. Making those also matrix tests, would result in
an insane amount of clicking. As those other tests are run much more
seldom, having them huge is probably fine.
We have many test configurations (i.e. distros like fedora:37,
debian:9). Almost all of them run manually triggered, because running
them every time would be wasteful.
Still, even as we trigger those tests only seldom, whenever we trigger
them all together, they consume still too many resources of the
freedesktop.org gitlab infrastructure.
One possibility would be to just drop old distros (e.g. fedora:30).
Which tests are setup in gitlab-ci is constantly refined and adjusted.
So dropping some distros is not necessarily wrong and bound to happen
eventually.
However, I also don't find it great to just disable tests that are still
passing. If we want to avoid consuming too many resources, we can just
choose not to run those tests. We don't need to enforce that by deleting
tests. Once deleted, such a configuration cannot be tested anymore as it
would be too cumbersome to recreate the setup manually.
Instead, introduce stages/tiers to clearer mark configuration that we
should test even less frequently.
Note that it is still required from the developer to not trigger too
many tests at once, to not monopolize the CI resources. The stages
should make that clearer to see, but don't solve it. Deleting tests
might solve it, but only if we delete a significant number of those
tests, which seems not desirable.
pip on Debian 12 semi-forces us to use a venv. That's hard enough but
even more so when we just want to run meson which only relies on the
standard library anyway.
Since that flag doesn't exist on earlier versions, try both and hope one
invocation succeeds.
https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/merge_requests/1595
It's not clear why this happens. But since recently in our gitlab-ci,
all the Fedora machines will fail. It happens in the step
check_run_clean 6 && test $IS_FEDORA = 1 -o $IS_CENTOS = 1 && ./contrib/fedora/rpm/build_clean.sh -g -w crypto_gnutls -w debug -w iwd -w test -W meson
which explains why it only affects Fedora configurations.
It does not always fail, but the probability of failure is high.
The failure is:
...
rm -f et.gmo && /usr/bin/msgmerge --for-msgfmt -o et.1po et.po NetworkManager.pot && /usr/bin/msgfmt -c --statistics --verbose -o et.gmo et.1po && rm -f et.1po
libgomp: Thread creation failed: Resource temporarily unavailable
make[3]: *** [Makefile:383: et.gmo] Error 1
Maybe some new resource restricting in gitlab. Let's add this workaround.
I don't really understand the cause, but this seems to avoid it, which is
good enough for me.
When we run `NM_TEST_SELECT_RUN=x ./.gitlab-ci/run-test.sh` to run one
step only, we should not do the final clean, so that the build artifacts
are preserved.
When the test in gitlab-ci fails, you might want to rerun the test
on your machine. You fire up podman, run "./.gitlab-ci/*-install.sh"
and "./.gitlab-ci/run-test.sh".
Make it possible to manually select parts that are tested by
"run-test.sh" by setting NM_TEST_SELECT_RUN. Otherwise, if you want to
test a particular configuration, you either have to run all earlier
steps (which takes a long time and can even be broken) or you have
to manually patch the file.
For example,
NM_TEST_SELECT_RUN=6 ./.gitlab-ci/run-test.sh
We want to follow current Fedora, so update to f37.
Also, we now use clang-format from Fedora 37 release, so the default
image in gitlab-ci must match, because that image is used for the
"check-tree" test.
Instead, hack gettext's Makefile.
gettext has an issue with parallel make. See [1] and [2].
Reproduce with:
git reset --hard &&
git clean -fdx &&
NOCONFIGURE=yes ./autogen.sh &&
./configure --enable-gtk-doc --enable-introspection &&
make -j distcheck V=1
We worked around this by setting "DIST_DEPENDS_ON_UPDATE_PO = yes",
however that (obviously) results in regenerating source files during
dist. "Source files" in the sense that the po files are commited to git
and get distributed in the release. Doing this is very ugly.
In particular it's ugly, because `make -C po update-po` is not reproducible
and the output depends on the current time (*had one job*).
Otherwise, we could just regenerate the files before doing a release.
This means, running "release.sh" script ends up with a dirty tree
afterwards. Also, the distributed po files are not the ones from the source
tree when we did the release. Also, since "release.sh rc1" does two distributions
(once for the rc1 and once for the next devel snapshot), the commit for the
second distribution will have a large diff for the po files.
This reverts commit 978d8eb699 ('po: make dist depend on update-po')
and hacks around the problem.
[1] https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/merge_requests/1094#note_1435313
[2] https://lists.gnu.org/archive/html/bug-gettext/2022-06/msg00022.htmlhttps://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/merge_requests/1405
Without it, the build artifacts were deleted before getting archived.
It means, the tarball and the docs were no longer archived and no
pages on gitlab no longer updated.
Fixes: e118276296 ('gitlab-ci: run unit tests for git subtree subprojects')
(cherry picked from commit cfe44c8832)
If we have a non-clean working directory after do_clean(), that
is a bug and something we need to investigate. Print information
to make that easier to debug.
Currently, for all tests we have python3 installed. So effectively,
even on CentOS 7, we would test the build with python3 only.
The package build on CentOS7/epel7 however uses python2. This happens
for example for our copr builds.
Also test that configuration in gitlab-ci.
In particular, `dpkg -l` likes to show a pager, when you are on the
terminal. Being on the terminal happens, if you try to reproduce
a test on your own container. So let's avoid that.
Previously, autotools would detect whether we have "black"
in the path. And if so, it would check formatting during `make check`.
That's problematic. When I run `./contrib/fedora/rpm/build_clean.sh -w test`
in certain cases, it would pick up black, but then fail with
Traceback (most recent call last):
File "/usr/bin/black", line 5, in <module>
from black import patched_main
File "/usr/lib/python3.6/site-packages/black.py", line 42, in <module>
from attr import dataclass, evolve, Factory
ModuleNotFoundError: No module named 'attr'
make[3]: *** [Makefile:21658: check-python-black] Error 1
That's an installation error of black, but still, during package build
there is no need to check the formatting. We could export
`NMTST_SKIP_PYTHON_BLACK=1` to prevent it, but it's still unnecessary.
We check proper formatting in gitlab-ci. That is enough, it doesn't
need to run during `make check`. In particular, because `black .`
takes 1.5 seconds on my machine.
During the test, we `tee` the output to a log file in "/tmp".
We do that, because the test script cleans the working directory
several times, so the file cannot reside there.
Afterwards, we need to move the file back into the git-tree, so that
gitlab can archive it.
Previously that was done by "after_script", but the "after_script" may not
see the same "/tmp" as the test run ([1]). This needs to be done as part of the
"script" step.
[1] https://docs.gitlab.com/ee/ci/yaml/#after_script
The "check-{patch,tree}" jobs use the same container as the default
test on Fedora ("pages_build", which also builds our documentation).
Previously, we thus extended "t_fedora:35". But that way we also
got things that we didn't want (.nm_artifacts and .build@template).
Solve this differently, by letting the jobs directly define what they
need. It's not much more, than extending "t_fedora:35" and workaround
to drop stuff we don't want.
Our test is long and verbose. The output gets truncated after
a few megabytes, but sometimes it's interesting to see what
happens afterwards. Redirect also to a file and archive it.
The output of our test scripts is captured by gitlab. It does however
sanitize things that look like secrets. So it was reasonably save
to call `env` from within the test script.
Next, we will redirect (`tee`) the output of the test script to a
file and archive it. When we do that, the output does not get sanitized
and can be downloaded from the artifacts page.
Stop running `env` as part of the test script. Do it instead as a
separate step. After all, it is useful to see the environment variables
of the test. But sanitized.
