New functionality -- mostly in the diffs we apply to
buildah's helpers.bash -- to enable running buildah-bud
tests under podman-remote. The gist of it is, we start
a 'podman system service' before each test, and clean
it up on test exit.
Design decision: the diff file for helpers.bash is no
longer trailing-whitespace-clean: that ended up producing
diffs that git wouldn't apply, because in some cases
the whitespace is actually important. In order to pass CI,
we need to exclude this file from some checks.
Signed-off-by: Ed Santiago <santiago@redhat.com>
A hidden non-obvious corner-case of temporary changes introduced by
https://github.com/containers/podman/pull/10451 could be unintended
pruning of some ubuntu images. This could be impactful if for some
reason the `.cirrus.yml: use c5521575421149184 for Ubuntu` commit is
reverted beyond 30-days (the disused image-prune interval) and the _old_
images are needed (for an unforeseen reason).
Mitigate this by temporarily including the old images in the timestamp
updating task. This commit may be reverted (and the problem ignored)
if new VM images are built and deployed for all OS's (i.e. replacing
the Fedora/Ubuntu tag split workaround needed for the BZ).
Signed-off-by: Chris Evich <cevich@redhat.com>
Both the `containers/automation` and `containers/automation_images` have
been updated with a default 'main' branch, renamed from 'master`. These
images include adjusted tooling in addition to a refresh of all
packages.
Signed-off-by: Chris Evich <cevich@redhat.com>
Reason: to catch errors before they surface in RHEL.
One of the Ubuntus is specially crafted to run with cgroups v1
and runc. Although this isn't quite the same as RHEL, it's as
close as we can come in our CI environment, and I suspect it
would have caught #10234 (a regression).
Sorry, team.
Also: play kube limits test: skip on all rootless, not just
rootless+fedora. There was a complicated and unnecessary
check in there for Fedora.
Also: workaround for bug #10248, a spurious error message on
the first invocation of rootless podman on Ubuntu.Old
Signed-off-by: Ed Santiago <santiago@redhat.com>
Also simplify `lib.sh` after supporting changes incorporated
into automation library 2.x+ (present in all VM and container images).
* No need to force-load `/etc/profile` and handle it's expectation
to **not** being in `errexit` mode.
* Slightly re-arrange loading of automation library files for
clarity.
* Update comments.
Signed-off-by: Chris Evich <cevich@redhat.com>
* Incorporate changes from abandoned #9918: Use dedicated `bin`
sub-directories for `windows` and `darwin` when building
`podman-remote`. The linux flavor remains under `bin` as before.
* Fix MacOS Documentation-generation for release-packaging.
The `install-podman-remote-%-docs` target requires local execution
of `podman-remote`, but it was assuming GOOS=linux. Fix this
by dynamically discovering the local OS/architecture type while
still permitting cross-building of MacOS binaries under Linux.
* Unify temporary directory/file behavior to use a common template.
In case of left-over temporary items left in the repository,
update the `clean` target accordingly to remove them.
* Fix broken podman-remote-static and MacOS release archive targets
mismatching the `podman-remote-%` target. Disambiguate this target
for all platforms by spelling each out in full, instead of using
a wild-card recipe.
* Fix Windows-installer target to properly recognize existing
output files and not constantly rebuild every time.
* Include the podman version number in the Windows-installer target
in case a user downloads multiple releases.
* Include a subdirectory containing the podman version number for
both `tar.gz` and `zip` targets. This prevents users clobbering
existing directories when un-archiving from releases.
Signed-off-by: Chris Evich <cevich@redhat.com>
Set of scripts to run buildah's bud.bats test using
podman build in podman CI.
podman build is not 100% compatible with buildah bud.
In particular:
* podman defaults to --layers=true; buildah to false
* podman defaults to --force-rm=true; buildah to false
* podman error exit status is 125; buildah is 2
* differences in error messages, command-line arguments
Some of the above can be dealt with programmatically,
by tweaking the buildah helpers.bash (BATS helpers).
Some need to be tweaked by patching bud.bats itself.
This PR includes a patch that will, I fear, need to
be periodically maintained over time.
There will likely be failures when vendoring in a
new buildah, possibly because new tests were added
for new features that don't exist in podman, possibly
(I hope unlikely) if existing tests are changed in
ways that make the patch file fail to apply. I've
tried to write good instructions and to write the run
script in such a way that it will offer helpful hints
on failure. My instructions and code will be imperfect;
I hope they will be good enough to merit continued use
of this test (possibly with improvements to the instructions
as we learn more about real-world failures).
Signed-off-by: Ed Santiago <santiago@redhat.com>
Initial validation of using podman-in-podman to create an
old-podman root, then use new-podman to play with the
containers created therein.
Signed-off-by: Ed Santiago <santiago@redhat.com>
In anticipation of F34beta support, preemptively disable testing on
"prior-Fedora". This will allow development to move forward without
a soon-to-be-EOL distro. holding anything back.
Signed-off-by: Chris Evich <cevich@redhat.com>
In rare cases, it's possible for one of the ginkgo processes to "hang".
When this occurs, the main output will contain this message:
``Ginkgo timed out waiting for all parallel nodes to report``
The only way to debug this was to look through concatenated printing
of the ginkgo node logs. This is a tedious and daunting task,
requiring special search knowledge, facing a "wall of text".
Simplify the situation by collecting the node logs separately, as
individual files in a cirrus-artifact. In this way, it's faster to
figure out which test "hung" by examining each log individually. The
log file which does not have a pass/fail summary at the end,
indicates the last test hung (for whatever reason), and includes it's
output (if any).
Signed-off-by: Chris Evich <cevich@redhat.com>
Prior to this commit, the "Static Build" task only produced a
`bin/podman`. Update this to also include a `bin/podman-remote`
binary.
Update the pr-should-include-tests checker to ignore the `nix`
directory, which isn't applicable.
Lastly, restore the static build task to 'required' for CI success.
Leaving the comment inplace in case it needs to be bypassed in the
future on short notice.
Signed-off-by: Chris Evich <cevich@redhat.com>
Followup to dbb9943
Despite skipping the `Smoke` check, it was observed on a *new* branch,
the `validate` task (specifically `git-validation`) will fail. This
is because:
* `$CIRRUS_LAST_GREEN_CHANGE` will be empty on a new branch.
* `$CIRRUS_BASE_SHA` is always empty for runs triggered by branch-push
* `$EPOCH_TEST_COMMIT` will be set to `YOU_FOUND_A_BUG`.
Fix this by eliminating the `Smoke` task entirely, simplifying all
the `make validate` operations into the `validate` cirrus task. Ensure
this task does not run when a new branch or tag is pushed.
Also, eliminate the `$CIRRUS_BUILD_ID` value as it's confusing and not
actually used anywhere. It was formerly used for building VM images,
but this has moved to another repo entirely.
Signed-off-by: Chris Evich <cevich@redhat.com>
It looks like we always hit the caching issue in Cirrus CI described
within #8313. A solution around that is to use cachix, which has been
pre-populated from my local machine.
To push all (runtime and build) dependencies, we can leverage a
pre-populated store by:
```
> nix-store -qR --include-outputs $(nix-instantiate nix/default.nix) | cachix push podman
```
The cache can be re-used by everybody to rapidly build static Podman
binaries: https://app.cachix.org/cache/podman
[NO TESTS NEEDED]
Signed-off-by: Sascha Grunert <mail@saschagrunert.de>
The podman documentation site uses javascript to display
API documentation at:
http://docs.podman.io/en/latest/Reference.html
As input, the javascript sources from a CORS-enabled Google Cloud
Storage object. This commit ensures the storage object is present and
updated for every Cirrus-CI execution context: Tags, Branches, and PRs.
As of this commit, the documentation site only utilizes the object
uploaded by the Cirrus-CI run on the `master` branch:
`swagger-master.yaml`. The file produced and uploaded due to a PR is
intended for testing purposes: Confirm it's generation and uploading are
both functional.
Signed-off-by: Chris Evich <cevich@redhat.com>
There is no need to re-run the same basic validation checks as were
presumably already run on a PR before it merged. There are also
possible problems properly determining `$EPOCH_TEST_COMMIT` when
there have been no successful CI-runs on the branch (i.e. it's new).
This needlessly fouls up the git-validation tool. Fix Both problems
by just skipping the 'smoke' task for branches and tags.
Signed-off-by: Chris Evich <cevich@redhat.com>
Make sure that bindings are in sync with the code. The check is similar
to what's already being done with `make vendor`, so integrate the two.
[NO TESTS NEEDED]
Signed-off-by: Valentin Rothberg <rothberg@redhat.com>
Specifically, the result of 'make install.tools' is needed. Part of
that target installs tooling into `$GOPATH/bin`. A future commit
requires this tooling for the `Build Each Commit` item of the
alt_build matrix. Re-use the cache of this directory for this
task to ensure the necessary tooling/libraries are available.
Signed-off-by: Chris Evich <cevich@redhat.com>
Due to various reasons, CI results (esp. testing tasks) are completely
ignored for builds triggered by a new tag-push. Additionally, since
many of the automation scripts are in the repo., any related
failures/flakes would require code changes (therefore a new tag).
Resolve this by skipping every testing-type task for builds triggered by
tag-push. Only retain tasks which build things intended for consumption
associated with a possible official release.
Signed-off-by: Chris Evich <cevich@redhat.com>
Previously automation always dropped the minor version number for
distributions. This was intended for presentation and conditional
simplicity. Bash does not support non-integer comparison natively.
With the release of version 20.10, supporting testing with it and
the LTS release (20.04) requires scripts to consider minor version
numbers for Ubuntu VMs. This is necessary because many times in
the past, some behaviors needed to be conditional on the release
version number.
With this commit, the images and embedded scripts/tooling uses an
altered format of `$UBUNTU_NAME', `$PRIOR_UBUNTU_NAME`, and (crucially)
`$OS_RELEASE_VER` and `$OS_REL_VER`. Any `.` characters appearing
in the official version (from `/etc/os-release`) are dropped, and
the result is concatenated.
For example the current Ubuntu LTS version is `20.04`. Prior to
this commit, `$OS_RELEASE_VER` would have been `20`. With this
change, `$OS_RELEASE_VER` will now show `2004`. Similarly `20.10`
is shown as `2010`.
Signed-off-by: Chris Evich <cevich@redhat.com>
It's important to periodically update the nix cache (about 1GB in size).
If not, it can grow stale and has been observed causing task failures.
Associating the nix cache update with a VM/Container image update,
ensures it happens first in PRs where environment-related failures
are less surprising.
Signed-off-by: Chris Evich <cevich@redhat.com>
On several occasions, there have been questions about CPU/Memory/IO
trends in testing over time. Start collecting this data for all jobs,
using a common/stable format so that trending analysis can be performed
within/across multiple Cirrus-CI builds. This PR doesn't add any related
tooling, it simply arranges for the collection of the data.
Stats generation is done at the orchestration level to guarantee they
reflect everything happening inside `runner.sh`. For example, the
container-based tests re-exec `runner.sh` inside a container, but
we're only interested in the top-level stats.
Update all tasks to include collection of the stats file.
Unfortunately, due to the way the Cirrus-CI YAML parser works,
it is *not* possible to alias the artifacts collection more clearly,
for example:
```yaml
always:
<<: *runner_stats
<<: *logs_artifacts
```
Signed-off-by: Chris Evich <cevich@redhat.com>
Includes disk-space increase for all Fedora images to accommodate
the static-build job disk space requirements. This job substantially
leverages task-cache, which was previously failing to restore early on
in the Cirrus-CI task setup, due to disk-space limitations.
Also simplify .cirrus.yml slightly by removing an unncessary setup
and run directory change step.
Signed-off-by: Chris Evich <cevich@redhat.com>
On several occasions, fatal task failures were observed
during the upload of artifacts after a otherwise successful
testing. Prior to this commit, most tasks were storing both
logs and binary artifacts. Avoid possible major inconveniences
of upload failures, by only collecting binary artifacts when
necessary.
Signed-off-by: Chris Evich <cevich@redhat.com>
***Warning***: `skip` has non-obvious side-effects vs `only_if`:
https://cirrus-ci.org/guide/writing-tasks/#conditional-task-execution
The skip instruction can give a false sense of security by always
marking tasks as passed in the UI, even if they didn't run. In
contrast, the `only_if` condition will avoid creating the task
all -together; therefore, a problematic task's absense is more likely to
be noticed if it introduced a problem.
Signed-off-by: Chris Evich <cevich@redhat.com>
Initially filed as #7967 but that has run into huge complicated
snags related to Ubuntu and environment.
It is crucial to get system tests working with podman-local.
It is less important to get them on Ubuntu. Let's please
expedite this PR while we settle the Ubuntu stuff in #7967
Signed-off-by: Ed Santiago <santiago@redhat.com>
