Adds a drone node selector so builds will be routed to the correct buildbox
Also:
1. changes the version of Visual Studio expected by npm to vs 2022 to match buildbox installation
2. Removes the concurrency limit for window push builds. As there are separate builders for v10, v11
and v12. Given that the concurrency limits are enforced across all builds with the same name, they
serialise builds from different teleport versions. it seems wasteful to have a builder for v10 sit idle
while waiting a v11 build completes.
The concurrent limits have been left for tag builds, as there are flow-on concurrency concern due to artifact
uploading, etc.
Instead of using `git://git.openssl.org/openssl.git`, swap to the official
GitHub mirror at `https://github.com/openssl/openssl.git`. git:// is inheritly
insecure, and while we have additional protections in place as far as commit
hash checking, best to always pull dependencies from https:// wherever possible.
Devices running these architectures are likely not powerful enough
to handle desktop sessions. This will also reduce the binary size
for these builds, making them slightly more convenient for smaller
resource-constrained devices.
* Added multiarch build support for teleport oss, ent, and fips
* Exported image/imageTag types
* Resigned dronegen
* Removed remainder of testing changes
* Removed changes to submodules
* Reverted dockerfile-fips change
* FIxed docs wording
* Un-exported most constants
* Removed teleport.e makefile deb call
* Moved "sed | cut magic" to files
* Re-added `mkdir -pv /go/cache` to push.go
* Command deterministic order fix
* Added staging-only tag pipeline
* Moved PR to teleport operator to minimize potential issue impact
* Updated promote to pull and push without build
* Made cron triggers not affect canonical tags
* Added check for pre-existing tags on immutable CRs
* Added immutability check to manifests
* Updated staging ecr to only apply $TIMESTAMP tag on cron triggers
* Updated triggerinfo struct to use a triggerflag struct
* Fixed makefile after git mistake
* Makefile fix
* PR fixes
* Moved internal tools Go version to constant
* Separated container images gofile into multiple files
* Moved testing comment
* Added licenses
* Reorganized and added docs for container images
* Moved const to correct file
* Tag trigger logic test
* Testing specific fix
* Moved testing to v10.3.2
* Make semver dirs
* Refactored local registry name/socket
* Merged previous dockerfile changes
* Added TARGETOS TARGETARCH args
* Updatd tag to testing tag
* Promotion logic test
* Promotion fixes
* Testing specific fix
* Removed prerelease check for testing
* Added staging login commands to promote
* Fixed missing credentials on promotion pull
* Rerun tag test with new "full" semver
* Made staging builds only publish full semver
* Added semver logging command
* Empty commit to trigger Drone
* Promotion test
* Fixed preceeding v on promote pull
* Empty commit to trigger Drone
* Re-enabled verify not prerelease step on promote
* Cron trigger test
* Testing fix
* Testing fix 2
* Added sleep timer on docker buildx build
* Testing cleanup
Moves from github.com/golang/protobuf protoc-gen-go plugin to google.golang.org/
plugins.
This change was a long-time coming, but is now possible to do since our
dependencies are up-to-date.
* Move away from deprecated protoc-gen-go plugin
* Embed unimplemented server in handler.Handler
* Embed unimplemented server in multiplexer_test.go
* Update generated protos
Add definitions for Device Trust RPCs.
DeviceTrustService is fairly isolated from other services, so it is generated
using plain `protoc-gen-go` instead of Gogo.
teleport.e#514
* Add Device Trust proto definitions
* Allow proto generation without gogo
* Drop Gogo from lib/multiplexer protos
* Update generated protos
* Tidy modules
* Use Teleport's standard buildbox
This commit edits the teleport-operator container image build process to
rely on Teleport's standard buildbox. This will make sure we are using a
single go version at all time.
This also removed unused environment variables from
`operator/Makefile`.
* Extract BUILDBOX variables out of build.assets/Makefile
* Put `teleport-operator` bin out of the Teleport source volume
This PR updates our various Drone pipelines to use AWS roles for publishing.
Our AWS FTR requires that we do not use any long lived credentials in our AWS accounts and instead use roles. This means we need to move from attaching policies directly to users to attaching policies to roles and having policyless users assume those roles.
https://aws.amazon.com/partners/foundational-technical-review/
Contributes to https://github.com/gravitational/SecOps/issues/213
Without these changes, the promote step will always fail because of a
mismatch between where the repo is cloned and where it is referenced:
/go/src/.../teleport.git
vs
/go/src/.../teleport
(cherry picked from commit b209b98f0d)
* Add piv build dependencies.
- Add LIBPCSCLITE build tag.
- Add libpcsclite static linking using gravitational/pcsc fork.
- Enable use of dynamic pcsc library with LIBPCSCLITE=dynamic.
- Refactor CGOFLAG in Makefile.
- Update Centos7 Dockerfile and drone.
* Refactor RELEASE_MESSAGE for readability. Now produces message like: "RELEASE_MESSAGE=Building with GOOS=linux GOARCH=amd64 REPRODUCIBLE= and with PIV support and without PAM support, FIPS support, BPF support, Windows RDP client, libfido2, Touch ID."
Co-authored-by: Jakub Nyckowski <jakub.nyckowski@goteleport.com>
Update metalinter, fix a few lint warnings and replace deprecated linters.
`deadcode`, `structcheck` and `varcheck` are abandoned and now replaced by [`unused`][1].
Since 1.19, `go fmt` reformats godocs according to https://go.dev/doc/comment. I've done a bulk-reformatting of the codebase to keep the linter happy. Backporting is mostly harmless (the exception being `lib/services/role_test.go`, that for some reason breaks the _old_ linter using the new format).
[1]: https://golangci-lint.run/usage/linters/
* Bump golangci-lint version
* Replace abandoned linters
* Fix bodyclose on lib/auth/github.com
* Fix bodyclose on lib/kube/proxy/streamproto/proto_test.go
* Fix bodyclose on lib/srv/alpnproxy/proxy_test.go
* Fix bodyclose on lib/web/conn_upgrade_test.go
* Silence staticcheck on lib/kube/proxy/forwarder_test.go
* Silence staticcheck on lib/utils/certs_test.go
* Address BuildNameToCertificate deprecation warnings
* Run `go fmt ./...`
* Run `go fmt ./...` on api/
* Ignore formatting in role_test.go
* Remove redundant initializers in lib/srv/uacc/
* Update e/
WARNING: Due to issues with the windows drone executor's poor escaping when it echoes commands, I have moved the error message functionality into the PS build functions in build.assets/windows/build.ps1. This means that any failures that occur during the code checkout step will not be reported.
I'm not sure that this is the correct tradeoff, but it may well suffice for now.
Applies linters to legacy protos and adds a few additional Makefile targets to
make it easier to manage protos locally.
Proto linters now run in CI.
#15187
* Apply linters to legacy protos
* Handle new folders in genproto.sh, reset gen/proto if exists
* Lint and format lib/teleterm as part of protos/all
Uses Drone to build Teleport Connect for Windows on a Native
Windows builder.
This PR adds 2 pipelines to the Drone YAML:
1. `push-build-native-windows-amd64`: Invoked on a push to master,
branch/v*, etc., and asserts that Teleport Connect can be built, and
2. `build-native-windows-amd64`: Invoked when a branch tag is
committed to the teleport Repo. Builds Teleport Connect and uploads
it to dronestorage
These builds are run on a native windows builder (as opposed to tsh,
which is built in a linux environment and cross-compiled for Windows)
Change the proto layout of `api/` to a more standard setup, allowing the use of
modern tools (like Buf) to format/lint (and maybe, one day, generate sources).
The new layout looks like this:
``` api/ proto/ <- root of protos and proto imports teleport/ <- base
package for Teleport protos (akin to "google/" or "gogoproto/") legacy/ <- root
of "legacy" protos (most linters disabled) client/ proto/ types/ events/
webauthn/ wrappers/ ```
Non-legacy `api/` protos are expected to follow this layout:
``` api/ proto/ teleport/ mynewpackage/ <- package name v1/ <- protos
explicitly versioned gen/ proto/ <- root for generated sources
(multi-language possible, separate from hand-written code) go/ mynewpackage/ v1
<- generate Go sources go here. ```
Some outstanding issues, like lack of `go_package` declarations and non-standard
import paths (`import "github.com/gravitational/teleport/.../some.proto"`) are
fixed.
Legacy protos still have irregular package declarations. It's possible to fix
that, but it's a bit harder to reason about, as generated sources change in
possibly-meaningful ways.
Future iterations could change legacy packages to match the directory structure
and apply a similar change to protos within lib/ packages, but this seems
sufficient for a first step.
* Add Buf to buildbox
* Unify API protos under Buf
* Fix proto generation
* Reformat protos
* Update generated protos
* Generate protos using Buf
* Appease linter
* Review: make sure gogo protobuf versions are in sync
* Clean leftovers from previous attempts
* Fix operator/Makefile
* Rename internal make gRPC targets to `*/host`
* Sort `make fix-license` targets (nit)
* Add proof of concept of Connect pipeline
The proof of concept includes a lot of copy-pasted lines which will get
cleared up in subsequent commits.
* Extract copying artifacts into separate functions
The tag pipeline no longer needs to worry about Connect artifacts.
* Reuse steps to install & cleanup toolchains
* Share toolchain configuration commands between pipelines
* Share build commands among different pipelines
* Download webapps only if a pipeline builds Connect
As seen by the changes to .drone.yml, this removes unnecessary webapps
clones from these tag pipelines: build-darwin-amd64, build-darwin-amd64-pkg,
build-darwin-amd64-pkg-tsh. None of them needs webapps to function anymore
and the pkg pipelines never needed webapps in the first place.
In order to do so, we add a new make target:
make teleterm
This (temporarily) assumes that the gravitational/webapps repo is
cloned at the right version as a sibling to the teleport repo.
(We'll be able to get rid of this when we merge webapps into Teleport)
Additionally, update dronegen to include the name of the calling
function that generated the snippet instead of the line number.
This gets rid of lots of superfluous diffs in the generated
.drone.yml file.
Lastly, rewrite the Go program for getting the right webapps version
in bash, because Go is not available at this step of the drone pipeline.
Co-authored-by: Grzegorz Zdunek <grzegorz.zdunek@goteleport.com>
Adds the GCB build yaml for controlling the build, and updates the test script
to work in both the GCB environment and on a local dev machine.
Also changes the centos buildbox to leave the default user as root. When
GCB mounts the workspace into the container, the source code is owned
by root, and there is no way to change this. This means that the build will
fail when the non-root user specified in the build image attempt to write files
into the workspace. Setting the root user fixes this.
See-Also: #15186
Now that we have automation in place for updating the webassets
repo, this script no longer needs to build webassets. Instead,
it just updates the webassets submodule to point at the tip of
whatever branch is specified and opens the Teleport PR.
This is a twofold change with the aim of reducing possible pains with the tsh
installer.
- Dropping the version number from "tsh.app" makes it more alike other apps
(including Connect)
- Making the installer non-relocatable makes it easy to reason about (and
ensures our postinstall script is correct!)
A relocatable installer will look for the app in places other the specified
install path, according to the bundle ID. This means that if the user moves or
renames the app, the installer will overwrite it no matter where it is. It also
means our path assumptions can be wrong.
Note that the installer itself is still numbered, so it won't break Houston or
change the downloads page.
Ports used by the unit tests have been allocated by pulling them out of a list, with no guarantee that the port is not actually in use. This central allocation point also means that tests cannot be split into separate packages to be run in parallel, as the ports allocated between the various packages will be allocated multiple times and end up intermittently clashing.
There is also no guarantee, even when the tests are run serially, that the ports will not clash with services already running on the machine.
This patch (largely) replaces the use of this centralised port allocation with pre-created listeners injected into the test via the file descriptor import mechanism use by Teleport to pass open ports to child processes.
There are still some cases where the old port allocation system is still in use. I felt this was already getting beyond the bounds of sensibly reviewable, so I have left those for a further PR after this.
See-Also: #12421
See-Also: #14408
- Enables the docker BuildKit in an attempt to speed up builds
- Trims slightly under 2GB off image size
- Break more dependencies out into separate build stages
- Adds some simple supply-chain protections for dependencies sourced
via git. The Docker build now checks that the commit SHAs are what
we expect, and not just assume that the tags haven't changed.
- Moves the `cbindgen` build to a stage to avoid pulling in extra
dependencies not needed for the Teleport build
- Combines the `gcloud` and firestore emulator install into one step to
reduce the layer count.
- Ports some of the above the Centos7 Dockerfile.
Drop the `v` from the tsh installer version number, which was inadvertently
changed by #12751. Makes the installer reappear as a download option in Houston.
Note that the final .app name still has the `v`. Ie:
* tsh-10.0.0-dev.pkg (installer)
* tsh-10.0.0-dev.pkg.sha256 (installer hash)
* tsh-v10.0.0-dev.app (Application package)
This code was unmaintained, created issues with our build system,
and didn't actually match the behavior of Teleport's RBAC engine.
We will revisit this functionality in the future when we investigate
"acess policies as code."
Attempt to detect builder environment inconsistencies by compiling a toy FIDO2
program - if this fails, then clear the cache and try again.
Builders are sometimes getting into inconsistent states, this should help
avoiding manual intervention in order to fix them.
Recent Rust dependency upgrades include a newer version of prost.
This new version no longer ships embedded protoc binaries, and
instead tries to build protoc from source. This would require us
to install cmake on our buildboxes. We want to avoid this and
instead leverage the version of protoc already installed.
This change was made to the standard buildbox, but the CentOS 7
buildbox was missed.
Additionally, I noticed that Rust was installed in
Dockerfile-centos7-fips, but not in Dockerfile-fips, which means
the FIPS binaries have different functionality depending on which
version you use. To correct this, I removed Rust from the CentOS 7
FIPS builds (since the Rust features are not FIPS compliant anyway).
Switch from `make release-amd64` to make release-windows in Drone builds, making
release builds similar to "regular" builds (that already use
`make release-windows-unsigned`).
Fixes current woes caused by FIDO2=yes in Windows release builds. (Note that
ARCH is implied by the build.)
* Use `make release-windows` on Drone, make it similar to `make release`
* Update .drone.yaml
After recent changes in #12257, Dockerfile-teleterm was made to accept
NODE_VERSION passed from a build arg.
The problem is that NODE_VERSION used to follow the format of `vX.Y.Z`,
while NODE_VERSION in build.assets/Makefile follows the `X.Y.Z` format.
This commit adds the missing `v`s to NODE_URL and NODE_PATH.
This commit updates drone to build Teleport Connect by:
* cloning `gravitational/webapps` as a sibling directory to
gravitational/teleport
* checkout out the right version of webapps by running a simple
Go program (this step is only necessary until we move webapps
into the teleport repo)
* Running the Teleport Connect build and copying artifacts
Code signing should run on tag builds automatically as part the
electron build, assuming the Apple Account credentials are
properly loaded into the keychain.
Notarization will also happen automatically if both
`$APPLE_USERNAME` and `$APPLE_PASSWORD` are set.
In order to make the above happen, this patch also includes:
* Installing and removing a per-build Node instance in the
toolchain directory on Darwin
* Moving the toolchain temporary directory out of ~/ and into /tmp.
Drone usually sets `$HOME` to a temporary directory for each build,
but unfortunately we need it to point to the actual build user's
home directory in order for the notarisation tooling to find the
right keychain. Having $HOME point to a long-lived directory risks
both pollution from build detritus and builds stomping on one another.
In an in an attempt to isolate the builds from each other and protect
`~build` as best we can, as much of the build state as possible
(including ephemeral toolchains) has been moved under `/tmp`.
Co-authored-by: Trent Clarke <trent@goteleport.com>
Set the macOS deployment target, ensuring that statically linked libfido2 `tsh`
builds run correctly on older macOS versions.
#9160
* Consistently set macOS min version
* Bump min macOS version to 10.13
Since #12794 we now build `tsh` binaries with touch ID capabilities. This calls
for a more sophisticated mechanism to determine if touch ID functions should be
enabled, as compile-time support only is not enough.
I've added the following checks, on top of compile-time / `touchid` build tag:
Binary is signed
Binary has entitlements
Machine is touch ID capable
Machine has a Secure Enclave
Put together this give us a much better proxy on whether to enable touch ID.
I've also added the `tsh touchid diag` command, mentioned in the Passwordless
macOS RFD (see
https://github.com/gravitational/teleport/blob/master/rfd/0054-passwordless-macos.md#tsh-support-commands).
#9160
* Improved touch ID availability and diagnostics
* Add the `tsh touchid diag` command
* Set min macOS version to 10.12 (macOS Sierra)
Add a script to build libfido2 (and its dependencies) on macOS and enable FIDO2
static builds.
I decided to build all dependencies instead of pulling from Homebrew for a few
reasons:
1. There is no libcbor.a in a brew package
2. This captures library versions within the Teleport source code, allowing us
to build binaries against different versions of libfido2 (and its
dependencies).
I've also bumped libfido2 to 1.11.0. I've been running it locally and we are
still pre-release, so it seems like a good time to do it.
(See https://developers.yubico.com/libfido2/Release_Notes.html.)
#9160
* Build libfido2 and dependencies for macOS
* Build tsh with static fido2 on Drone
* Bump libfido2 versions in all builds
* Attempt to appease linters
* Use temp dirs inside LIB_CACHE
* Move LIB_CACHE outside of HOME
HOME is reassigned in macOS builders, but we want a "stable" cache
directory. /tmp is used by build-package.sh and build-pkg-tsh.sh.
* Rename script to build-fido2-macos.sh
* Regenerate Drone files
Changes how `make pkg-tsh` works so instead of building an installer for the
`tsh` binary, placed under `/usr/local/bin`, we install an app to
`/Applications/tsh-vXXX.app` and link its `tsh` binary to `/usr/local/bin`.
The app shell is necessary to distribute a provisioning profile along with the
signed/entitled/notarized binary. All of that is required for Touch ID to work.
Naked `tsh` binaries are unable to use Touch ID, even if built with the correct
build tags.
I've elected to split the logic from `build-package.sh` into a separate script -
it already does too much as-is. `build-pkg-tsh.sh` is more idiomatic, clears
additional `shellcheck` rules and is easier to dry-run.
#9160
* Build macOS installer for tsh.app
* Add resources to build the tshdev app
Moved from e/
* Add resources to build the tsh app (prod)
* Use production values
* Remove 'tsh' mode from build-package.tsh
* Appease buildbox linter
* Clarify one-time setup
* Update golangci-lint
To accomodate the recent Go 1.18 upgrade
* Fix new lint warnings as a result of linter upgrade
* Set golangci-lint to Go 1.18 mode
golangci-lint will automatically skip linters that don't have support
for Go 1.18.
See: https://github.com/golangci/golangci-lint/issues/2649
Reinstates Linux/amd64 and Centos7/amd64 builds using libfido2, now hidden
behind an explicit FIDO2 flag (similarly to FIPS).
This PR pulls in gravitational/go-libfido2#4 and adds the required pkg-config
setup so we can perform both dynamic (mostly testing) and static (tsh) builds.
Additionally, pkg-config is now the gateway for whether we run libfido2-related
tests (which should always happen in CI).
#9160
* Re-enable libfido2 builds for amd64 and Centos7
* Use pkg-config to build tsh with libfido2
* Install Centos7 libudev-zero to /usr/local/lib64
* Update gravitational/go-libfido2
* Remove /usr/local/lib from Centos PKG_CONFIG_PATH
Original behaviour did not take effect in CI due to a different entrypoint.
This restores the original behaviour (which will link external links when using make -C build.assets test-docs) but disables the external linting in CI for reliability.
Updates #11940
Build `tsh` with static `libfido2`, `libcbor`,`libcrypto` and `libudev-zero`.
Dockerfiles for buildbox and Centos7 changed. FIPS and macOS to be addressed at
a later date.
Add the `tsh fido2 diag` hidden command for ease of testing.
#9160
* Update go-libfido2 and tidy modules
* Add a fido2 diagnostic command to tsh
* Add a few build artifacts to .gitignore
* Build tsh with static libfido2 in buildbox
* Build tsh with static libfido2 in centos7
* Add a few relevant cmake flags
* Use illiliti/libudev-zero
* Do multi-stage build on centos7, image tweaks
* Add `make enter/centos7`
* s/OFf/OFF/g
The grpc-tools package is needed to generate gRPC files for JavaScript.
However, at the moment it can't be installed on M1 MacBooks because of
missing prebuilt binaries for arm64. [1]
One of them, protoc, is already installed in our buildbox. We still need
to compile grpc_node_plugin from source though. This adds significant
overhead as we need to pull in cmake, build-essential and then about
300 MB of git repos from protocolbuffers/protobuf.
Initially, those Teleterm gRPC were generated within `make grpc` with other
files. M1 users who don't work on Teleterm would not be happy about incurring
that additional overhead, hence I extracted everything into separate target
and Dockerfile.
Teleterm proto files don't depend on any other proto files. Once grpc-tools
adds support for arm64, we'll be able to essentially almost revert this
commit and generate Teleterm gRPC files within `make grpc`.
[1] https://github.com/grpc/grpc-node/issues/1405
- Lint libfido2 (and other) Go build tags
- `make test-go` exercises the libfido2 build tag, as long as `libfido2` is present in the system
- Install `libfido2` (and dependencies) in the teleport-buildbox image
Libraries are installed from source, instead of apt or ppas, so we can guarantee deterministic (and current!) versions.
(Binary releases are not available.)
At the present moment, `librdp_client` and `libfido2` can't be used together. This is because `librdp_client` embeds
openssl/`libcrypto`, which is also a dependency for `libfido2`, causing duplicate symbol errors. In practice both
libraries never coexist in the same binary, so it's easy to sidestep the issue (`librdp_client` links to `teleport`,
while FIDO2 code is only used by `tsh`). I may be able to make them coexist, but not without changes to how go-libfido2
builds.
This change is only for linting/testing libfido2 code, I'll address `tsh` releases in a future PR.
#9160
* Install libfido2 in buildbox
libfido2 and libcbor are installed from source to make sure we get
deterministic versions (apt is outdated and ppas are likely to move
forward with time).
* Run libfido2 tests on test-go
* Lint libfido2 Go build tag
* Lint other Go build tags
* Comment build tags that break the linter
* Tidy modules
* Re-enable roletester linter
* Pass tags conditionally to golangci-lint
* Clarify and improve libfido2 wildcard
* Drop `:$LD_LIBRARY_PATH` from variable
* Replace LD_LIBRARY_PATH with `ldconfig`
* Test for ARM homebrew location too
* POC for Helm unit tests
This uses https://github.com/vbehar/helm3-unittest to define
expectations of our helm templates
* Test that enterprise is configured correctly
* Added tests for teleport-cluster
* Added tests for teleport-kube-agent
* Removed tests for teleport chart
* Add tests for teleport-cluster Deployment
* Run shorter tests first
* Fix Docker plugin installation and add update-helm-snapshots target
* Add README
* Fix lint syntax error and add some missing linters
* Add missing ImagePullPolicy to Deployment and StatefulSet
* Add Deployment tests for teleport-kube-agent
* Fix replicaCount logic
* Add clarification to values
* Add StatefulSet suite for teleport-kube-agent
* Update snapshots after merge with master
* Helm tests are quicker than bash tests
* Add tests for extraEnv
* Random space
* Tidy up formatting of multiple tests
* [debug] List helm plugins and directories
* Special case Helm linting when running in CI
* Make trailing line breaks consistent
* Special case Helm linting when running in CI
* Add contribution guidelines for Helm charts
* Add contribution guidelines to READMEs
* Deprecate old charts
* Typo
* Spacing
* Clarification
* Update examples/chart/CONTRIBUTING.md
* Don't erroneously set extraEnv for initContainers
* Rename update-helm-snapshots -> test-helm-update-snapshots for clarity
Co-authored-by: Gus Luxton <gus@goteleport.com>
Co-authored-by: Roman Tkachenko <roman@goteleport.com>
We do not publish pre-releases to apt repos, but we do publish them to
github. That means we need to filter them out when considering if an
apt release should be published. We don't want v8.3.3 to be blocked by
v9.0.0-dev.1, only by v9.0.0.
Honestly, this is a bit of a mess, but it only needs to hold out a bit
longer until https://github.com/gravitational/teleport/pull/10746 lands.
Contributes to https://github.com/gravitational/teleport/issues/10800
* Include tbot binary in Teleport packages and installs
This includes the tbot binary in .rpm, .deb, and .pkg distributions,
and ensures the binary is installed using the `install` script in
.tar.gz packages.
* Remove tbot from macOS client-only builds
Prior to this patch the teleport buildbox version has been tagged with the Go version for the current release. This bit us during the Teleport 9 development cycle, as both Teleport 8 and 9 use the same version of Go but require different versions of Rust, and we were unable to distinguish between the 2 buildbox versions.
At the time, Teleport 8 was individually patched to create a new `teleport8` buildbox tag, decoupling the buildbox version from the Go version. This was never ported into master and now we find the teleport 9 branch sharing the same buildbox tag as master.
This patch forward-ports all the changes made to `branch/v8` and updates them for master, creating a new `teleport10` buildbox tag. The idea is that we will create a new tag for teleport11 at the same time the release branch for Teleport 10 is mad at some point in the future.
Once this is merged, Drone will create and push new buildbox images, which will become available for CI. A subsequent patch will update the CI scripts to use the new `teleport10` buildbox images.
API_IMPORT_PATH is consistently being resolved as an empty string, breaking
proto generation.
Since the path is fixed, it seems simpler to read api/go.mod and do away with
the Go program.
* Explicitly set API_IMPORT_PATH
* Delete the print-import-path program
* Read api module from api/go.mod, push variables to target
This gives us a robust way to find the latest published release for a
Major or Major.Minor version. This logic is useful for our automation
that publishes up-to-date teleport:X docker images
Contributes to https://github.com/gravitational/teleport/issues/9494
These should not be factored in when checking for the latest release
when we decide if we should release apt packages.
This also fixes a bug in sorting logic, where we were sorting
lexigraphically instead of by semver.
9 was comparing greater than 10, due to use of lexographic sorting
This would cause us to fail to publish apt packages when we roll over to
a patch release > 9.
The script for updating webassets uses the commit message from
webapps as the commit message for the PR to teleport.
This commit message is almost always a merged PR, which has the format:
do some awesome thing (#123)
Where '#123' is the number of the **webapps** PR that was merged.
The problem with this is, when the teleport PR is created, it interprets
the #123 as the number of a **teleport** PR. And since the Teleport repo
has a lot more issues/PRs than webapps, Github ends up linking to an old
and completely unrelated PR.
Fix this by replacing (#123) with (gravitational/webapps#123), which
Github correctly renders as a link to the webapps PR in question.
* Add more lint coverage
golanglint-ci doesn't pick up subdirectories with their own go.mod
which left certain directories unlinted. To get around this we can
run golanglint-ci directly against those submodules.
In Rust 1.58, deriving Debug no longer counts as using a struct's
fields, so we need to allow dead_code for our structs that implement
RDP protocols. (Just because we don't use the fields doesn't mean
we shoudln't decode them)
Starting with the Teleport 9 release, we will be versioning the
API module. This change ensures that the generated protobuf code
imports the correct version of the API by:
- introducing a small new command to print the correct version
- adding import rewrite rules to the protoc invocation
This patch makes a couple changes:
1. deb archives are not published to apt if they're not the latest
release ever
2. both rpm and deb archives are no longer published to yum / apt if
they contain any pre-release indicator or build metadata
3. nothing is published if the commit isn't tagged.
Contributes to https://github.com/gravitational/teleport/issues/8166
Linting for non-go files was accidentally dropped in the transition to
GCB (sorry!). This patch restores linting for non-go files and fixes
any lint failures that have crept in during the interim.
Actually tracking down the cause of a failure in the integration tests can
be hard:
* It's hard to get an overall summary of what failed
* The tests sometimes emit no output before timing out, meaning any
diagnostic info is lost
* The emitted logs are too voluminous for a human to parse
* The emitted logs can present information out of order
* It's often hard to tell where the output from one test ends
and the next one begins
This patch attempts to address these concerns without attempting to rewrite
any of the underlying teleport logging.
* It improves the render-tests script to (optionally) report progress per-
test, rather than on a per-package basis. My working hypothesis on the
tests that time out with no output is that go test ./integration is
waiting for the entire set of integration tests tests to be complete
before reporting success or failure. Reporting on a per-test cycle gives
faster feedback and means that any timed-out builds should give at least
some idea of where they are stuck.
* Adds the render-tests filter to the integration and integration-root make
targets. This will show an overall summary of test results, as well as
- Discarding log output from passing tests to increase signal-to-noise
ratio, and
- Strongly delimiting the output from each failed test, making failures
easier to find.
* Removes the notion of a failure-only logger in favour of post-processing
the log events with render-tests. The failure-only logger catches log
output from the tests and only forwards it to the console if the test
fails. Unfortunately, not all log output is guaranteed to pass through
this logger (some teleport packages do not honour the configured logger,
and reports from the go race detector certainly don't), meaning some
output is presented at the time it happens, and other output is batched
and displayed at the end of the test. This makes working out what
happened where harder than it need be.
In addition, this patch also promotes the render-tests script into a fully-
fledged program, with appropriate makefile targets, make clean support, etc.
It is now also more robust in the face on non-JSON output from go test
(which happens if a package fails to compile).
* Update buildbox to use Python3.
* Remove non default rust targets from arm64 image.
* Add ETCD_UNSUPPORTED_ARCH for arm64 to etcd script to allow running etcd on arm64.
* Ensure that slice.pb.go is generated by `make grpc`
* Clean up `make grpc`
* Disable the test target rules in Makefile when running inside the devbox
- Ensure that the protoc include directory is readable by all users
- Switch back to the root user by default
Either of these changes would have fixed the issue on their own,
but I decided to include both as GRPC should be readable by non-root
users, and I wanted to preserve the original behavior of running
as root unless the $(NOROOT) flags are specified.
Additionally: clarify comments on the make targets, which are
confusingly named, and stop installing goimports since it seems
it was never used.
Add new buildboxes for centos7 and centos7-fips.
For now, we will continue to support both CentOS 6 and 7.
Eventually we will drop support for CentOS 6, and the only
supported CentOS builds will be these new CentOS 7 builds.
Fixes#9028
* update deps in manifest and lockfile
* fixes and updates to docker and profiles
* lint rust
* fix typo
* resolve clippy lints
* fix typo
* mark risk functions unsafe
* fmt + clean up the last lints#
* verify lockfile up to date
* disable lto since it doesn't work with two rust libs
* merge lock check and lint
* Add missing license header to Rust files
And update Makefile to ensure they are checked
Co-authored-by: Zac Bergquist <zac.bergquist@goteleport.com>
The race condition detector is being tripped by a concurrent `Write` and
`Close` in the `PipeNetCon` in several integration tests. This is a naive
fix to serialize the write and close operations to resolve the race
condition.
The affected tests were also not handling asynchronous error reporting
correctly (i.e. it's not legal to call `require.XYZ()` from a goroutine
other than the one executing the test function.). This patch introduces
some plumbing to marshal asynchronous errors back into the main test
routine before failing the test.
The Rust code now uses vendored mode [1] to statically link openssl,
so we no longer need dynamic linking for these libraries.
This also resolves an issue where extra flags were needed to build
locally on macOS.
[1]: https://docs.rs/openssl/0.10.36/openssl/#vendored
- Ensure Rust is installed in the buildbox image
- Install Rust toolchains for each arch we support
- Use openssl's vendor feature to ensure we always link a static lib
- Automatically include RDP client if Rust is detected
In some cases, it's possible for a package to be marked as a test
failure even if no tests inside it have failed. The motivating example
for this change is a timeout: a test overshooting the allotted timeout
is considered by go test to be a package-level failure, even if no
tests inside the package are considered failures.
This led to cases where the user would see an "All tests passed"
message from the go test filter, but still mysteriously fail the make
step.
To address this, the test renderer now:
* treats package-level pass/fail/skip events as first-class citizens
and includes them in its event count,
* tracks the cached test output at both a package and individual test
level, and
* displays the whole package output if a package is marked as failed,
but only if there is no obvious failed test top account for the
package-level failure.
This patch also removes the json files created by the unit tests, as
they are not yet needed for anything.
