WARNING: Due to issues with the windows drone executor's poor escaping when it echoes commands, I have moved the error message functionality into the PS build functions in build.assets/windows/build.ps1. This means that any failures that occur during the code checkout step will not be reported.
I'm not sure that this is the correct tradeoff, but it may well suffice for now.
Applies linters to legacy protos and adds a few additional Makefile targets to
make it easier to manage protos locally.
Proto linters now run in CI.
#15187
* Apply linters to legacy protos
* Handle new folders in genproto.sh, reset gen/proto if exists
* Lint and format lib/teleterm as part of protos/all
Uses Drone to build Teleport Connect for Windows on a Native
Windows builder.
This PR adds 2 pipelines to the Drone YAML:
1. `push-build-native-windows-amd64`: Invoked on a push to master,
branch/v*, etc., and asserts that Teleport Connect can be built, and
2. `build-native-windows-amd64`: Invoked when a branch tag is
committed to the teleport Repo. Builds Teleport Connect and uploads
it to dronestorage
These builds are run on a native windows builder (as opposed to tsh,
which is built in a linux environment and cross-compiled for Windows)
Change the proto layout of `api/` to a more standard setup, allowing the use of
modern tools (like Buf) to format/lint (and maybe, one day, generate sources).
The new layout looks like this:
``` api/ proto/ <- root of protos and proto imports teleport/ <- base
package for Teleport protos (akin to "google/" or "gogoproto/") legacy/ <- root
of "legacy" protos (most linters disabled) client/ proto/ types/ events/
webauthn/ wrappers/ ```
Non-legacy `api/` protos are expected to follow this layout:
``` api/ proto/ teleport/ mynewpackage/ <- package name v1/ <- protos
explicitly versioned gen/ proto/ <- root for generated sources
(multi-language possible, separate from hand-written code) go/ mynewpackage/ v1
<- generate Go sources go here. ```
Some outstanding issues, like lack of `go_package` declarations and non-standard
import paths (`import "github.com/gravitational/teleport/.../some.proto"`) are
fixed.
Legacy protos still have irregular package declarations. It's possible to fix
that, but it's a bit harder to reason about, as generated sources change in
possibly-meaningful ways.
Future iterations could change legacy packages to match the directory structure
and apply a similar change to protos within lib/ packages, but this seems
sufficient for a first step.
* Add Buf to buildbox
* Unify API protos under Buf
* Fix proto generation
* Reformat protos
* Update generated protos
* Generate protos using Buf
* Appease linter
* Review: make sure gogo protobuf versions are in sync
* Clean leftovers from previous attempts
* Fix operator/Makefile
* Rename internal make gRPC targets to `*/host`
* Sort `make fix-license` targets (nit)
* Add proof of concept of Connect pipeline
The proof of concept includes a lot of copy-pasted lines which will get
cleared up in subsequent commits.
* Extract copying artifacts into separate functions
The tag pipeline no longer needs to worry about Connect artifacts.
* Reuse steps to install & cleanup toolchains
* Share toolchain configuration commands between pipelines
* Share build commands among different pipelines
* Download webapps only if a pipeline builds Connect
As seen by the changes to .drone.yml, this removes unnecessary webapps
clones from these tag pipelines: build-darwin-amd64, build-darwin-amd64-pkg,
build-darwin-amd64-pkg-tsh. None of them needs webapps to function anymore
and the pkg pipelines never needed webapps in the first place.
In order to do so, we add a new make target:
make teleterm
This (temporarily) assumes that the gravitational/webapps repo is
cloned at the right version as a sibling to the teleport repo.
(We'll be able to get rid of this when we merge webapps into Teleport)
Additionally, update dronegen to include the name of the calling
function that generated the snippet instead of the line number.
This gets rid of lots of superfluous diffs in the generated
.drone.yml file.
Lastly, rewrite the Go program for getting the right webapps version
in bash, because Go is not available at this step of the drone pipeline.
Co-authored-by: Grzegorz Zdunek <grzegorz.zdunek@goteleport.com>
Adds the GCB build yaml for controlling the build, and updates the test script
to work in both the GCB environment and on a local dev machine.
Also changes the centos buildbox to leave the default user as root. When
GCB mounts the workspace into the container, the source code is owned
by root, and there is no way to change this. This means that the build will
fail when the non-root user specified in the build image attempt to write files
into the workspace. Setting the root user fixes this.
See-Also: #15186
Now that we have automation in place for updating the webassets
repo, this script no longer needs to build webassets. Instead,
it just updates the webassets submodule to point at the tip of
whatever branch is specified and opens the Teleport PR.
This is a twofold change with the aim of reducing possible pains with the tsh
installer.
- Dropping the version number from "tsh.app" makes it more alike other apps
(including Connect)
- Making the installer non-relocatable makes it easy to reason about (and
ensures our postinstall script is correct!)
A relocatable installer will look for the app in places other the specified
install path, according to the bundle ID. This means that if the user moves or
renames the app, the installer will overwrite it no matter where it is. It also
means our path assumptions can be wrong.
Note that the installer itself is still numbered, so it won't break Houston or
change the downloads page.
Ports used by the unit tests have been allocated by pulling them out of a list, with no guarantee that the port is not actually in use. This central allocation point also means that tests cannot be split into separate packages to be run in parallel, as the ports allocated between the various packages will be allocated multiple times and end up intermittently clashing.
There is also no guarantee, even when the tests are run serially, that the ports will not clash with services already running on the machine.
This patch (largely) replaces the use of this centralised port allocation with pre-created listeners injected into the test via the file descriptor import mechanism use by Teleport to pass open ports to child processes.
There are still some cases where the old port allocation system is still in use. I felt this was already getting beyond the bounds of sensibly reviewable, so I have left those for a further PR after this.
See-Also: #12421
See-Also: #14408
- Enables the docker BuildKit in an attempt to speed up builds
- Trims slightly under 2GB off image size
- Break more dependencies out into separate build stages
- Adds some simple supply-chain protections for dependencies sourced
via git. The Docker build now checks that the commit SHAs are what
we expect, and not just assume that the tags haven't changed.
- Moves the `cbindgen` build to a stage to avoid pulling in extra
dependencies not needed for the Teleport build
- Combines the `gcloud` and firestore emulator install into one step to
reduce the layer count.
- Ports some of the above the Centos7 Dockerfile.
Drop the `v` from the tsh installer version number, which was inadvertently
changed by #12751. Makes the installer reappear as a download option in Houston.
Note that the final .app name still has the `v`. Ie:
* tsh-10.0.0-dev.pkg (installer)
* tsh-10.0.0-dev.pkg.sha256 (installer hash)
* tsh-v10.0.0-dev.app (Application package)
This code was unmaintained, created issues with our build system,
and didn't actually match the behavior of Teleport's RBAC engine.
We will revisit this functionality in the future when we investigate
"acess policies as code."
Attempt to detect builder environment inconsistencies by compiling a toy FIDO2
program - if this fails, then clear the cache and try again.
Builders are sometimes getting into inconsistent states, this should help
avoiding manual intervention in order to fix them.
Recent Rust dependency upgrades include a newer version of prost.
This new version no longer ships embedded protoc binaries, and
instead tries to build protoc from source. This would require us
to install cmake on our buildboxes. We want to avoid this and
instead leverage the version of protoc already installed.
This change was made to the standard buildbox, but the CentOS 7
buildbox was missed.
Additionally, I noticed that Rust was installed in
Dockerfile-centos7-fips, but not in Dockerfile-fips, which means
the FIPS binaries have different functionality depending on which
version you use. To correct this, I removed Rust from the CentOS 7
FIPS builds (since the Rust features are not FIPS compliant anyway).
Switch from `make release-amd64` to make release-windows in Drone builds, making
release builds similar to "regular" builds (that already use
`make release-windows-unsigned`).
Fixes current woes caused by FIDO2=yes in Windows release builds. (Note that
ARCH is implied by the build.)
* Use `make release-windows` on Drone, make it similar to `make release`
* Update .drone.yaml
After recent changes in #12257, Dockerfile-teleterm was made to accept
NODE_VERSION passed from a build arg.
The problem is that NODE_VERSION used to follow the format of `vX.Y.Z`,
while NODE_VERSION in build.assets/Makefile follows the `X.Y.Z` format.
This commit adds the missing `v`s to NODE_URL and NODE_PATH.
This commit updates drone to build Teleport Connect by:
* cloning `gravitational/webapps` as a sibling directory to
gravitational/teleport
* checkout out the right version of webapps by running a simple
Go program (this step is only necessary until we move webapps
into the teleport repo)
* Running the Teleport Connect build and copying artifacts
Code signing should run on tag builds automatically as part the
electron build, assuming the Apple Account credentials are
properly loaded into the keychain.
Notarization will also happen automatically if both
`$APPLE_USERNAME` and `$APPLE_PASSWORD` are set.
In order to make the above happen, this patch also includes:
* Installing and removing a per-build Node instance in the
toolchain directory on Darwin
* Moving the toolchain temporary directory out of ~/ and into /tmp.
Drone usually sets `$HOME` to a temporary directory for each build,
but unfortunately we need it to point to the actual build user's
home directory in order for the notarisation tooling to find the
right keychain. Having $HOME point to a long-lived directory risks
both pollution from build detritus and builds stomping on one another.
In an in an attempt to isolate the builds from each other and protect
`~build` as best we can, as much of the build state as possible
(including ephemeral toolchains) has been moved under `/tmp`.
Co-authored-by: Trent Clarke <trent@goteleport.com>
Set the macOS deployment target, ensuring that statically linked libfido2 `tsh`
builds run correctly on older macOS versions.
#9160
* Consistently set macOS min version
* Bump min macOS version to 10.13
Since #12794 we now build `tsh` binaries with touch ID capabilities. This calls
for a more sophisticated mechanism to determine if touch ID functions should be
enabled, as compile-time support only is not enough.
I've added the following checks, on top of compile-time / `touchid` build tag:
Binary is signed
Binary has entitlements
Machine is touch ID capable
Machine has a Secure Enclave
Put together this give us a much better proxy on whether to enable touch ID.
I've also added the `tsh touchid diag` command, mentioned in the Passwordless
macOS RFD (see
https://github.com/gravitational/teleport/blob/master/rfd/0054-passwordless-macos.md#tsh-support-commands).
#9160
* Improved touch ID availability and diagnostics
* Add the `tsh touchid diag` command
* Set min macOS version to 10.12 (macOS Sierra)
Add a script to build libfido2 (and its dependencies) on macOS and enable FIDO2
static builds.
I decided to build all dependencies instead of pulling from Homebrew for a few
reasons:
1. There is no libcbor.a in a brew package
2. This captures library versions within the Teleport source code, allowing us
to build binaries against different versions of libfido2 (and its
dependencies).
I've also bumped libfido2 to 1.11.0. I've been running it locally and we are
still pre-release, so it seems like a good time to do it.
(See https://developers.yubico.com/libfido2/Release_Notes.html.)
#9160
* Build libfido2 and dependencies for macOS
* Build tsh with static fido2 on Drone
* Bump libfido2 versions in all builds
* Attempt to appease linters
* Use temp dirs inside LIB_CACHE
* Move LIB_CACHE outside of HOME
HOME is reassigned in macOS builders, but we want a "stable" cache
directory. /tmp is used by build-package.sh and build-pkg-tsh.sh.
* Rename script to build-fido2-macos.sh
* Regenerate Drone files
Changes how `make pkg-tsh` works so instead of building an installer for the
`tsh` binary, placed under `/usr/local/bin`, we install an app to
`/Applications/tsh-vXXX.app` and link its `tsh` binary to `/usr/local/bin`.
The app shell is necessary to distribute a provisioning profile along with the
signed/entitled/notarized binary. All of that is required for Touch ID to work.
Naked `tsh` binaries are unable to use Touch ID, even if built with the correct
build tags.
I've elected to split the logic from `build-package.sh` into a separate script -
it already does too much as-is. `build-pkg-tsh.sh` is more idiomatic, clears
additional `shellcheck` rules and is easier to dry-run.
#9160
* Build macOS installer for tsh.app
* Add resources to build the tshdev app
Moved from e/
* Add resources to build the tsh app (prod)
* Use production values
* Remove 'tsh' mode from build-package.tsh
* Appease buildbox linter
* Clarify one-time setup
