* Add piv build dependencies.
- Add LIBPCSCLITE build tag.
- Add libpcsclite static linking using gravitational/pcsc fork.
- Enable use of dynamic pcsc library with LIBPCSCLITE=dynamic.
- Refactor CGOFLAG in Makefile.
- Update Centos7 Dockerfile and drone.
* Refactor RELEASE_MESSAGE for readability. Now produces message like: "RELEASE_MESSAGE=Building with GOOS=linux GOARCH=amd64 REPRODUCIBLE= and with PIV support and without PAM support, FIPS support, BPF support, Windows RDP client, libfido2, Touch ID."
Co-authored-by: Jakub Nyckowski <jakub.nyckowski@goteleport.com>
Uses Drone to build Teleport Connect for Windows on a Native
Windows builder.
This PR adds 2 pipelines to the Drone YAML:
1. `push-build-native-windows-amd64`: Invoked on a push to master,
branch/v*, etc., and asserts that Teleport Connect can be built, and
2. `build-native-windows-amd64`: Invoked when a branch tag is
committed to the teleport Repo. Builds Teleport Connect and uploads
it to dronestorage
These builds are run on a native windows builder (as opposed to tsh,
which is built in a linux environment and cross-compiled for Windows)
Change the proto layout of `api/` to a more standard setup, allowing the use of
modern tools (like Buf) to format/lint (and maybe, one day, generate sources).
The new layout looks like this:
``` api/ proto/ <- root of protos and proto imports teleport/ <- base
package for Teleport protos (akin to "google/" or "gogoproto/") legacy/ <- root
of "legacy" protos (most linters disabled) client/ proto/ types/ events/
webauthn/ wrappers/ ```
Non-legacy `api/` protos are expected to follow this layout:
``` api/ proto/ teleport/ mynewpackage/ <- package name v1/ <- protos
explicitly versioned gen/ proto/ <- root for generated sources
(multi-language possible, separate from hand-written code) go/ mynewpackage/ v1
<- generate Go sources go here. ```
Some outstanding issues, like lack of `go_package` declarations and non-standard
import paths (`import "github.com/gravitational/teleport/.../some.proto"`) are
fixed.
Legacy protos still have irregular package declarations. It's possible to fix
that, but it's a bit harder to reason about, as generated sources change in
possibly-meaningful ways.
Future iterations could change legacy packages to match the directory structure
and apply a similar change to protos within lib/ packages, but this seems
sufficient for a first step.
* Add Buf to buildbox
* Unify API protos under Buf
* Fix proto generation
* Reformat protos
* Update generated protos
* Generate protos using Buf
* Appease linter
* Review: make sure gogo protobuf versions are in sync
* Clean leftovers from previous attempts
* Fix operator/Makefile
* Rename internal make gRPC targets to `*/host`
* Sort `make fix-license` targets (nit)
* Add proof of concept of Connect pipeline
The proof of concept includes a lot of copy-pasted lines which will get
cleared up in subsequent commits.
* Extract copying artifacts into separate functions
The tag pipeline no longer needs to worry about Connect artifacts.
* Reuse steps to install & cleanup toolchains
* Share toolchain configuration commands between pipelines
* Share build commands among different pipelines
* Download webapps only if a pipeline builds Connect
As seen by the changes to .drone.yml, this removes unnecessary webapps
clones from these tag pipelines: build-darwin-amd64, build-darwin-amd64-pkg,
build-darwin-amd64-pkg-tsh. None of them needs webapps to function anymore
and the pkg pipelines never needed webapps in the first place.
In order to do so, we add a new make target:
make teleterm
This (temporarily) assumes that the gravitational/webapps repo is
cloned at the right version as a sibling to the teleport repo.
(We'll be able to get rid of this when we merge webapps into Teleport)
Additionally, update dronegen to include the name of the calling
function that generated the snippet instead of the line number.
This gets rid of lots of superfluous diffs in the generated
.drone.yml file.
Lastly, rewrite the Go program for getting the right webapps version
in bash, because Go is not available at this step of the drone pipeline.
Co-authored-by: Grzegorz Zdunek <grzegorz.zdunek@goteleport.com>
- Enables the docker BuildKit in an attempt to speed up builds
- Trims slightly under 2GB off image size
- Break more dependencies out into separate build stages
- Adds some simple supply-chain protections for dependencies sourced
via git. The Docker build now checks that the commit SHAs are what
we expect, and not just assume that the tags haven't changed.
- Moves the `cbindgen` build to a stage to avoid pulling in extra
dependencies not needed for the Teleport build
- Combines the `gcloud` and firestore emulator install into one step to
reduce the layer count.
- Ports some of the above the Centos7 Dockerfile.
Recent Rust dependency upgrades include a newer version of prost.
This new version no longer ships embedded protoc binaries, and
instead tries to build protoc from source. This would require us
to install cmake on our buildboxes. We want to avoid this and
instead leverage the version of protoc already installed.
This change was made to the standard buildbox, but the CentOS 7
buildbox was missed.
Additionally, I noticed that Rust was installed in
Dockerfile-centos7-fips, but not in Dockerfile-fips, which means
the FIPS binaries have different functionality depending on which
version you use. To correct this, I removed Rust from the CentOS 7
FIPS builds (since the Rust features are not FIPS compliant anyway).
Switch from `make release-amd64` to make release-windows in Drone builds, making
release builds similar to "regular" builds (that already use
`make release-windows-unsigned`).
Fixes current woes caused by FIDO2=yes in Windows release builds. (Note that
ARCH is implied by the build.)
* Use `make release-windows` on Drone, make it similar to `make release`
* Update .drone.yaml
This commit updates drone to build Teleport Connect by:
* cloning `gravitational/webapps` as a sibling directory to
gravitational/teleport
* checkout out the right version of webapps by running a simple
Go program (this step is only necessary until we move webapps
into the teleport repo)
* Running the Teleport Connect build and copying artifacts
Code signing should run on tag builds automatically as part the
electron build, assuming the Apple Account credentials are
properly loaded into the keychain.
Notarization will also happen automatically if both
`$APPLE_USERNAME` and `$APPLE_PASSWORD` are set.
In order to make the above happen, this patch also includes:
* Installing and removing a per-build Node instance in the
toolchain directory on Darwin
* Moving the toolchain temporary directory out of ~/ and into /tmp.
Drone usually sets `$HOME` to a temporary directory for each build,
but unfortunately we need it to point to the actual build user's
home directory in order for the notarisation tooling to find the
right keychain. Having $HOME point to a long-lived directory risks
both pollution from build detritus and builds stomping on one another.
In an in an attempt to isolate the builds from each other and protect
`~build` as best we can, as much of the build state as possible
(including ephemeral toolchains) has been moved under `/tmp`.
Co-authored-by: Trent Clarke <trent@goteleport.com>
Reinstates Linux/amd64 and Centos7/amd64 builds using libfido2, now hidden
behind an explicit FIDO2 flag (similarly to FIPS).
This PR pulls in gravitational/go-libfido2#4 and adds the required pkg-config
setup so we can perform both dynamic (mostly testing) and static (tsh) builds.
Additionally, pkg-config is now the gateway for whether we run libfido2-related
tests (which should always happen in CI).
#9160
* Re-enable libfido2 builds for amd64 and Centos7
* Use pkg-config to build tsh with libfido2
* Install Centos7 libudev-zero to /usr/local/lib64
* Update gravitational/go-libfido2
* Remove /usr/local/lib from Centos PKG_CONFIG_PATH
Original behaviour did not take effect in CI due to a different entrypoint.
This restores the original behaviour (which will link external links when using make -C build.assets test-docs) but disables the external linting in CI for reliability.
Updates #11940
Build `tsh` with static `libfido2`, `libcbor`,`libcrypto` and `libudev-zero`.
Dockerfiles for buildbox and Centos7 changed. FIPS and macOS to be addressed at
a later date.
Add the `tsh fido2 diag` hidden command for ease of testing.
#9160
* Update go-libfido2 and tidy modules
* Add a fido2 diagnostic command to tsh
* Add a few build artifacts to .gitignore
* Build tsh with static libfido2 in buildbox
* Build tsh with static libfido2 in centos7
* Add a few relevant cmake flags
* Use illiliti/libudev-zero
* Do multi-stage build on centos7, image tweaks
* Add `make enter/centos7`
* s/OFf/OFF/g
The grpc-tools package is needed to generate gRPC files for JavaScript.
However, at the moment it can't be installed on M1 MacBooks because of
missing prebuilt binaries for arm64. [1]
One of them, protoc, is already installed in our buildbox. We still need
to compile grpc_node_plugin from source though. This adds significant
overhead as we need to pull in cmake, build-essential and then about
300 MB of git repos from protocolbuffers/protobuf.
Initially, those Teleterm gRPC were generated within `make grpc` with other
files. M1 users who don't work on Teleterm would not be happy about incurring
that additional overhead, hence I extracted everything into separate target
and Dockerfile.
Teleterm proto files don't depend on any other proto files. Once grpc-tools
adds support for arm64, we'll be able to essentially almost revert this
commit and generate Teleterm gRPC files within `make grpc`.
[1] https://github.com/grpc/grpc-node/issues/1405
* POC for Helm unit tests
This uses https://github.com/vbehar/helm3-unittest to define
expectations of our helm templates
* Test that enterprise is configured correctly
* Added tests for teleport-cluster
* Added tests for teleport-kube-agent
* Removed tests for teleport chart
* Add tests for teleport-cluster Deployment
* Run shorter tests first
* Fix Docker plugin installation and add update-helm-snapshots target
* Add README
* Fix lint syntax error and add some missing linters
* Add missing ImagePullPolicy to Deployment and StatefulSet
* Add Deployment tests for teleport-kube-agent
* Fix replicaCount logic
* Add clarification to values
* Add StatefulSet suite for teleport-kube-agent
* Update snapshots after merge with master
* Helm tests are quicker than bash tests
* Add tests for extraEnv
* Random space
* Tidy up formatting of multiple tests
* [debug] List helm plugins and directories
* Special case Helm linting when running in CI
* Make trailing line breaks consistent
* Special case Helm linting when running in CI
* Add contribution guidelines for Helm charts
* Add contribution guidelines to READMEs
* Deprecate old charts
* Typo
* Spacing
* Clarification
* Update examples/chart/CONTRIBUTING.md
* Don't erroneously set extraEnv for initContainers
* Rename update-helm-snapshots -> test-helm-update-snapshots for clarity
Co-authored-by: Gus Luxton <gus@goteleport.com>
Co-authored-by: Roman Tkachenko <roman@goteleport.com>
Prior to this patch the teleport buildbox version has been tagged with the Go version for the current release. This bit us during the Teleport 9 development cycle, as both Teleport 8 and 9 use the same version of Go but require different versions of Rust, and we were unable to distinguish between the 2 buildbox versions.
At the time, Teleport 8 was individually patched to create a new `teleport8` buildbox tag, decoupling the buildbox version from the Go version. This was never ported into master and now we find the teleport 9 branch sharing the same buildbox tag as master.
This patch forward-ports all the changes made to `branch/v8` and updates them for master, creating a new `teleport10` buildbox tag. The idea is that we will create a new tag for teleport11 at the same time the release branch for Teleport 10 is mad at some point in the future.
Once this is merged, Drone will create and push new buildbox images, which will become available for CI. A subsequent patch will update the CI scripts to use the new `teleport10` buildbox images.
In Rust 1.58, deriving Debug no longer counts as using a struct's
fields, so we need to allow dead_code for our structs that implement
RDP protocols. (Just because we don't use the fields doesn't mean
we shoudln't decode them)
* Update buildbox to use Python3.
* Remove non default rust targets from arm64 image.
* Add ETCD_UNSUPPORTED_ARCH for arm64 to etcd script to allow running etcd on arm64.
* Ensure that slice.pb.go is generated by `make grpc`
* Clean up `make grpc`
* Disable the test target rules in Makefile when running inside the devbox
- Ensure that the protoc include directory is readable by all users
- Switch back to the root user by default
Either of these changes would have fixed the issue on their own,
but I decided to include both as GRPC should be readable by non-root
users, and I wanted to preserve the original behavior of running
as root unless the $(NOROOT) flags are specified.
Additionally: clarify comments on the make targets, which are
confusingly named, and stop installing goimports since it seems
it was never used.
Add new buildboxes for centos7 and centos7-fips.
For now, we will continue to support both CentOS 6 and 7.
Eventually we will drop support for CentOS 6, and the only
supported CentOS builds will be these new CentOS 7 builds.
Fixes#9028
On Apple silicon, Docker will default to building arm64 images.
Some of the packages our image tries to install are only available
for amd64, so the build cannot complete.
This should have no impact for devs running on amd64, and makes it
possible for devs on Apple silion to run build tasks that require
the buildbox (generating protos, for example)
* Sign tsh.exe on tag builds
This adds a Makefile step to sign tsh.exe when the
`$WINDOWS_SIGNING_CERTIFICATE` env var is set to a base64-encoded
pkcs12 code signing certificate. The certificate must not be password
protected.
This includes a sample cert (`cert-dummy.pfx`) for CI pipeline
testing. It should be removed in any eventual PR, along with the
other modifications to the drone pipeline. The cert is imported into
the environment in the `Makefile` for testing purposes; in practice
it will be imported from a secure secret store (drone secrets, etc).
* Improve Windows code signing
- Split signing into a separate step; `release-windows-unsigned` now
performs the build, `release-windows` signs the binary.
- Require `release-windows` to successfully generate a signed
binary.
- Clearly mark unsigned binaries and archives as such.
- Guard against stdout secret leakage in Makefiles.
- Move temporary cert data from Makefile into dronegen to test
full pipeline.
* Use an invalid cert string for testing purposes.
* Pass certs to the build process via a statically named file
Signed Windows builds now depend on a `.gitignore`'d
`windows-signing-cert.pfx` at the root of the source directory. This
should ease testing and help avoid accidental secret leakage.
* Use production secret
* Remove windows-signing-cert.pfx before continuing to the next step
Additionally, fix variable reference as the bracket syntax does not
seem to play nice with Drone.
* Update .gitignore
Co-authored-by: Andrew Lytvynov <andrew@goteleport.com>
Co-authored-by: Andrew Lytvynov <andrew@goteleport.com>
Reduced Teleport shared library dependencies on libbpf, libelf, libz.
For libbpf, switched to forked version that does not rely on "fmemopen"
which brings in a glibc 2.22 dependency. This allows binaries built on
Ubuntu 18.04 box to run on CentOS 7 as well.
For libelf and libz (which libbpf uses), the build process has been
updated to statically link both of them during the build process.
When `SHELL` is not set, `make` defaults to `/bin/sh`.
On systems where `/bin/sh` is an alias for `/bin/bash`, everything works
as expected.
On systems where `/bin/sh` is actually the original Bourne Shell, some
bash-isms don't work. For example: `if [[ condition ]]` results in
`/bin/sh: 1: [[: not found`
* Update Go runtime to 1.16.2 and bump the boringcrypto version correspondingly for linux FIPS builds
* Address review comments
* Don't fail if buildbox image is not present
* Update other go1.15.5 references not yet handled by dronegen
* Build from source on CentOS 6
Co-authored-by: Gus Luxton <gus@goteleport.com>
An extra dockerfile for gRPC generation is extra maintenance burden. It
was also using a really old base image that has a ton of known vulns.
Also, update GOGO_PROTO_TAG to match the version we have vendored via
go.mod.
* Bump Go to 1.15.5
* Downgraded Go version to 1.15.3.
* Sign .drone.yml
Co-authored-by: Russell Jones <rjones@gravitational.com>
Co-authored-by: Gus Luxton <gus@gravitational.com>
* Fix S3 upload path for FIPS artifacts
* Remove RUNTIME from RPM build
* Build full binaries for Docker images with webassets included
* Export correct variables for Enterprise builds
The glibc version requirement imposed by 20.04 doesn't mix well with
centos 7.
Ubuntu 20.04: glibc 2.28
Centos 7: glibc 2.17
As a result, teleport binaries build in the buildbox fail to start.
Going down to 18.04 seems to get us back far enough.
Top-level `make lint` rule that scans everything and a CI-specific rule
for Jenkins.
Currently only enable "unused", since it's reliable. The list will
expand.
Also clean up stragglers that somehow slipped through in #3552.
Updates #3551
Added package cgroup to orchestrate cgroups. Only support for cgroup2
was added to utilize because cgroup2 cgroups have unique IDs that can be
used correlated with BPF events.
Added bpf package that contains three BPF programs: execsnoop,
opensnoop, and tcpconnect. The bpf package starts and stops these
programs as well correlating their output with Teleport sessions
and emitting them to the audit log.
Added support for Teleport to re-exec itself before launching a shell.
This allows Teleport to start a child process, capture it's PID, place
the PID in a cgroup, and then continue to process. Once the process is
continued it can be tracked by it's cgroup ID.
Reduced the total number of connections to a host so Teleport does not
quickly exhaust all file descriptors. Exhausting all file descriptors
happens very quickly when disk events are emitted to the audit log which
are emitted at a very high rate.
Added tarballs for exec sessions. Updated session.start and session.end
events with additional metadata. Updated the format of session tarballs
to include enhanced events.
Added file configuration for enhanced session recording. Added code to
startup enhanced session recording and pass package to SSH nodes.
