* POC for Helm unit tests
This uses https://github.com/vbehar/helm3-unittest to define
expectations of our helm templates
* Test that enterprise is configured correctly
* Added tests for teleport-cluster
* Added tests for teleport-kube-agent
* Removed tests for teleport chart
* Add tests for teleport-cluster Deployment
* Run shorter tests first
* Fix Docker plugin installation and add update-helm-snapshots target
* Add README
* Fix lint syntax error and add some missing linters
* Add missing ImagePullPolicy to Deployment and StatefulSet
* Add Deployment tests for teleport-kube-agent
* Fix replicaCount logic
* Add clarification to values
* Add StatefulSet suite for teleport-kube-agent
* Update snapshots after merge with master
* Helm tests are quicker than bash tests
* Add tests for extraEnv
* Random space
* Tidy up formatting of multiple tests
* [debug] List helm plugins and directories
* Special case Helm linting when running in CI
* Make trailing line breaks consistent
* Special case Helm linting when running in CI
* Add contribution guidelines for Helm charts
* Add contribution guidelines to READMEs
* Deprecate old charts
* Typo
* Spacing
* Clarification
* Update examples/chart/CONTRIBUTING.md
* Don't erroneously set extraEnv for initContainers
* Rename update-helm-snapshots -> test-helm-update-snapshots for clarity
Co-authored-by: Gus Luxton <gus@goteleport.com>
Co-authored-by: Roman Tkachenko <roman@goteleport.com>
API_IMPORT_PATH is consistently being resolved as an empty string, breaking
proto generation.
Since the path is fixed, it seems simpler to read api/go.mod and do away with
the Go program.
* Explicitly set API_IMPORT_PATH
* Delete the print-import-path program
* Read api module from api/go.mod, push variables to target
* Add certificate renewal bot
This adds a new `tbot` tool to continuously renew a set of
certificates after registering with a Teleport cluster using a
similar process to standard node joining.
This makes some modifications to user certificate generation to allow
for certificates that can be renewed beyond their original TTL, and
exposes new gRPC endpoints:
* `CreateBotJoinToken` creates a join token for a bot user
* `GenerateInitialRenewableUserCerts` exchanges a token for a set of
certificates with a new `renewable` flag set
A new `tctl` command, `tctl bots add`, creates a bot user and calls
`CreateBotJoinToken` to issue a token. A bot instance can then be
started using a provided command.
* Cert bot refactoring pass
* Use role requests to split renewable certs from end-user certs
* Add bot configuration file
* Use `teleport.dev/bot` label
* Remove `impersonator` flag on initial bot certs
* Remove unnecessary `renew` package
* Misc other cleanup
* Do not pass through `renewable` flag when role requests are set
This adds additional restrictions on when a certificate's `renewable`
flag is carried over to a new certificate. In particular, it now also
denies the flag when either role requests are present, or the
`disallowReissue` flag has been previously set.
In practice `disallow-reissue` would have prevented any undesired
behavior but this improves consistency and resolves a TODO.
* Various tbot UX improvements; render SSH config
* Fully flesh out config template rendering
* Fix rendering for SSH configuration templates
* Added `String()` impls for destination types
* Improve certificate renewal logging; show more detail
* Properly fall back to default (all) roles
* Add mode hints for files
* Add/update copyright headers
* Add stubs for tbot init and watch commands
* Add gRPC endpoints for managing bots
* Add `CreateBot`, `DeleteBot`, and `GetBotUsers` gRPC endpoints
* Replace `tctl bot (add|rm|ls)` implementations with gRPC calls
* Define a few new constants, `DefaultBotJoinTTL`, `BotLabel`,
`BotGenerationLabel`
* Fix outdated destination flag in example tbot command
* Bugfix pass for demo
* Fixed a few nil pointer derefs when using config from CLI args
* Properly create destination if `--destination-dir` flag is used
* Remove improper default on CLI flag
* `DestinationConfig` is now a list of pointers
* Address first wave of review feedback
Fixes the majority of smaller issues caught by reviewers, thanks all!
* Add doc comments for bot.go functions
* Return the token TTL from CreateBot
* Split initial user cert issuance from `generateUserCerts()`
Issuing initial renewable certificate ended up requiring a lot of
hacks to skip checks that prevented anonymous bots from getting
certs even though we'd verified their identity elsewhere (via token).
This reverts all those hacks and splits initial bot cert logic into a
dedicated `generateInitialRenewableUserCerts()` function which should
make the whole process much easier to follow.
* Set bot traits to silence log messages
* tbot log message consistency pass
* Resolve lints
* Add config tests
* Remove CreateBotJoinToken endpoint
Users should instead use the CreateBot/DeleteBot endpoints.
* Create a fresh private key for every impersonated identity renewal
* Hide `config` subcommand
* Rename bot label prefix to `teleport.internal/`
* Use types.NewRole() to create bot roles
* Clean up error handling in custom YAML unmarshallers
Also, add notes about the supported YAML shapes.
* Fetch proxy host via gRPC Ping() instead of GetProxies()
* Update lib/auth/bot.go
Co-authored-by: Zac Bergquist <zmb3@users.noreply.github.com>
* Fix some review comments
* Add renewable certificate generation checks (#10098)
* Add renewable certificate generation checks
This adds a new validation check for renewable certificates that
maintains a renewal counter as both a certificate extension and a
user label. This counter is used to ensure only a single certificate
lineage can exist: for example, if a renewable certificate is stolen,
only one copy of the certificate can be renewed as the generation
counter will not match
When renewing a certificate, first the generation counter presented
by the user (via their TLS identity) is compared to a value stored
with the associated user (in a new `teleport.dev/bot-generation`
label field). If they aren't equal, the renewal attempt fails.
Otherwise, the generation counter is incremented by 1, stored to the
database using a `CompareAndSwap()` to ensure atomicity, and set on
the generated certificate for use in future renewals.
* Add unit tests for the generation counter
This adds new unit tests to exercise the generation counter checks.
Additionally, it fixes two other renewable cert tests that were
failing.
* Remove certRequestGeneration() function
* Emit audit event when cert generations don't match
* Fully implement `tctl bots lock`
* Show bot name in `tctl bots ls`
* Lock bots when a cert generation mismatch is found
* Make CompareFailed respones from validateGenerationLabel() more actionable
* Update lib/services/local/users.go
Co-authored-by: Nic Klaassen <nic@goteleport.com>
* Backend changes for tbot IoT and AWS joining (#10360)
* backend changes
* add token permission check
* pass ctx from caller
Co-authored-by: Roman Tkachenko <roman@goteleport.com>
* fix comment typo
Co-authored-by: Roman Tkachenko <roman@goteleport.com>
* use UserMetadata instead of Identity in RenewableCertificateGenerationMismatch event
* Client changes for tbot IoT joining (#10397)
* client changes
* delete replaced APIs
* delete unused tbot/auth.go
* add license header
* don't unecessarily fetch host CA
* log fixes
* s/tunnelling/tunneling/
Co-authored-by: Zac Bergquist <zmb3@users.noreply.github.com>
* auth server addresses may be proxies
Co-authored-by: Zac Bergquist <zmb3@users.noreply.github.com>
* comment typo fix
Co-authored-by: Zac Bergquist <zmb3@users.noreply.github.com>
* move *Server methods out of auth_with_roles.go (#10416)
Co-authored-by: Tim Buckley <tim@goteleport.com>
Co-authored-by: Zac Bergquist <zmb3@users.noreply.github.com>
Co-authored-by: Tim Buckley <tim@goteleport.com>
Co-authored-by: Roman Tkachenko <roman@goteleport.com>
Co-authored-by: Tim Buckley <tim@goteleport.com>
Co-authored-by: Zac Bergquist <zmb3@users.noreply.github.com>
Co-authored-by: Nic Klaassen <nic@goteleport.com>
Co-authored-by: Roman Tkachenko <roman@goteleport.com>
Co-authored-by: Zac Bergquist <zmb3@users.noreply.github.com>
* Address another batch of review feedback
* Addres another batch of review feedback
Add `Role.SetMetadata()`, simplify more `trace.WrapWithMessage()`
calls, clear some TODOs and lints, and address other misc feedback
items.
* Fix lint
* Add missing doc comments to SaveIdentity / LoadIdentity
* Remove pam tag from tbot build
* Update note about bot lock deletion
* Another pass of review feedback
Ensure all requestable roles exist when creating a bot, adjust the
default renewable cert TTL down to 1 hour, and check types during
`CompareAndSwapUser()`
Co-authored-by: Zac Bergquist <zmb3@users.noreply.github.com>
Co-authored-by: Nic Klaassen <nic@goteleport.com>
Co-authored-by: Roman Tkachenko <roman@goteleport.com>
In order to make it easier to do post-analysis in test logs, this patch makes the CI build save the JSON-formatted logs into a Google Cloud Storage bucket.
GCB has a built-in artifact storage system, but unfortunately it only stores artifacts on successful builds. Given that we're interested on doing post-analysis on failed builds, this is unhelpful. The build scripts instead use the GCP API to explicitly upload the test logs to a bucket.
This patch
* Adds an artifact uploader using the GCP Storage API
* Updates the build yaml files to point to the appropriate artifacts & bucket
* Updates the makefile to save JSON logs to $TELEPORT/test-logs when running tests
* Adds entries to .gitignore to not automatically commit test reports
The existing test rendering filter formats the JSON outut for human consumption, meaning that we can both save the JSON logs and that humans manually running tests with via make will still get an intelligible report.
In addition to the above, this patch also:
* standardizes the build scripts on `logrus` for logging, as this was used by some dependencies.
* adds a self-test to the CI code, run as part of `make test`
* Begin Rust implementation of MS_RDPECLIP
Includes encoding/decoding support for the Clipboard Capabilities
PDU, as specified in section 2.2.2.1.
* Add support for the format list PDU (and response)
This message is the notification from one endpoint to another that
the data on the local clipboard has been updated.
* Start integrating cliprdr client
We're receiving the start of the initialization flow from the server,
but it doesn't like our response just yet. Probably missing a header
of some kind.
* Allow providing flags for the channel PDU header
This didn't appear to fix anything, but we will want to be able
to adjust those flags based on the message we're sending.
* Continue to flesh out cliprdr client
- Add docs
- Build out handlers for remaining messages
- Set the SHOW_PROTOCOL channel header flag
(this was key to getting the RDP server to respond to us)
* Respond to format data request
* Add lint and test targets for Rust code
And fix lint warnings
* Suppress collapsible_if warning for role_tester
This is a giant crepe macro, so the lint warning isn't even accurate.
* Hook up Rust/Go callbacks for passing clipboard data
* Address first round of review feedback
* Address review feedback
* Mark RFD 49 as implemented
* Add more lint coverage
golanglint-ci doesn't pick up subdirectories with their own go.mod
which left certain directories unlinted. To get around this we can
run golanglint-ci directly against those submodules.
In Rust 1.58, deriving Debug no longer counts as using a struct's
fields, so we need to allow dead_code for our structs that implement
RDP protocols. (Just because we don't use the fields doesn't mean
we shoudln't decode them)
The workspace includes both parts of the project that use Rust.
(The roletester and RDP client).
This has several advantages:
- Rust Analyzer will work on the codebase as a whole, so we get
nice development features without needing to open the Rust
projects separately
- Dependencies are resolved at the project level, ensuring that
role tester and RDP client use the same version of common
dependencies.
Starting with the Teleport 9 release, we will be versioning the
API module. This change ensures that the generated protobuf code
imports the correct version of the API by:
- introducing a small new command to print the correct version
- adding import rewrite rules to the protoc invocation
Linting for non-go files was accidentally dropped in the transition to
GCB (sorry!). This patch restores linting for non-go files and fixes
any lint failures that have crept in during the interim.
Actually tracking down the cause of a failure in the integration tests can
be hard:
* It's hard to get an overall summary of what failed
* The tests sometimes emit no output before timing out, meaning any
diagnostic info is lost
* The emitted logs are too voluminous for a human to parse
* The emitted logs can present information out of order
* It's often hard to tell where the output from one test ends
and the next one begins
This patch attempts to address these concerns without attempting to rewrite
any of the underlying teleport logging.
* It improves the render-tests script to (optionally) report progress per-
test, rather than on a per-package basis. My working hypothesis on the
tests that time out with no output is that go test ./integration is
waiting for the entire set of integration tests tests to be complete
before reporting success or failure. Reporting on a per-test cycle gives
faster feedback and means that any timed-out builds should give at least
some idea of where they are stuck.
* Adds the render-tests filter to the integration and integration-root make
targets. This will show an overall summary of test results, as well as
- Discarding log output from passing tests to increase signal-to-noise
ratio, and
- Strongly delimiting the output from each failed test, making failures
easier to find.
* Removes the notion of a failure-only logger in favour of post-processing
the log events with render-tests. The failure-only logger catches log
output from the tests and only forwards it to the console if the test
fails. Unfortunately, not all log output is guaranteed to pass through
this logger (some teleport packages do not honour the configured logger,
and reports from the go race detector certainly don't), meaning some
output is presented at the time it happens, and other output is batched
and displayed at the end of the test. This makes working out what
happened where harder than it need be.
In addition, this patch also promotes the render-tests script into a fully-
fledged program, with appropriate makefile targets, make clean support, etc.
It is now also more robust in the face on non-JSON output from go test
(which happens if a package fails to compile).
* Ensure that slice.pb.go is generated by `make grpc`
* Clean up `make grpc`
* Disable the test target rules in Makefile when running inside the devbox
- Ensure that the protoc include directory is readable by all users
- Switch back to the root user by default
Either of these changes would have fixed the issue on their own,
but I decided to include both as GRPC should be readable by non-root
users, and I wanted to preserve the original behavior of running
as root unless the $(NOROOT) flags are specified.
Additionally: clarify comments on the make targets, which are
confusingly named, and stop installing goimports since it seems
it was never used.
Part of this change is implementing a "no secrets" policy for CI. Given that
we have to support CI for arbitrary external contributors, and
it is easy to craft a malicious PR that exfiltrates secrets during a CI build
any test that runs under CI must be able to do so without any injected secrets.
This means that several of the test we currently run under Drone will not be run on GCB, at least as part of the regular CI. The plan is to create a separate task that periodically runs tests that require external credentials (e.g. Kube tests, various backend data stores, etc.) in a more secure way and report failures asynchronously. And while these tests will not run under CI, the should still be built under CI so that required changes are caught during review.
* update deps in manifest and lockfile
* fixes and updates to docker and profiles
* lint rust
* fix typo
* resolve clippy lints
* fix typo
* mark risk functions unsafe
* fmt + clean up the last lints#
* verify lockfile up to date
* disable lto since it doesn't work with two rust libs
* merge lock check and lint
* Add missing license header to Rust files
And update Makefile to ensure they are checked
Co-authored-by: Zac Bergquist <zac.bergquist@goteleport.com>
Download Rust and Go per-build to ensure that the right version is used
and that builds do not step on each other.
Also rungs cbindgen in quiet mode to suppress the annoying output it
spews for non-public symbols.
The race condition detector is being tripped by a concurrent `Write` and
`Close` in the `PipeNetCon` in several integration tests. This is a naive
fix to serialize the write and close operations to resolve the race
condition.
The affected tests were also not handling asynchronous error reporting
correctly (i.e. it's not legal to call `require.XYZ()` from a goroutine
other than the one executing the test function.). This patch introduces
some plumbing to marshal asynchronous errors back into the main test
routine before failing the test.
- Ensure Rust is installed in the buildbox image
- Install Rust toolchains for each arch we support
- Use openssl's vendor feature to ensure we always link a static lib
- Automatically include RDP client if Rust is detected
In some cases, it's possible for a package to be marked as a test
failure even if no tests inside it have failed. The motivating example
for this change is a timeout: a test overshooting the allotted timeout
is considered by go test to be a package-level failure, even if no
tests inside the package are considered failures.
This led to cases where the user would see an "All tests passed"
message from the go test filter, but still mysteriously fail the make
step.
To address this, the test renderer now:
* treats package-level pass/fail/skip events as first-class citizens
and includes them in its event count,
* tracks the cached test output at both a package and individual test
level, and
* displays the whole package output if a package is marked as failed,
but only if there is no obvious failed test top account for the
package-level failure.
This patch also removes the json files created by the unit tests, as
they are not yet needed for anything.
- Don't assume an explicit $GOPATH is set
- Remove golint from linters - it's been deprecated for over a year
and golangci-lint prints a warning instead of running it.
This change makes the Teleport unit tests write a JSON log of all the tests that we run `make test-go`. It also includes a parsing script that will render the JSON log into a human-readable format during the test run, so that the working developer can see what the tests are doing without having to wade through JSON.
The log output is somewhat of a cross between the standard go test output, with pytest-style summaries at the end.
I have limited the realtime report to package-level results (a package-level skip result means no tests file were found). It's trivial to output each test as it comes in, if that works better (but at ~1500 tests, it's a lot).
This is another step towards getting better visibility on our test suite. The idea is that we will eventually collect these test reports as build artifacts for further analysis.
* PIV authentication for RDP
This uncomfortably large change fully implements smartcard PIV
authentication for RDP clients using the Teleport CA:
- PIV applet implementation in emulated RDP smartcard
- generating Windows-compatible certificates using Teleport CA with a
dedicated RPC
- generating dummy CRLs for Teleport CA and publishing it via LDAP
The CRLs are required by Windows for any smartcard login certificate, we
can't avoid that. But we can avoid making it public: the CRL can live in
ActiveDirectory instead of a public endpoint of a Teleport service.
Here, we use LDAP to publish the CRL on startup, valid for a year.
There are a few unhandled cases in the current implementation:
- LDAP server certificate is not validated when upgrading to TLS
- multiple active CAs (with HSMs) are not supported, only one CRL is
published
- CA rotation is not supported, CRL is not re-published on rotation
All of the above issues will be handled in future PRs as this one is
already too large.
* Address review feedback
* Fix linter errors
* Sign tsh.exe on tag builds
This adds a Makefile step to sign tsh.exe when the
`$WINDOWS_SIGNING_CERTIFICATE` env var is set to a base64-encoded
pkcs12 code signing certificate. The certificate must not be password
protected.
This includes a sample cert (`cert-dummy.pfx`) for CI pipeline
testing. It should be removed in any eventual PR, along with the
other modifications to the drone pipeline. The cert is imported into
the environment in the `Makefile` for testing purposes; in practice
it will be imported from a secure secret store (drone secrets, etc).
* Improve Windows code signing
- Split signing into a separate step; `release-windows-unsigned` now
performs the build, `release-windows` signs the binary.
- Require `release-windows` to successfully generate a signed
binary.
- Clearly mark unsigned binaries and archives as such.
- Guard against stdout secret leakage in Makefiles.
- Move temporary cert data from Makefile into dronegen to test
full pipeline.
* Use an invalid cert string for testing purposes.
* Pass certs to the build process via a statically named file
Signed Windows builds now depend on a `.gitignore`'d
`windows-signing-cert.pfx` at the root of the source directory. This
should ease testing and help avoid accidental secret leakage.
* Use production secret
* Remove windows-signing-cert.pfx before continuing to the next step
Additionally, fix variable reference as the bracket syntax does not
seem to play nice with Drone.
* Update .gitignore
Co-authored-by: Andrew Lytvynov <andrew@goteleport.com>
Co-authored-by: Andrew Lytvynov <andrew@goteleport.com>
Introduce new make targets to check and add license headers to files
("make lint-license" and "make fix-license"). License checking is now a part of
"make lint" as well.
Initial attempts used goheader, but it caused "make lint-go" to become about 9x
slower (if not more), plus it only targets go files. Google's addlicense is fast
enough and targets however many file types we want.
Existing files that were missing licenses got the header added, using the
current year as the license date.
* Introduce lint-license and fix-license make targets
* Ignore generated files
* Add license to go files
* Replace irregular licenses with standard copyright/license
* Add license to proto files
* Install addlicense in build.assets Dockerfile
Created an access tester for troubleshooting access related issues with Teleport RBAC system. This access tester allows admins to answer questions like:
Can user alice SSH into a node node-1 as root?
If not, which role(-s) prevents access?
Which roles allow access to production as login admin?
We were using switches specific to GNU coreutils for the api symlink.
These aren't supported on macOS with its BSD variants, so use a more
primitive syntax that will work on both platforms.
Reduced Teleport shared library dependencies on libbpf, libelf, libz.
For libbpf, switched to forked version that does not rely on "fmemopen"
which brings in a glibc 2.22 dependency. This allows binaries built on
Ubuntu 18.04 box to run on CentOS 7 as well.
For libelf and libz (which libbpf uses), the build process has been
updated to statically link both of them during the build process.
