Linting for non-go files was accidentally dropped in the transition to
GCB (sorry!). This patch restores linting for non-go files and fixes
any lint failures that have crept in during the interim.
Actually tracking down the cause of a failure in the integration tests can
be hard:
* It's hard to get an overall summary of what failed
* The tests sometimes emit no output before timing out, meaning any
diagnostic info is lost
* The emitted logs are too voluminous for a human to parse
* The emitted logs can present information out of order
* It's often hard to tell where the output from one test ends
and the next one begins
This patch attempts to address these concerns without attempting to rewrite
any of the underlying teleport logging.
* It improves the render-tests script to (optionally) report progress per-
test, rather than on a per-package basis. My working hypothesis on the
tests that time out with no output is that go test ./integration is
waiting for the entire set of integration tests tests to be complete
before reporting success or failure. Reporting on a per-test cycle gives
faster feedback and means that any timed-out builds should give at least
some idea of where they are stuck.
* Adds the render-tests filter to the integration and integration-root make
targets. This will show an overall summary of test results, as well as
- Discarding log output from passing tests to increase signal-to-noise
ratio, and
- Strongly delimiting the output from each failed test, making failures
easier to find.
* Removes the notion of a failure-only logger in favour of post-processing
the log events with render-tests. The failure-only logger catches log
output from the tests and only forwards it to the console if the test
fails. Unfortunately, not all log output is guaranteed to pass through
this logger (some teleport packages do not honour the configured logger,
and reports from the go race detector certainly don't), meaning some
output is presented at the time it happens, and other output is batched
and displayed at the end of the test. This makes working out what
happened where harder than it need be.
In addition, this patch also promotes the render-tests script into a fully-
fledged program, with appropriate makefile targets, make clean support, etc.
It is now also more robust in the face on non-JSON output from go test
(which happens if a package fails to compile).
* Ensure that slice.pb.go is generated by `make grpc`
* Clean up `make grpc`
* Disable the test target rules in Makefile when running inside the devbox
- Ensure that the protoc include directory is readable by all users
- Switch back to the root user by default
Either of these changes would have fixed the issue on their own,
but I decided to include both as GRPC should be readable by non-root
users, and I wanted to preserve the original behavior of running
as root unless the $(NOROOT) flags are specified.
Additionally: clarify comments on the make targets, which are
confusingly named, and stop installing goimports since it seems
it was never used.
Part of this change is implementing a "no secrets" policy for CI. Given that
we have to support CI for arbitrary external contributors, and
it is easy to craft a malicious PR that exfiltrates secrets during a CI build
any test that runs under CI must be able to do so without any injected secrets.
This means that several of the test we currently run under Drone will not be run on GCB, at least as part of the regular CI. The plan is to create a separate task that periodically runs tests that require external credentials (e.g. Kube tests, various backend data stores, etc.) in a more secure way and report failures asynchronously. And while these tests will not run under CI, the should still be built under CI so that required changes are caught during review.
* update deps in manifest and lockfile
* fixes and updates to docker and profiles
* lint rust
* fix typo
* resolve clippy lints
* fix typo
* mark risk functions unsafe
* fmt + clean up the last lints#
* verify lockfile up to date
* disable lto since it doesn't work with two rust libs
* merge lock check and lint
* Add missing license header to Rust files
And update Makefile to ensure they are checked
Co-authored-by: Zac Bergquist <zac.bergquist@goteleport.com>
Download Rust and Go per-build to ensure that the right version is used
and that builds do not step on each other.
Also rungs cbindgen in quiet mode to suppress the annoying output it
spews for non-public symbols.
The race condition detector is being tripped by a concurrent `Write` and
`Close` in the `PipeNetCon` in several integration tests. This is a naive
fix to serialize the write and close operations to resolve the race
condition.
The affected tests were also not handling asynchronous error reporting
correctly (i.e. it's not legal to call `require.XYZ()` from a goroutine
other than the one executing the test function.). This patch introduces
some plumbing to marshal asynchronous errors back into the main test
routine before failing the test.
- Ensure Rust is installed in the buildbox image
- Install Rust toolchains for each arch we support
- Use openssl's vendor feature to ensure we always link a static lib
- Automatically include RDP client if Rust is detected
In some cases, it's possible for a package to be marked as a test
failure even if no tests inside it have failed. The motivating example
for this change is a timeout: a test overshooting the allotted timeout
is considered by go test to be a package-level failure, even if no
tests inside the package are considered failures.
This led to cases where the user would see an "All tests passed"
message from the go test filter, but still mysteriously fail the make
step.
To address this, the test renderer now:
* treats package-level pass/fail/skip events as first-class citizens
and includes them in its event count,
* tracks the cached test output at both a package and individual test
level, and
* displays the whole package output if a package is marked as failed,
but only if there is no obvious failed test top account for the
package-level failure.
This patch also removes the json files created by the unit tests, as
they are not yet needed for anything.
- Don't assume an explicit $GOPATH is set
- Remove golint from linters - it's been deprecated for over a year
and golangci-lint prints a warning instead of running it.
This change makes the Teleport unit tests write a JSON log of all the tests that we run `make test-go`. It also includes a parsing script that will render the JSON log into a human-readable format during the test run, so that the working developer can see what the tests are doing without having to wade through JSON.
The log output is somewhat of a cross between the standard go test output, with pytest-style summaries at the end.
I have limited the realtime report to package-level results (a package-level skip result means no tests file were found). It's trivial to output each test as it comes in, if that works better (but at ~1500 tests, it's a lot).
This is another step towards getting better visibility on our test suite. The idea is that we will eventually collect these test reports as build artifacts for further analysis.
* PIV authentication for RDP
This uncomfortably large change fully implements smartcard PIV
authentication for RDP clients using the Teleport CA:
- PIV applet implementation in emulated RDP smartcard
- generating Windows-compatible certificates using Teleport CA with a
dedicated RPC
- generating dummy CRLs for Teleport CA and publishing it via LDAP
The CRLs are required by Windows for any smartcard login certificate, we
can't avoid that. But we can avoid making it public: the CRL can live in
ActiveDirectory instead of a public endpoint of a Teleport service.
Here, we use LDAP to publish the CRL on startup, valid for a year.
There are a few unhandled cases in the current implementation:
- LDAP server certificate is not validated when upgrading to TLS
- multiple active CAs (with HSMs) are not supported, only one CRL is
published
- CA rotation is not supported, CRL is not re-published on rotation
All of the above issues will be handled in future PRs as this one is
already too large.
* Address review feedback
* Fix linter errors
* Sign tsh.exe on tag builds
This adds a Makefile step to sign tsh.exe when the
`$WINDOWS_SIGNING_CERTIFICATE` env var is set to a base64-encoded
pkcs12 code signing certificate. The certificate must not be password
protected.
This includes a sample cert (`cert-dummy.pfx`) for CI pipeline
testing. It should be removed in any eventual PR, along with the
other modifications to the drone pipeline. The cert is imported into
the environment in the `Makefile` for testing purposes; in practice
it will be imported from a secure secret store (drone secrets, etc).
* Improve Windows code signing
- Split signing into a separate step; `release-windows-unsigned` now
performs the build, `release-windows` signs the binary.
- Require `release-windows` to successfully generate a signed
binary.
- Clearly mark unsigned binaries and archives as such.
- Guard against stdout secret leakage in Makefiles.
- Move temporary cert data from Makefile into dronegen to test
full pipeline.
* Use an invalid cert string for testing purposes.
* Pass certs to the build process via a statically named file
Signed Windows builds now depend on a `.gitignore`'d
`windows-signing-cert.pfx` at the root of the source directory. This
should ease testing and help avoid accidental secret leakage.
* Use production secret
* Remove windows-signing-cert.pfx before continuing to the next step
Additionally, fix variable reference as the bracket syntax does not
seem to play nice with Drone.
* Update .gitignore
Co-authored-by: Andrew Lytvynov <andrew@goteleport.com>
Co-authored-by: Andrew Lytvynov <andrew@goteleport.com>
Introduce new make targets to check and add license headers to files
("make lint-license" and "make fix-license"). License checking is now a part of
"make lint" as well.
Initial attempts used goheader, but it caused "make lint-go" to become about 9x
slower (if not more), plus it only targets go files. Google's addlicense is fast
enough and targets however many file types we want.
Existing files that were missing licenses got the header added, using the
current year as the license date.
* Introduce lint-license and fix-license make targets
* Ignore generated files
* Add license to go files
* Replace irregular licenses with standard copyright/license
* Add license to proto files
* Install addlicense in build.assets Dockerfile
Created an access tester for troubleshooting access related issues with Teleport RBAC system. This access tester allows admins to answer questions like:
Can user alice SSH into a node node-1 as root?
If not, which role(-s) prevents access?
Which roles allow access to production as login admin?
We were using switches specific to GNU coreutils for the api symlink.
These aren't supported on macOS with its BSD variants, so use a more
primitive syntax that will work on both platforms.
Reduced Teleport shared library dependencies on libbpf, libelf, libz.
For libbpf, switched to forked version that does not rely on "fmemopen"
which brings in a glibc 2.22 dependency. This allows binaries built on
Ubuntu 18.04 box to run on CentOS 7 as well.
For libelf and libz (which libbpf uses), the build process has been
updated to statically link both of them during the build process.
Adds the ability to block network traffic on SSH sessions.
The deny/allow lists of IPs are specified in teleport.yaml file.
Supports both IPv4 and IPv6 communication.
This feature currently relies on enhanced recording for
cgroup management so that needs to be enabled as well.
-- Design rationale:
This patch uses Linux Security Module (LSM) hooks, specifically
security_socket_connect and security_socket_sendmsg, to control
egress traffic. The LSM provides two advantages over socket filtering
program types.
- It's executed early enough that the task information is available.
This makes it easy to report PID, COMM, etc.
- It becomes a model for extending restrictions beyond networking.
The set of enforced cgroups is stored in a BPF hash map and the
deny/allow lists are stored in BPF trie maps. An IP address is
first checked against the allow list. If found, it's checked for
an override in the deny list. The policy is default deny. However,
the absence of the NetworkRestrictions API object is allow all.
IPv4 addresses are additionally registered in IPv6 trie (as mapped)
to account for dual stacks. However it is unclear if this is sufficient
as 4-to-6 transition methods utilize a multitude of translation and
tunneling methods.
When `SHELL` is not set, `make` defaults to `/bin/sh`.
On systems where `/bin/sh` is an alias for `/bin/bash`, everything works
as expected.
On systems where `/bin/sh` is actually the original Bourne Shell, some
bash-isms don't work. For example: `if [[ condition ]]` results in
`/bin/sh: 1: [[: not found`
* Revert "darwin fips builds (#5866)"
This reverts commit 32ac67db06.
* Remove GO_BINARY references
* Re-add dronegen changes for commands/image
* make dronegen
* Update e ref
* Re-add package signing/notarization for full MacOS builds
* Switch to go1.16. Use embed package to embed webassets instead of ad-hoc attaching to binary
* Fix pipeline duplicate step error
* Resolve duplicate pipeline step name error. Explicitly define platform for 'exec' pipelines. Remove the uid/gid environment from 'exec' pipelines as redundant.
* Set proper dependencies when building darwin package fips pipelines. Use enterprise build directory for tsh
* Address review comments
An extra dockerfile for gRPC generation is extra maintenance burden. It
was also using a really old base image that has a ton of known vulns.
Also, update GOGO_PROTO_TAG to match the version we have vendored via
go.mod.
* Add 'make update-webassets' script
Copying over from `teleport/ops` to make it easier to discover.
Also changed the script to clone the repos into a temp directory and
clean it up later. Without this, a nested `teleport` checkout would
cause all kinds of problems.
* Update Makefile
Co-authored-by: Gus Luxton <gus@gravitational.com>
Co-authored-by: Gus Luxton <gus@gravitational.com>
Each user can now have multiple devices. This commit only changes the
backend structure to support it, the client and API haven't been updated
yet.
Also added a migration for existing MFA data on auth server startup.
* Update logrus package to fix data races
* Introduce a logger that uses the test context to log the messages so they are output if a test fails for improved trouble-shooting.
* Revert introduction of test logger - simply leave logger configuration at debug level outputting to stderr during tests.
* Run integration test for e as well
* Use make with a cap and append to only copy the relevant roles.
* Address review comments
* Update integration test suite to use test-local logger that would only output logs iff a specific test has failed - no logs from other test cases will be output.
* Revert changes to InitLoggerForTests API
* Create a new logger instance when applying defaults or merging with file service configuration
* Introduce a local logger interface to be able to test file configuration merge.
* Fix kube integration tests w.r.t log
* Move goroutine profile dump into a separate func to handle parameters consistently for all invocations
This helps with ELB and similar L5 load balancers that don't respect
TCP-level keep-alives. ELB for example kills connections after 60s of no
application traffic.
With this PR, you can leave a `kubectl exec` session open indefinitely
without any activity.
Shellcheck is a linter for shell scripts. Since we have quite a few of
those for release packaging and examples, we'll benefit from an extra
set of (robot) eyes.
Note: I disabled https://github.com/koalaman/shellcheck/wiki/Sc2086 to
make this PR smaller. That specific check is for the most frequent
mistake in our scripts - not quoting env var expansions. I'll do a
separate PR cleaning those up.
`build.assets/pkg` is no longer used and was removed.
This commit introduces GRPC API for streaming sessions.
It adds structured events and sync streaming
that avoids storing events on disk.
You can find design in rfd/0002-streaming.md RFD.
* SEO changes in Documentation
* All documentation pages have dedicated <title> tag
* All documentation pages have dedicated <meta description> tag
* Fixed a few broken links
* Fixed missing <H1> tags
* Renamed some pages to make SEO-friendly URLs
* Found and updated all links to the renamed pages
* Compress PNGs
Co-authored-by: Ben Arent <ben@gravitational.com>
Store the signing algorithm along the CA private key. When reading old
CAs that don't have it set, default to UNKNOWN proto enum which
corresponds to the old SHA1-based signing alg.
The only time you get a SHA2 signature is when creating a fresh cluster
and generating a new CA. This can be disabled in the config.
