mockTSHDEventsClient will most likely be called outside of the test
goroutine, so we shouldn't use require.NoError there.
helpers.SetupUserCreds returns an error which I did not inspect. The linter
didn't warn me about it for some reason.
Added the ability to specify two CLI flags for Access Requests:
"--request-ttl" and "--session-ttl".
Updated "CreateAccessRequest" to adhere to the following rules.
If an Access Request does not have a TTL set (expiration time
types.AccessRequest resource itself), a default of 1 hour is used. Next,
the request value is truncated by the lifetime of the certificate,
requested expiration, and then strictest session TTL on all roles
requested.
Similar logic is followed for the expiration time of the elevated
certificate that will be issued if the Access Request is approved. First
the requested value is truncated by the lifetime of the certificate,
requested expiration, and then strictest session TTL on all roles
requested.
The output of "tsh requests ls" and "tctl requests ls" has been updated
to display the TTL values.
* Connection Diagnostics: Postgres Database tester
When adding a new resource using the Web UI we want to allow users to
test connecting to it.
We have two connection testers already:
- for SSH Nodes
- for Kube clusters
This PR adds a third tester: Postgres Database.
Most of the required changes for any Database are already present but
we're focusing on Postgres for now. Other databases will be added as
future PRs.
Testing a Database is similar to the other tests:
- generate certs for the logged in user
- connect to the resource using those certs
When generating the certificate, we inject an ID so that the Database
Service can add Connection Diagnostic traces.
The TestTCPCertExpiration test may have a race condition where the connection
made to the test application immediately after the cert expiry may succeed.
Due to the wonky nature of using the fake clock in this test, I'm introducing
a requires.Eventually here to hopefully mitigate this race.
While working on integration tests, I noticed that testOptions.leafConfig
is simply ignored because after we set up the leaf cluster config, we call
testOptions.rootConfig instead.
* Add TTL field to integration/helpers.UserCredsRequest
This will let us create expired user certs by providing a negative TTL.
* Reissue gateway cert if middleware detects it expired
* Add integration test for gateway cert renewal
Many of our tests (db package, I'm looking at you) generate many RSA keys. This has two main side effects; makes our tests slow and flaky as CPU usage spikes in random moments when the tests are run in parallel.
This change pre-generates RSA keys at the beginning of each test module and reuse them in randomized order to reduce the situation that one key has been used multiple times in one test.
I had to move a few files to avoid circular dependencies.
With the arrival of Teleport 12, we stop supporting v10.x.x clients, and we no longer require the `types.KindKubernetesService` legacy heartbeat.
This change removes the legacy heartbeat from Kubernetes Service but keeps the legacy Auth Server CRUD methods and heartbeat support to maintain compatibility with Teleport 11 Kubernetes Service - Teleport 11 still heartbeats the legacy type.
We postponed the related `DELETES` to Teleport 13.
There is an occasional race condition where the app session does not appear to
be reflected in the backend after it's created by the time the test logic in
the app access integration tests is run. This will (hopefully) address this
issue.
Session control logic existed within `HandleNewConn` of `srv/regular.Server`.
This prevented any of it from being used by other components that
also needed to enforce session control.
All the logic from within `HandleNewConn` was refactored to a new
`srv.SessionController` object which the `regular.Server` now uses
to perform session control. There were a few additional changes
needed to accomadate that session control now exists outside
the server and to make tests easier to write. Namely, altering
`srv.ComputLockTargets` to not take a `Server` as a parameter and
leveraging a clock within `services.SemaphoreLock`.
This is step 2 in addressing #15167. Before the web apiserver can
leverage the newly introduced proxy.Router and bypass making ssh
connections to the proxy ssh server it needs to be able to perfrom
session control.
* Add mutex for certs in local proxy
* Update lib/srv/alpnproxy/local_proxy.go
Co-authored-by: Ryan Clark <ryan.clark@goteleport.com>
* Update lib/srv/alpnproxy/local_proxy.go
Co-authored-by: Ryan Clark <ryan.clark@goteleport.com>
* Update lib/srv/alpnproxy/local_proxy.go
Co-authored-by: Ryan Clark <ryan.clark@goteleport.com>
* Move cert checking out of middleware into local proxy
* Configure a logger for local proxy
* Fixup imports
* Add tests for local proxy
* test for data race
* test for cert checking
* Update integration test for local proxy
* Mark err assert fns as helpers
Co-authored-by: Ryan Clark <ryan.clark@goteleport.com>
* Allow connections to nodes when Auth is offline
`tsh ssh user@foo` currently fails if Auth is unreachable. Prior to
connecting to the target, `tsh` sends a `proto.AuthService/IsMFARequired`
request to determine if per-session MFA is required. If necessary, the MFA
ceremony then occurs before `tsh` attempts to connect to the node.
However, nodes ultimately makes the decision on whether the user has
access. To both increase availability of nodes and reduce initial
connection latency, `tsh` can first attempt to connect to a node and
fallback to checking if per-session MFA is required and then performing
the ceremony if necessary followed by attempting another connection to the
node. The major tradeoff is now when per-session MFA is required, `tsh`
will connect to the target **twice**. The added latency is likely to be
minimized by the amount of time it takes to perform the MFA ceremony.
## Connection flows prior to this change:
### Auth online and per-session MFA required
```mermaid
sequenceDiagram
participant tsh
participant Auth
participant Node
tsh->>+Auth: IsMFARequired
Auth-->>-tsh: yes
tsh->>+Auth: perform mfa ceremony
Auth-->>+tsh: issue challenge
tsh-->>+Auth: challenge response
Auth-->>+tsh: issue certificates
tsh->>+Node: Connect
Node-->>+tsh: Success
```
### Auth online and per-session MFA not required
```mermaid
sequenceDiagram
participant tsh
participant Auth
participant Node
tsh->>+Auth: IsMFARequired
Auth-->>-tsh: no
tsh->>+Node: Connect
Node-->>+tsh: Success
```
### Auth offline and per-session MFA not required
```mermaid
sequenceDiagram
participant tsh
participant Auth
participant Node
tsh->>+Auth: IsMFARequired
Auth-->>-tsh: Error
```
### Auth offline and per-session MFA required
```mermaid
sequenceDiagram
participant tsh
participant Auth
participant Node
tsh->>+Auth: IsMFARequired
Auth-->>-tsh: Error
```
## Connection flows after this change:
### Auth online and per-session MFA required
```mermaid
sequenceDiagram
participant tsh
participant Auth
participant Node
tsh->>+Node: Connect
Node-->>+tsh: Access Denied
tsh->>+Auth: IsMFARequired
Auth-->>-tsh: yes
tsh->>+Auth: perform mfa ceremony
Auth-->>+tsh: issue challenge
tsh-->>+Auth: challenge response
Auth-->>+tsh: issue certificates
tsh->>+Node: Connect
Node-->>+tsh: Success
```
### Auth online and per-session MFA not required
```mermaid
sequenceDiagram
participant tsh
participant Auth
participant Node
tsh->>+Node: Connect
Node-->>+tsh: Success
```
### Auth offline and per-session MFA not required
```mermaid
sequenceDiagram
participant tsh
participant Auth
participant Node
tsh->>+Node: Connect
Node-->>+tsh: Success
```
### Auth offline and per-session MFA required
```mermaid
sequenceDiagram
participant tsh
participant Auth
participant Node
tsh->>+Node: Connect
Node-->>+tsh: Access Denied
tsh->>+Auth: IsMFARequired
Auth-->>-tsh: Error
```
We were inconsistent throughout the codebase and would sometimes
use the slices package and other times use our own equivalents
in api/.
This removes our versions in favor of the golang.org/x package that
does the same, which has the added benefit of reducing the surface
area of the public API module.
Note: despite existing uses of the slices package, for some reason
it didn't show up in go.mod or go.sum. Fixed that too.
This change re-enables the AuditOn system test and fixes the TTY connection between the Teleport parent and child process. It should allow the child to send the error code to the parent, which should fix the test.
The CertificateTTL causes problems after the introduction of the fake clock
into the appaccess tests due to the fake clock not being propagated properly
throughout the Teleport stack. Increasing the TTL should make this test much
less flaky.
* plumb context to keystore methods
Contexts are more relevant now with the GCP KMS keystore which makes
http requests, this commit makes sure they are passed everywhere they
are needed, with only a few `context.TODO`s sprinkled in.
* feat: add GCP KMS support for Teleport CA key material
This commit implements support for GCP KMS as a backend for CA
operations in Teleport.
This is able to take advantage of much of the infrastucture that we have
already created for HSM support, and simply appears as a new backend for
the private key material.
The necessary configuration parameters include only the name of the KMS
keyring to use, and the protection level (which can be HSM or SOFTWARE).
These are configured in the teleport.yaml directly, in a new section
under the existing `ca_key_params` used for HSM configuration.
The GCP credentials are expected to be provided to the Teleport auth
server via the
[Application Default Credentials](https://cloud.google.com/docs/authentication/provide-credentials-adc).
This means that it "just works" if the auth server is running on a GCP
compute instance with the correct attached role, and you can run tests
locally by authenticating with `gcloud auth login`.
This does not support Teleport Cloud, as our current HSM support does
not, because the Auth server needs the configuration and the access to
the GCP account. That would be a larger effort probably requiring a new
Teleport service.
* shrink/remove keystore interface
This commit introduces the keystore.Manager type to handle all
interaction between CA and the keystore backends.
Why:
* reduces the code that needs to be implemented per keystore backend to
only the necessary operations
* separate concerns of managing key material and handling CA data
structures
* define interfaces where they're used, not implemented
* delete net 245 lines of code
* reduce keystore.KeyStore stutter
The appaccess TestTCP* tests are highly reliant on time. This has been
reduced (but not eliminated) by using a fakeClock and a channel for
signaling monitor triggered connection closures.
* Fix flaky basic auth dialer test
* Block on sending error to waiters in ProxyAuthorizer, to avoid racing
against the waiter.
* Change the test to start the node in a separate goroutine and
manipulate the auth creds while it is still attempting to connect.
This will allow us to significantly speed up the test since we can
verify that the proxy authorizer is rejecting bad credentials but then
change the credentials to be valid afterwards, allowing the node to
succeed in registering and avoid long wait for it to fail to register.
* Use buffered chan
* Move the stopall defer up in case the test fails earlier
* Remove extra zero count check
Application access connection monitoring has been introduced so that, when a
lock is created, application access connections will be interrupted until the
lock has been cleared. This includes web sockets and TCP applications.
* Ensure invalid tunnel agent connections get closed
Connections from reverse tunnel agents were being marked
as invalid by the proxy under certain conditions but would
ultimately never be closed. This could lead to scenarios where
the agent thought things were fine but the proxy considered
that agent unhealthy and unroutable.
Pruning of invalid connections used to occur when a proxy
tried to retrieve a connection for that tunnel. This also
further muddied the point in time at which the proxy could
close a connection as it never explicitly stopped tracking
the connection and closed it at the same time.
To remedy this, connections are explicitly closed by the proxy
and removed from the mapping to stop tracking immediately. In order
to prevent a connection that is servicing an active connection
from being closed the proxy now tracks which connections have
sessions. Closing does not occur when there are any active
sessions to prevent them from being force terminated.
When the proxy receives a heartbeat from an agent it now restores
the connection to a valid state. In the event that too many heart
beats have been missed for an agent, the proxy will now terminate
the connection, again only if it is not serving any sessions.
Fixes#15911
* Update proxy handler and authorizer mocks
* Use a condition variable to properly sync checks for the last error received
* Fix test to check for all 3 nodes registering correctly
* Update naming of LastError to be more descriptive
* Refactor authorizer to not use a condition variable
* Remove 3rd node to speed up test
* Make test more robust
* Remove the reset count func
* Close client conns before new requests to be sure
* Connection -> Request terminology
* Change test to not mutate the environment variable, but instead manipulate the auth proxy credentials
* this way we can be sure that the test will work correctly when the credentials match. If we mutate env, we don't know whether the callers are still holding a dialer using the old env variable
* Remove extra node
* fix lint
* Fix req waiting
* Change wording in debug message
* fix comment
* Update integration/helpers/proxy.go
Co-authored-by: Edoardo Spadolini <edoardo.spadolini@goteleport.com>
* Fix defer func
Co-authored-by: Edoardo Spadolini <edoardo.spadolini@goteleport.com>
* Add X-Forwarded-SSL and X-Forwarded-Port to appaccess.
Application Access now adds in X-Forwarded-Ssl and X-Forwarded-Port headers.
Tests have been added and adjusted to look for these new headers as well.
* Update lib/srv/app/header_rewriter.go
Co-authored-by: Ryan Clark <ryan.clark@goteleport.com>
* Update integration/appaccess/fixtures.go
Co-authored-by: Roman Tkachenko <roman@goteleport.com>
* Remove common.XForwardedPort
* Change order of websocket delegates.
* Make ReservedHeaders more future-proofed.
* Add local proxy middleware for db cert checking
* Use tls conversion util instead of inline
* Add middleware to local proxy config
* Add middleware configuration in tsh
* Use route to database check and set defaults func
* Dont trigger normal db login flow if using local proxy tunnel
* Split out adding client creds into helper func for testing
* Add integration test for local proxy tunnel db cert middleware
* Add unit test for local proxy middleware
* Update comment
* Make middleware on new conn block
* godoc
* Make any cert check error trigger cert renewal in local proxy middleware
* Move dbcertchecker into lib/client
* Remove unneeded mutex in local proxy and unused func in lib/utils
* Make local proxy middleware integration test more robust
* Print message before mfa prompt in proxy tunnel
* Add before prompt option to test
* Remove unneeded comment
* Change local proxy messages to be more clear
* Pass local proxy opts by reference
* Pass certs in opts instead of cert/key file path
This is so we can check if the error is recoverable while preparing
local proxy options. A tunneled local proxy can ignore the error because
it does not rely on cert files - it can just renew its certs if
necessary.
* Move db route checking back to tsh
* Fix lint err
* Fix typo and print the hint to same writer as the mfa prompt
