The use of non-UTF8 keys with the DynamoDB back-end causes a failure
deep within the AWS request deserialization code, presenting a
non-obvious failure to the user.
This change adds validation to all backends that requires all keys
are valid UTF8 strings. It also adds a warning to the Backend
interface declaration that the keys may be constrained to valid
UTF8.
Other changes include:
* Updating the `Backend` conformance test suite to not present binary
keys to the various backend implementations.
* Adding a `region` value to the DynamoDB configuration test input
* Adding missing imports to `_test` files.
* Updating build instructions in README
Updated storage configuration to not only apply to DynamoDB in the
backend package, but also DynamoDB in the events package. This allows
configuring continuous backups and auto scaling for the events table.
This commit fixes#4598
Config with multiple event backends was crashing on 4.4:
```yaml
storage:
audit_events_uri: ['dynamodb://streaming', 'stdout://', 'dynamodb://streaming2']
```
* Fix local etcd test failures when etcd is not running
* Add kubernetes_service to teleport.yaml
This plumbs config fields only, they have no effect yet.
Also, remove `cluster_name` from `proxy_config.kubernetes`. This field
will only exist under `kubernetes_service` per
https://github.com/gravitational/teleport/pull/4455
* Handle IPv6 in kubernetes_service and rename label fields
* Disable k8s cluster name defaulting in user TLS certs
Need to implement service registration first.
`require` is a sister package to `assert` that terminates the test on
failure. `assert` records the failure but lets the test proceed, which
is un-intuitive.
Also update all existing tests to match.
This comit fixes#4508
Gogoproto is not compatible with APIv2 protoc-gen-go.
Track the issue here: https://github.com/gogo/protobuf/issues/678
Meanwhile, this commit switches to google protobuf to unmarshal firebase struct.
Add a missing method EmitAuditEvent causing teleport to crash
with firestore events backend.
* Always collect metrics about top backend requests
Previously, it was only done in debug mode. This makes some tabs in
`tctl top` empty, when auth server is not in debug mode.
* backend: use an LRU cache for top requests in Reporter
This LRU cache tracks the most frequent recent backend keys. All keys in
this cache map to existing labels in the requests metric. Any evicted
keys are also deleted from the metric.
This will keep an upper limit on our memory usage while still always
reporting the most active keys.
* DynamoDB: Build http transport from defaults before manipulating parameters, this allows the transport to be pre-populated with proxy information if set by HTTPS_PROXY/NO_PROXY environment variables.
When custom prefix is `/foo/`, the migration would move `/teleport/a` to
`/foo/a`, but the backend later tries to read `/foo//a`.
Also added tests to cover these edge cases.
The prefix fetching logic has a bug: it treats everything starting with
`/teleport` as the legacy prefix data, even if it's `/teleport-foo/bar`.
This is an issue if user specifies `/teleport-foo` as their custom
prefix. Each restart will copy the data from `/teleport-foo/...` to
`/teleport-foo-foo/...`.
Set the legacy prefix const to `/teleport/` instead. This avoids
excessive copying during startup.
Prefixes can still be confused later on, with `Watch` and `GetRange`,
but this is harder to migrate with backwards-compatibility.
Go 1.15 catches these:
lib/backend/test/suite.go:402:12: conversion from OpType (int) to string yields a string of one rune, not a string of digits (did you mean fmt.Sprint(x)?)
Since these are ints, they can be compared for equality without
stringifying.
Same as values, `backend.Item` passes keys as `[]byte` and has no
guarantees about the encoding. GetRange queries need extra care because
of this type mismatch too: `where` clauses are type-sensitive.
Firestore marshaler requires all string data to be valid UTF-8. Our
values are sometimes binary (like QR codes for OTP signup), which would
fail to marhal.
On reads, try using both new and old formats to support existing data.
Added tests for this fallback and for writing binary data into all
backends.
https://github.com/gravitational/teleport/pull/3766 removed a nil
pointer check when creating firestore indexes. This was a legitimate
check, for when indexes already exist.
Tested this manually. Unit testing is trickier because the firestore
emulator in gcloud doesn't support indexes.
This code is only intended for 4.3 and will be removed in 4.4.
The goal is to automatically move data to the correct etcd prefix that
the customer has specified in their config. Prior to 4.3, their data was
always written to `/teleport` prefix, ignoring the config.
The copy happens iff:
- /teleport is not empty
- config prefix is empty OR older than /teleport
This double-writing will only exist in 4.3 and the config prefix is
still source-of-truth for reads.
Double-writes are necessary to allow users to downgrade to 4.2 and back
up again without losing data.
Prefix-handling code was using a hardcoded prefix (`/teleport`) instead
of the prefix specified in config. Use the correct config prefix and add
a test.
List of fixed items:
```
integration/helpers.go:1279:2 gosimple S1000: should use for range instead of for { select {} }
integration/integration_test.go:144:5 gosimple S1009: should omit nil check; len() for nil slices is defined as zero
integration/integration_test.go:173:5 gosimple S1009: should omit nil check; len() for nil slices is defined as zero
integration/integration_test.go:296:28 gosimple S1019: should use make(chan error) instead
integration/integration_test.go:570:41 gosimple S1019: should use make(chan interface{}) instead
integration/integration_test.go:685:40 gosimple S1019: should use make(chan interface{}) instead
integration/integration_test.go:759:33 gosimple S1019: should use make(chan string) instead
lib/auth/init_test.go:62:2 gosimple S1021: should merge variable declaration with assignment on next line
lib/auth/tls_test.go:1658:22 gosimple S1024: should use time.Until instead of t.Sub(time.Now())
lib/backend/dynamo/dynamodbbk.go:420:5 gosimple S1004: should use !bytes.Equal(expected.Key, replaceWith.Key) instead
lib/backend/dynamo/dynamodbbk.go:656:12 gosimple S1039: unnecessary use of fmt.Sprintf
lib/backend/etcdbk/etcd.go:458:5 gosimple S1004: should use !bytes.Equal(expected.Key, replaceWith.Key) instead
lib/backend/firestore/firestorebk.go:407:5 gosimple S1004: should use !bytes.Equal(expected.Key, replaceWith.Key) instead
lib/backend/lite/lite.go:317:5 gosimple S1004: should use !bytes.Equal(expected.Key, replaceWith.Key) instead
lib/backend/lite/lite.go:336:6 gosimple S1004: should use !bytes.Equal(value, expected.Value) instead
lib/backend/memory/memory.go:365:5 gosimple S1004: should use !bytes.Equal(expected.Key, replaceWith.Key) instead
lib/backend/memory/memory.go:376:5 gosimple S1004: should use !bytes.Equal(existingItem.Value, expected.Value) instead
lib/backend/test/suite.go:327:10 gosimple S1024: should use time.Until instead of t.Sub(time.Now())
lib/client/api.go:1410:9 gosimple S1003: should use strings.ContainsRune(name, ':') instead
lib/client/api.go:2355:32 gosimple S1019: should use make([]ForwardedPort, len(spec)) instead
lib/client/keyagent_test.go:85:2 gosimple S1021: should merge variable declaration with assignment on next line
lib/client/player.go:54:33 gosimple S1019: should use make(chan int) instead
lib/config/configuration.go:1024:52 gosimple S1019: should use make(services.CommandLabels) instead
lib/config/configuration.go:1025:44 gosimple S1019: should use make(map[string]string) instead
lib/config/configuration.go:930:21 gosimple S1003: should use strings.Contains(clf.Roles, defaults.RoleNode) instead
lib/config/configuration.go:931:22 gosimple S1003: should use strings.Contains(clf.Roles, defaults.RoleAuthService) instead
lib/config/configuration.go:932:23 gosimple S1003: should use strings.Contains(clf.Roles, defaults.RoleProxy) instead
lib/service/supervisor.go:387:2 gosimple S1001: should use copy() instead of a loop
lib/tlsca/parsegen.go:140:9 gosimple S1034: assigning the result of this type assertion to a variable (switch generalKey := generalKey.(type)) could eliminate type assertions in switch cases
lib/utils/certs.go:140:9 gosimple S1034: assigning the result of this type assertion to a variable (switch generalKey := generalKey.(type)) could eliminate type assertions in switch cases
lib/utils/certs.go:167:40 gosimple S1010: should omit second index in slice, s[a:len(s)] is identical to s[a:]
lib/utils/certs.go:204:5 gosimple S1004: should use !bytes.Equal(certificateChain[0].SubjectKeyId, certificateChain[0].AuthorityKeyId) instead
lib/utils/parse/parse.go:116:45 gosimple S1003: should use strings.Contains(variable, "}}") instead
lib/utils/parse/parse.go:116:6 gosimple S1003: should use strings.Contains(variable, "{{") instead
lib/utils/socks/socks.go:192:10 gosimple S1025: should use String() instead of fmt.Sprintf
lib/utils/socks/socks.go:199:10 gosimple S1025: should use String() instead of fmt.Sprintf
lib/web/apiserver.go:1054:18 gosimple S1024: should use time.Until instead of t.Sub(time.Now())
lib/web/apiserver.go:1954:9 gosimple S1039: unnecessary use of fmt.Sprintf
tool/tsh/tsh.go:1193:14 gosimple S1024: should use time.Until instead of t.Sub(time.Now())
```
