* Metrics: expose install method counter
This PR adds a new metric that exposes the number of servers currently
running grouped by their install method.
Note: install method is a list o strings, so the metric sorts its values
and then joins them by "," to create a single identifier.
* do not mutate original install methods list
* Expose Ping() in bare auth server
* Handle both pointer and bare PluginStatusV1
* Add metric name
* Add StatusSink
* Run GCI
* Move comment back to auth_with_roles
* Update lib/auth/auth.go
Co-authored-by: Alan Parra <alan.parra@goteleport.com>
* Rework SetStatus
* Inline TryEmitStatus and use a proper context
* Fix copyright notice
* Fix bug in statusFromStatusCode
* Test statusFromResponse
* Add link to Slack API schema
* Refactor statusFromStatusCode
* Expand comment for Ping()
* Add basic check for status in slack test
* Address nits
---------
Co-authored-by: Alan Parra <alan.parra@goteleport.com>
This commit adds a new Prometheus gauge `teleport_migrations` that
tracks for each migration if it is active (1) or not (0).
This gauge is then leveraged in `tctl top` to show a set of active
migrations.
* [draft] Add a new usage reporter
This adds a new usage reporter service to the auth server. It's
disabled by default in OSS and can only be turned on via startup hook
in Cloud / Enterprise. In OSS, the audit log wrapper is never
configured and any usage events are sent to a no-op discard reporter.
Usage events are defined in prehog and can be sent to the new
UsageReporter Service on the auth server. An audit event wrapper is
used to capture certain events that are otherwise difficult to hook.
Events are anonymized before submission, then held in a non-blocking
queue for batching and submission purposes.
* Remove dead code
* Add SubmitUsageEvent RPC to Auth.
This adds a new SubmitUsageEvent RPC to the Auth API that external
clients (e.g. the UI) can use to submit usage events externally.
* Slight refactor for unit testing
* Add Prometheus metrics and add initial working prehog submitter
* Add more metrics, tweak prehog client, and add unit tests
* Further tweak http transport settings based on Teleport defaults
* Add missing metrics
* Fix goimports
* Add new UI usage events
* Update e ref
* Add prehog directly for now. Improve logging.
* update prehog
* Add new prehog events; use username from request identity
* add HTTP server for user events
* Add username back to pre-onboard events
* unauthenticated user events
* Fix userevent build error
* Use event-provided username where appropriate
* Move barebones prehog reqs to lib/prehog and generate here.
Also, use prod tunable values.
* Fix license lints
* De-flake tests by adding unfortunate amounts of synchronization.
* Add missing license header
* Misc PR cleanup for review
* Update lib/events/usageevents/usageevents.go
Co-authored-by: Edoardo Spadolini <edoardo.spadolini@goteleport.com>
* Address a batch of review comments
Adds `anonymizer.AnonymizeString` and parent loggers
* Update e ref
* Clean up comments
* Remove onboard prefix from recovery code event
* Address another batch of feedback
* Use defaults.HTTPClient()
* Remove a noisy log message
* Demote noisy log message to debug
* Temporarily revert e ref for merge
Co-authored-by: Michelle Bergquist <michelle.bergquist@goteleport.com>
Co-authored-by: Edoardo Spadolini <edoardo.spadolini@goteleport.com>
This adds the Prometheus metric teleport_connected_resources. Gauge increments when the keepalive is established and will decrement whenever the connection is broken/closed.
Adds teleport_reverse_tunnels_connected Prometheus metric which tracks reverse tunnels connected to the proxy server by type.
* Update prometheus help
Co-authored-by: Paul Gottschling <paul.gottschling@goteleport.com>
* Update metrics wording
Co-authored-by: Paul Gottschling <paul.gottschling@goteleport.com>
Adds the Prometheus metric teleport_registered_servers which is a gauge indicating the unique number of Teleport instances connected to the cluster by version.
Co-authored-by: Zac Bergquist <zmb3@users.noreply.github.com>
Co-authored-by: Paul Gottschling <paul.gottschling@goteleport.com>
Adds the ability to block network traffic on SSH sessions.
The deny/allow lists of IPs are specified in teleport.yaml file.
Supports both IPv4 and IPv6 communication.
This feature currently relies on enhanced recording for
cgroup management so that needs to be enabled as well.
-- Design rationale:
This patch uses Linux Security Module (LSM) hooks, specifically
security_socket_connect and security_socket_sendmsg, to control
egress traffic. The LSM provides two advantages over socket filtering
program types.
- It's executed early enough that the task information is available.
This makes it easy to report PID, COMM, etc.
- It becomes a model for extending restrictions beyond networking.
The set of enforced cgroups is stored in a BPF hash map and the
deny/allow lists are stored in BPF trie maps. An IP address is
first checked against the allow list. If found, it's checked for
an override in the deny list. The policy is default deny. However,
the absence of the NetworkRestrictions API object is allow all.
IPv4 addresses are additionally registered in IPv6 trie (as mapped)
to account for dual stacks. However it is unclear if this is sufficient
as 4-to-6 transition methods utilize a multitude of translation and
tunneling methods.
If we leave `TextStyle` empty on UI elements, it will use the default
foreground color defined by the terminal (light for dark terminals and
vice versa). Same goes for `BorderStyle`.
A few other tweaks to UI and source metrics:
- update table ratios to prevent hiding output rows on short (height)
terminal windows
- update tab selector style to use bold/underline instead of colors to
mark selected tab
- print `No data` in histogram tables when there are no values
- don't report the local cluster in `remote_clusters` metric
Added package cgroup to orchestrate cgroups. Only support for cgroup2
was added to utilize because cgroup2 cgroups have unique IDs that can be
used correlated with BPF events.
Added bpf package that contains three BPF programs: execsnoop,
opensnoop, and tcpconnect. The bpf package starts and stops these
programs as well correlating their output with Teleport sessions
and emitting them to the audit log.
Added support for Teleport to re-exec itself before launching a shell.
This allows Teleport to start a child process, capture it's PID, place
the PID in a cgroup, and then continue to process. Once the process is
continued it can be tracked by it's cgroup ID.
Reduced the total number of connections to a host so Teleport does not
quickly exhaust all file descriptors. Exhausting all file descriptors
happens very quickly when disk events are emitted to the audit log which
are emitted at a very high rate.
Added tarballs for exec sessions. Updated session.start and session.end
events with additional metadata. Updated the format of session tarballs
to include enhanced events.
Added file configuration for enhanced session recording. Added code to
startup enhanced session recording and pass package to SSH nodes.
Buffer fan out used simple prefix match
in a loop, what resulted in high CPU load
on many connected watchers.
This commit switches to RADIX trees for
prefix matching what reduces CPU load
substantially for 5K+ connected watchers.
This commit expands the usage of the caching layer
for auth server API:
* Introduces in-memory cache that is used to serve all
Auth server API requests. This is done to achieve scalability
on 10K+ node clusters, where each node fetches certificate authorities,
roles, users and join tokens. It is not possible to scale
DynamoDB backend or other backends on 10K reads per seconds
on a single shard or partition. The solution is to introduce
an in-memory cache of the backend state that is always used
for reads.
* In-memory cache has been expanded to support all resources
required by the auth server.
* Experimental `tctl top` command has been introduced to display
common single node metrics.
Replace SQLite Memory Backend with BTree
SQLite in memory backend was suffering from
high tail latencies under load (up to 8 seconds
in 99.9%-ile on load configurations).
This commit replaces the SQLite memory caching
backend with in-memory BTree backend that
brought down tail latencies to 2 seconds (99.9%-ile)
and brought overall performance improvement.
