Uploader retries slower on network errors and picks the pace
after any upload has succeeded.
Records that were corrupted, will never get uploaded.
The uploader will create streams indefinitely, clogging the auth server
with streams. Now uploader writes marker for bad session uploads
and does not attempt to reupload.
`require` is a sister package to `assert` that terminates the test on
failure. `assert` records the failure but lets the test proceed, which
is un-intuitive.
Also update all existing tests to match.
This commit introduces GRPC API for streaming sessions.
It adds structured events and sync streaming
that avoids storing events on disk.
You can find design in rfd/0002-streaming.md RFD.
Fixed findings:
```
lib/events/test/suite.go:88:2: SA4006: this value of `err` is never used (staticcheck)
err := s.Log.EmitAuditEvent(events.UserLocalLogin, events.EventFields{
^
lib/events/auditlog_test.go:98:2: SA4006: this value of `err` is never used (staticcheck)
fileHandler, err := filesessions.NewHandler(filesessions.Config{
^
lib/events/auditlog_test.go:121:2: SA4006: this value of `err` is never used (staticcheck)
err = os.MkdirAll(filepath.Join(uploadDir, "upload", "sessions", defaults.Namespace), 0755)
^
lib/events/auditlog_test.go:224:2: SA4006: this value of `err` is never used (staticcheck)
fileHandler, err := filesessions.NewHandler(filesessions.Config{
^
lib/events/auditlog_test.go:246:2: SA4006: this value of `err` is never used (staticcheck)
err = os.MkdirAll(filepath.Join(uploadDir, "upload", "sessions", defaults.Namespace), 0755)
^
lib/events/auditlog_test.go:381:2: SA4006: this value of `err` is never used (staticcheck)
fileHandler, err := filesessions.NewHandler(filesessions.Config{
^
lib/events/auditlog_test.go:403:2: SA4006: this value of `err` is never used (staticcheck)
fileHandler, err := filesessions.NewHandler(filesessions.Config{
^
lib/events/auditlog_test.go:431:2: SA4006: this value of `err` is never used (staticcheck)
err := os.MkdirAll(filepath.Join(uploadDir, "upload", "sessions", defaults.Namespace), 0755)
^
lib/events/filesessions/fileuploader.go:84:2: SA4006: this value of `err` is never used (staticcheck)
_, err := os.Stat(filepath.Dir(path))
^
lib/events/dynamoevents/dynamoevents.go:352:2: SA4006: this value of `err` is never used (staticcheck)
attributeValues, err := dynamodbattribute.MarshalMap(attributes)
^
lib/events/dynamoevents/dynamoevents.go:405:2: SA4006: this value of `err` is never used (staticcheck)
attributeValues, err := dynamodbattribute.MarshalMap(attributes)
^
```
Spring cleaning!
A very mechanical cleanup using several linters (unused, deadcode,
structcheck). Build and tests still pass so no behavior should be
affected.
Updates #1755
Design
------
This commit adds support for pluggable events and
sessions recordings and adds several plugins.
In case if external sessions recording storage
is used, nodes or proxies depending on configuration
store the session recordings locally and
then upload the recordings in the background.
Non-print session events are always sent to the
remote auth server as usual.
In case if remote events storage is used, auth
servers download recordings from it during playbacks.
DynamoDB event backend
----------------------
Transient DynamoDB backend is added for events
storage. Events are stored with default TTL of 1 year.
External lambda functions should be used
to forward events from DynamoDB.
Parameter audit_table_name in storage section
turns on dynamodb backend.
The table will be auto created.
S3 sessions backend
-------------------
If audit_sessions_uri is specified to s3://bucket-name
node or proxy depending on recording mode
will start uploading the recorded sessions
to the bucket.
If the bucket does not exist, teleport will
attempt to create a bucket with versioning and encryption
turned on by default.
Teleport will turn on bucket-side encryption for the tarballs
using aws:kms key.
File sessions backend
---------------------
If audit_sessions_uri is specified to file:///folder
teleport will start writing tarballs to this folder instead
of sending records to the file server.
This is helpful for plugin writers who can use fuse or NFS
mounted storage to handle the data.
Working dynamic configuration.
