teleport/lib/ai
Hugo Shaka c326b8c3ef
ai: Add a node embedding watcher (#27204)
* ai: add embeddings basic support

- add Embeddings service and its local implementation
- add Embedding type and proto message
- add nodeEmbeddingCollector tracking nodes
- add NodeEmbeddingWatcher watching for events adn sending them to the
  collector
- add the Embedder interface and its openai implementation

* ai: adapt embeddings to the vector index

* fixup! ai: adapt embeddings to the vector index

* fixup! fixup! ai: adapt embeddings to the vector index

* Update lib/service/service.go

Co-authored-by: Jakub Nyckowski <jakub.nyckowski@goteleport.com>

* address feedback pt.1

* address feedback pt.2: store protobuf message in backend

* address feedback pt.3: have GetEmbeddings return a stream

* Update lib/services/embeddings.go

Co-authored-by: rosstimothy <39066650+rosstimothy@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Edoardo Spadolini <edoardo.spadolini@goteleport.com>

* address feedback pt.4: extract embedding logic out of Embeddings service

* fixup! address feedback pt.4: extract embedding logic out of Embeddings service

* address feedback pt.5: simpler error handling when embedding fails

* fix tests pt.1

* fix tests pt.2

* fix tests pt.3

* [Assist] Replace embedding watcher (#27953)

Change the way how the embeddings are calculated. Instead of creating a watcher in Auth, we will process all nodes every hour and process embeddings if any embeddings are missing or any node has been updated.

---------

Co-authored-by: Jakub Nyckowski <jakub.nyckowski@goteleport.com>
Co-authored-by: rosstimothy <39066650+rosstimothy@users.noreply.github.com>
Co-authored-by: Edoardo Spadolini <edoardo.spadolini@goteleport.com>
2023-06-21 01:28:56 +00:00
..
model [Assist] Scaffold the chat-loop onto a multi-step thinking model (#27075) 2023-06-08 14:15:29 +00:00
chat.go [Assist] Scaffold the chat-loop onto a multi-step thinking model (#27075) 2023-06-08 14:15:29 +00:00
client.go [Assist] Scaffold the chat-loop onto a multi-step thinking model (#27075) 2023-06-08 14:15:29 +00:00
embedding.go ai: Add a node embedding watcher (#27204) 2023-06-21 01:28:56 +00:00
embeddings.go ai: Add a node embedding watcher (#27204) 2023-06-21 01:28:56 +00:00
embeddings_test.go ai: Add a node embedding watcher (#27204) 2023-06-21 01:28:56 +00:00
knnretriever.go ai: Add a node embedding watcher (#27204) 2023-06-21 01:28:56 +00:00
knnretriever_test.go ai: Add a node embedding watcher (#27204) 2023-06-21 01:28:56 +00:00