mirror of
https://github.com/gravitational/teleport
synced 2024-10-19 16:53:57 +00:00
2b15263e49
With the actor model, tokens can be used in multiple ways (picking tools, invoking them, ...), which don't necessarily end up in a final action (sometimes we return a nextStep instead). Streaming responses were another challenge: the agent returned without the completion being over (it returned a routine streaming the deltas sent by the model). This PR introduces a TokenCounter interface that abstracts synchronous and asynchronous token counting. All token-consuming operations must return a TokenCounter. TokensCounters are stored in the agent state and returned once the agent exists. Finally, the token counters are evaluated asynchronously to give the streaming completion requests enough time to finish. |
||
|---|---|---|
| .. | ||
| model | ||
| testutils | ||
| chat.go | ||
| chat_test.go | ||
| client.go | ||
| embedding.go | ||
| embeddings.go | ||
| embeddings_test.go | ||
| knnretriever.go | ||
| knnretriever_test.go | ||
| simpleretriever.go | ||
| simpleretriever_test.go | ||