valkey task distribution

This commit is contained in:
JMARyA 2025-03-08 21:25:02 +01:00
parent e827faaa3f
commit f381f30e27
Signed by: jmarya
GPG key ID: 901B2ADDF27C2263
7 changed files with 866 additions and 26 deletions

View file

@ -33,7 +33,7 @@ let res: (i32, i32) = rally(items, |item: &i32| {
### Background Tasks
Easily run tasks in the background without blocking the main thread. This is useful for code that needs be run without waiting for a result.
Easily run tasks in the background without blocking the main thread. This is useful for code that needs to be run without waiting for a result.
```rust
fn handle() {
@ -85,6 +85,7 @@ pub fn multiply(a: i32, b: i32) -> i32 {
```
After initialization these functions can then be called anywhere and will be processed eventually by whatever worker picks it up.
Additionally there are new functions derived from your function. See the below example:
```rust
fn main() {
@ -94,12 +95,63 @@ fn main() {
manager = multiply_init(manager);
let manager = manager.spawn();
// works like the original function
let res = multiply(2, 2);
// async
let e = take_time_async(1500);
println!("This will run right after!");
// ...
// is OUR value ready?
println!("the value is {}", e.wait());
// Shutdown worker thread
multiply_shutdown();
manager.join().unwrap();
}
```
**Future Plans**: The current system works in-memory with a single worker thread processing tasks. In the future, we plan to extend this to support distributed task queues across multiple machines, enabling shared workloads and infinite scalability.
These tasks can now be distributed with Valkey.
Make sure you have a Valkey server running and the `$VALKEY_URL` environment variable is set for your application:
```yml
services:
valkey:
image: valkey/valkey
ports:
- 6379:6379
```
Then you can spawn worker threads like that:
```rust
fn main() {
let mut s = ServiceManager::new().mode(comrade::service::ServiceMode::Decay);
s = multiply_init_union(s);
s = myfn_init_union(s);
let s = s.spawn();
log::info!("Spawned workers. Working for 1 minute");
std::thread::sleep(Duration::from_secs(60));
myfn_shutdown();
multiply_shutdown();
s.join().unwrap();
}
```
When workers are running, you can use them like:
```rust
fn main() {
// Register workers in union
myfn_register_union();
// Will be computed somewhere else
let x = myfn(50);
println!("x is {x}");
}
```