2019-12-16 17:11:33 +00:00
# Pueue
2020-01-29 19:47:24 +00:00
[![GitHub Actions Workflow ](https://github.com/nukesor/pueue/workflows/Test%20build/badge.svg )](https://github.com/Nukesor/comfy-table/actions)
2019-12-16 22:03:11 +00:00
[![Crates.io ](https://img.shields.io/crates/v/pueue )](https://crates.io/crates/pueue)
2019-12-16 17:11:33 +00:00
[![MIT Licence ](https://img.shields.io/pypi/l/pueue.svg )](https://github.com/Nukesor/pueue/blob/master/LICENSE)
2020-01-29 19:47:24 +00:00
[![dependency status ](https://deps.rs/repo/github/nukesor/pueue/status.svg )](https://deps.rs/repo/github/nukesor/pueue)
2019-12-16 17:11:33 +00:00
[![Patreon ](https://github.com/Nukesor/images/blob/master/patreon-donate-blue.svg )](https://www.patreon.com/nukesor)
2019-12-20 15:52:08 +00:00
[![Paypal ](https://github.com/Nukesor/images/blob/master/paypal-donate-blue.svg )](https://www.paypal.me/arnebeer/)
2019-12-16 17:11:33 +00:00
2019-12-17 02:19:27 +00:00
![Pueue ](https://raw.githubusercontent.com/Nukesor/images/master/pueue.gif )
2019-12-16 17:11:33 +00:00
2019-12-17 02:41:24 +00:00
Pueue is a command-line task management tool for sequential and parallel execution of long-running tasks.
2020-01-29 20:39:07 +00:00
Simply put, it's a tool that processes a queue of shell commands.
2019-12-16 22:03:11 +00:00
On top of that, there are a lot of convenience features and abstractions.
2019-12-16 17:11:33 +00:00
2019-12-17 02:41:24 +00:00
Since Pueue is not bound to any terminal, you can control your tasks from any terminal on the same machine.
The queue will be continuously processed, even if you no longer have any active ssh session.
2019-12-16 17:11:33 +00:00
It provides functionality for:
2019-12-17 02:41:24 +00:00
- Scheduling commands that will be executed in their respective working directories
2019-12-16 17:11:33 +00:00
- Easy output inspection.
- Interaction with running processes
2020-05-11 09:04:44 +00:00
- Pause/Resume resume tasks, when you need some processing power right NOW!
2019-12-16 17:11:33 +00:00
- Manipulation of the scheduled task order
- Running multiple tasks at once (You can decide how many concurrent tasks you want to run)
2020-05-11 09:04:44 +00:00
- Group. Each group acts as their own queue.
2020-02-02 13:35:45 +00:00
- Works on Linux and MacOS and partially on Windows.
2020-05-04 16:33:08 +00:00
**Disclaimer:** Windows support isn't fully there yet. This means:
- Pausing/resuming commands doesn't work for now.
- Pueue only supports `powershell` for executing commands, keep this in mind when writing commands.
2019-12-16 17:11:33 +00:00
## Why should I use it?
2020-01-29 20:47:39 +00:00
Imagine having to unpack or transfer large amounts of data from different directories to other directories.
Usually something like this ends with about 10 open terminals/tmux sessions and an over-challenged hard drive.
2019-12-16 17:11:33 +00:00
2019-12-17 02:39:35 +00:00
Pueue is specifically designed for these situations. It executes long-running tasks in their respective directories, without being bound to any terminal.
2019-12-16 17:11:33 +00:00
2020-01-19 16:37:25 +00:00
**A few possible applications:**
2019-12-16 17:11:33 +00:00
- Copying huge amounts of stuff
- Compression tasks
- Movie encoding
- `rsync` tasks
2020-01-29 20:47:39 +00:00
- Anything that takes longer than 5 minutes
2019-12-16 17:11:33 +00:00
Pueue made at least my life a lot easier on many occasions.
2020-01-29 20:39:07 +00:00
If you like the project and feel like something is missing, please create an issue.
2019-12-17 02:39:35 +00:00
I'm always open to suggestions and already implemented a few users requested features.
2019-12-16 17:11:33 +00:00
2020-01-29 20:39:07 +00:00
PRs are of course very welcome!
2019-12-16 17:11:33 +00:00
## Installation:
There are three different ways to install Pueue.
2020-02-12 21:59:02 +00:00
**Package Manager**
Use your system's package manager.
This will usually deploy service files and completions automatically.
Pueue has been packaged for:
2020-01-19 16:37:25 +00:00
2020-02-12 21:59:02 +00:00
- Arch Linux's AUR: e.g. `yay -S pueue` .
- NixOS
- Homebrew
**Via Cargo**
2020-01-19 16:37:25 +00:00
You'll need Rust version `>=1.39`
2020-01-19 16:38:16 +00:00
```
cargo install pueue
```
2020-01-19 16:37:25 +00:00
This will install pueue to `~/.cargo/bin/pueue`
2020-01-19 16:38:16 +00:00
**From Source**
2020-01-19 16:37:25 +00:00
You'll need Rust version `>=1.39`
```
git clone git@github.com:Nukesor/pueue
cd pueue
cargo install --path .
```
This will install pueue to `~/.cargo/bin/pueue`
2019-12-16 17:11:33 +00:00
2020-01-29 20:39:07 +00:00
## Starting the Daemon
### Local
Just run `pueued` anywhere on your commandline. It'll exit if you close the terminal, though.
### Background
To fork `pueued` into the background, add the `-d` or `--daemonize` flag. E.g. `pueued -d` . \
The daemon can be then shut down using the client: `pueue shutdown`
### Systemd
If you use Systemd and don't install Pueue with a package manager, place `pueued.service` in `/etc/systemd/user/` .
Afterward, every user can start/enable their own session with:
systemctl --user start pueued.service
systemctl --user enable pueued.service
2019-12-16 17:11:33 +00:00
## How to use it:
2020-02-02 15:19:14 +00:00
**Adding Commands:**
2020-02-02 15:18:43 +00:00
2020-02-02 15:14:04 +00:00
To add a command just write: `pueue add sleep 60` \
2020-02-02 15:15:38 +00:00
If you want to add flags to the command, you can either:
2020-02-02 15:14:04 +00:00
- add `--` => `pueue add -- ls -al`
- surround the command with a string `pueue add 'ls -al'`
2020-02-02 15:18:43 +00:00
The command will then be added and scheduled for execution, as if you executed it right now and then.
2020-02-02 15:14:04 +00:00
For normal operation it's recommended to add an alias to your shell's rc.\
2020-02-02 15:15:38 +00:00
E.g.: `alias pad='pueue add --'`
2020-02-02 15:14:04 +00:00
Surrounding a command with quotes is also required, if your command contains escaped characters.\
2020-02-02 15:15:38 +00:00
For instance `pueue add ls /tmp/long\ path` will result in the execution of `sh -c ls /tmp/long path` , which will then break, as the escaped space is not passed to Pueue.
2019-12-17 01:51:06 +00:00
2020-02-02 15:18:43 +00:00
**See what's going on:**
2020-02-02 15:20:30 +00:00
To get the status of currently running commands, just type `pueue status` .
2020-02-02 15:18:43 +00:00
To look at the output of a finished command use `pueue log` or `pueue log $task_id` .
2020-02-02 15:20:30 +00:00
If you want to see output of a running command use `git show $task_id` .
2020-02-02 15:18:43 +00:00
You can also use the `-f` and `-e` flag to get a live view of the output.
2019-12-16 17:11:33 +00:00
2020-02-02 15:19:14 +00:00
**Pitfalls:**
2019-12-17 01:51:06 +00:00
2020-02-02 13:35:45 +00:00
To avoid common pitfalls, please read the [FAQ Section ](https://github.com/Nukesor/pueue/blob/master/FAQ.md ).
2019-12-17 01:51:06 +00:00
There is a help option (-h) for all commands.
2019-12-16 17:11:33 +00:00
```
2020-05-04 16:33:08 +00:00
Pueue client 0.4.0
2019-12-16 17:11:33 +00:00
Arne Beer < contact @ arne . beer >
Interact with the Pueue daemon
USAGE:
2020-01-19 16:45:54 +00:00
pueue [FLAGS] [OPTIONS] < SUBCOMMAND >
2019-12-16 17:11:33 +00:00
FLAGS:
-h, --help Prints help information
-V, --version Prints version information
-v, --verbose Verbose mode (-v, -vv, -vvv)
OPTIONS:
2020-01-19 16:45:54 +00:00
-p, --port < port > The port for the daemon. Overwrites the port in the config file
2019-12-16 17:11:33 +00:00
SUBCOMMANDS:
2020-04-03 16:22:15 +00:00
add Enqueue a task for execution
clean Remove all finished tasks from the list (also clears logs)
completions Generates shell completion files. Ingore for normal operations
edit Edit the command or the path of a stashed or queued task
enqueue Enqueue stashed tasks. They'll be handled normally afterwards
help Prints this message or the help of the given subcommand(s)
kill Kill either all or only specific running tasks
log Display the log output of finished tasks
parallel Set the amount of allowed parallel tasks
pause Pause the daemon and all running tasks. A paused daemon won't start any new tasks. Daemon and
tasks can be continued with `start` Can also be used to pause specific tasks
remove Remove tasks from the list. Running or paused tasks need to be killed first
reset Kill all running tasks, remove all tasks and reset max_id
restart Enqueue tasks again
send Send something to a task. Useful for sending confirmations ('y\n')
show Show the output of a currently running task This command allows following (like `tail -f` )
shutdown Remotely shut down the daemon. Should only be used if the daemon isn't started by a service
manager
start Wake the daemon from its paused state and continue all paused tasks. Can be used to resume or
start specific tasks
stash Stashed tasks won't be automatically started. Either `enqueue` them, to be normally handled or
explicitly `start` them
status Display the current status of all tasks
switch Switches the queue position of two commands. Only works on queued and stashed commands
2020-05-04 16:33:08 +00:00
2019-12-16 17:11:33 +00:00
```
## Configs
2020-04-03 14:56:00 +00:00
The configuration file of Pueue is located in `~/.config/pueue.yml` .
The default will be generated after starting pueue once.
2019-12-16 17:11:33 +00:00
```
---
client:
daemon_port: "6924"
2019-12-16 23:46:59 +00:00
secret: "your_secret"
2020-05-01 19:39:16 +00:00
read_local_logs: true
2019-12-16 17:11:33 +00:00
daemon:
pueue_directory: /home/$USER/.local/share/pueue
default_parallel_tasks: 1
2020-04-03 14:56:00 +00:00
pause_on_failure: false
2019-12-16 17:11:33 +00:00
port: "6924"
2019-12-16 23:46:59 +00:00
secret: "your_secret"
2019-12-16 17:11:33 +00:00
```
2019-12-16 23:46:59 +00:00
**Client**:
- `daemon_port` The port the client tries to connect to.
- `secret` The secret, that's used for authentication
2020-05-01 19:39:16 +00:00
- `read_local_logs` If the client runs as the same user (and on the same machine) as the daemon, logs don't have to be sent via the socket but rather read directly.
2019-12-16 17:11:33 +00:00
**Daemon**:
2019-12-17 02:39:35 +00:00
- `pueue_directory` The location Pueue uses for its intermediate files and logs.
2019-12-16 17:11:33 +00:00
- `default_parallel_tasks` Determines how many tasks should be processed concurrently.
2020-05-01 20:02:44 +00:00
- `pause_on_failure` If set to `true` , the daemon stops starting new task as soon as a single task fails. Already running tasks will continue.
2019-12-16 17:11:33 +00:00
- `port` The port the daemon should listen to.
2019-12-16 23:46:59 +00:00
- `secret` The secret, that's used for authentication
2019-12-16 17:11:33 +00:00
## Logs
All logs can be found in `${pueue_directory}/logs` .
Logs of previous Pueue sessions will be created whenever you issue a `reset` or `clean` .
In case the daemon fails or something goes wrong, the daemon will print to `stdout` /`stderr`.
If the daemon crashes or something goes wrong, please set the debug level to `-vvvv` and create an issue with the log!
If you want to dig right into it, you can compile and run it yourself with a debug build.
This would help me a lot!
2020-01-25 20:26:01 +00:00
## Utilities
2020-05-11 09:04:44 +00:00
### Groups
Grouping tasks can be useful, whenever your tasks utilize different system resources.
For example, it could be very useful to have an `io` group for tasks that copy large files, while your cpu-heavy reencoding tasks are in group `cpu` .
The parallelism setting of `io` could then be set to `1` and `cpu` be set to `2` .
As a result, there'll always be a single task that copies stuff, while two tasks try to utilize your cpu as good as possible.
This removes the problem of scheduling tasks in a way that the system might get slow.
At the same time, you're able to maximize resource utilization.
2020-05-09 21:33:08 +00:00
### Callbacks
2020-05-09 22:13:34 +00:00
You can specify a callback that will be called every time a task finishes.
2020-05-09 21:33:08 +00:00
The callback can be parameterized with some variables.
2020-05-09 22:13:34 +00:00
These are the available variables that can be used to create a command:
2020-05-09 21:33:08 +00:00
- `{{ id }}`
- `{{ command }}`
- `{{ path }}`
- `{{ result }}` (Success, Killed, etc.)
2020-05-11 12:02:08 +00:00
- `{{ group }}`
2020-05-09 21:33:08 +00:00
Example callback:
```
notify-send "Task {{ id }}" "Command: {{ command }} finished with {{ result }}"
```
2020-03-25 21:10:55 +00:00
### Shell completion files
Shell completion files can be created on the fly with `pueue completions $shell $directory` .
There's also a `build_completions.sh` script, which creates all completion files in the `utiles/completions` directory.
2019-12-17 02:39:35 +00:00
### JSON Support
2019-12-16 17:11:33 +00:00
The Pueue client `status` and `log` commands support JSON output with the `-j` flag.
This can be used to easily incorporate it into window manager bars, such as i3bar.
2020-01-25 20:26:01 +00:00
## Scripting
When calling pueue commands in a script, you might need to sleep for a short amount of time for now.
The pueue server processes requests asynchronously, whilst the TaskManager runs it's own update loop with a small sleep.
(The TaskManager handles everything related to starting, stopping and communicating with processes.)
A sleep in scripts will probably become irrelevant, as soon as this bug in rust-lang is fixed: https://github.com/rust-lang/rust/issues/39364
2019-12-16 17:11:33 +00:00
Copyright © 2019 Arne Beer ([@Nukesor](https://github.com/Nukesor))