mirror of
https://github.com/nukesor/pueue
synced 2024-09-30 13:03:36 +00:00
250 lines
11 KiB
Markdown
250 lines
11 KiB
Markdown
# Pueue
|
|
|
|
[](https://github.com/Nukesor/pueue/actions)
|
|
[](https://crates.io/crates/pueue)
|
|
[](https://opensource.org/licenses/MIT)
|
|
[](https://aur.archlinux.org/packages/pueue/)
|
|
[](https://github.com/nukesor/pueue/releases)
|
|
[](https://www.patreon.com/nukesor)
|
|
[](https://www.paypal.me/arnebeer/)
|
|
|
|
<!---[](https://deps.rs/repo/github/nukesor/pueue) --->
|
|
|
|
|
|
|
|
Pueue is a command-line task management tool for sequential and parallel execution of long-running tasks.
|
|
|
|
Simply put, it's a tool that processes a queue of shell commands.
|
|
On top of that, there are a lot of convenience features and abstractions.
|
|
|
|
Since Pueue is not bound to any terminal, you can control your tasks from any terminal on the same machine.
|
|
The queue will be continuously processed, even if you no longer have any active ssh session.
|
|
|
|
**Announcement:** The location of the configuration file changed with v0.8.0.
|
|
The configuration and alias file is now located in a `pueue` subdirectory.
|
|
|
|
- [Features](https://github.com/Nukesor/pueue#features)
|
|
- [Why should I use it](https://github.com/Nukesor/pueue#why-should-i-use-it)
|
|
- [Installation](https://github.com/Nukesor/pueue#installation)
|
|
- [How to use it](https://github.com/Nukesor/pueue#how-to-use-it)
|
|
- [Advantages over Using a Terminal Multiplexer](https://github.com/Nukesor/pueue#advantages-over-using-a-terminal-multiplexer)
|
|
|
|
### Features
|
|
|
|
- Schedule commands that will be executed in their respective working directories.
|
|
- Easy output inspection.
|
|
- Interaction with running processes.
|
|
- Pause/resume tasks, when you need some processing power right NOW!
|
|
- Manipulation of the scheduled task order.
|
|
- Run multiple tasks at once. You can decide how many concurrent tasks you want to run.
|
|
- Group tasks. Each group acts as their own queue and can have several tasks running in parallel.
|
|
- A callback hook to, for instance, set up desktop notifications.
|
|
- A lot more. Check the -h options for each subcommand for detailed options.
|
|
|
|
Works on Linux and partially on MacOS and Windows.
|
|
|
|
**Disclaimer:** Windows and MacOS don't yet support the full feature set:
|
|
|
|
#### Windows
|
|
|
|
The Windows version needs someone to implement process handling!
|
|
Right now there's pretty much no Windows specific code which means:
|
|
|
|
- Pausing/resuming commands doesn't work for now.
|
|
- Sending signals to processes' children doesn't work.
|
|
- We cannot ensure that there aren't dangling processes on `kill`.
|
|
- Pueue only supports `powershell` for executing commands, keep this in mind when writing commands.
|
|
|
|
#### MacOs
|
|
|
|
The `psutil` library, which is used for MacOS is a horrible mess.
|
|
It's missing a lot of essential features and has a really bad code style.\
|
|
Someone has to add a proper replacement and re-implement the MacOs specific process handling code.
|
|
Until then:
|
|
|
|
- Sending signals to processes' children doesn't work.
|
|
- We cannot ensure that there aren't dangling processes on `kill`.
|
|
|
|
## Why should I use it
|
|
|
|
Consider this scenario: You have to unpack large amounts of data into various directories.
|
|
Usually something like this ends with 10+ open terminals/tmux sessions and an over-challenged hard drive.
|
|
|
|
Another scenario might be, that you want to re-encode 10 movies and each re-encode takes 10+ hours.
|
|
Creating a chained command with `&&`s isn't ergonomic at all and running that many re-encodes in parallel will break your CPU.
|
|
|
|
Pueue is specifically designed for these situations.\
|
|
You can schedule your task and continue on the same shell without waiting.
|
|
You can specify how many tasks should run in parallel and group tasks to maximize system resource utilization.\
|
|
Since everything is run by a daemon, you can simply log off your server and check on your tasks' progress whenever you want.\
|
|
Heck, you can even set up desktop notifications to get notified or execute parameterized commands every time a task finishes.
|
|
|
|
**A few possible applications:**
|
|
|
|
- Copying large amounts of data
|
|
- Machine learning
|
|
- Compression tasks
|
|
- Movie encoding
|
|
- `rsync` tasks
|
|
- Anything that takes longer than 5 minutes
|
|
|
|
Pueue made at least my life a lot easier on many occasions.
|
|
|
|
If you like the project, feel free to give it at try!
|
|
If you feel like something is missing, please create an issue :).\
|
|
PRs are of course very welcome!
|
|
|
|
## Installation
|
|
|
|
There are four different ways to install Pueue.
|
|
|
|
#### Package Manager
|
|
|
|
Use your system's package manager.
|
|
This will usually deploy service files and completions automatically.
|
|
Pueue has been packaged for:
|
|
|
|
- Arch Linux's AUR: e.g. `yay -S pueue`.
|
|
- NixOS
|
|
- Homebrew
|
|
|
|
#### Prebuild Binaries
|
|
|
|
Statically linked (if possible) binaries for Linux (incl. ARM), Mac OS and Windows are built on each release.
|
|
|
|
You can download the binaries for the client and the daemon (`pueue` and `pueued`) for each release on the [release page](https://github.com/Nukesor/pueue/releases).
|
|
|
|
Just download both binaries for your system, rename them to `pueue` and `pueued` and place them in your \$PATH/program folder.
|
|
|
|
#### Via Cargo
|
|
|
|
You'll need Rust version `>=1.39`
|
|
|
|
```bash
|
|
cargo install pueue
|
|
```
|
|
|
|
This will install pueue to `~/.cargo/bin/pueue`
|
|
|
|
#### From source
|
|
|
|
You'll need Rust version `>=1.39`
|
|
|
|
```bash
|
|
git clone git@github.com:Nukesor/pueue
|
|
cd pueue
|
|
cargo install --path .
|
|
```
|
|
|
|
This will install pueue to `~/.cargo/bin/pueue`
|
|
|
|
## How to use it
|
|
|
|
Check out wiki to [get you started](https://github.com/Nukesor/pueue/wiki/Get-started) :).
|
|
|
|
There are also detailed sections for (hopefully) every important feature:
|
|
|
|
- [Configuration](https://github.com/Nukesor/pueue/wiki/Configuration)
|
|
- [Groups](https://github.com/Nukesor/pueue/wiki/Groups)
|
|
- [Miscellaneous](https://github.com/Nukesor/pueue/wiki/Miscellaneous)
|
|
- [Connect to remote](https://github.com/Nukesor/pueue/wiki/Connect-to-remote)
|
|
|
|
On top of that, there is a help option (-h) for all commands.
|
|
|
|
```text
|
|
Pueue client 0.8.0
|
|
Arne Beer <contact@arne.beer>
|
|
Interact with the Pueue daemon
|
|
|
|
USAGE:
|
|
pueue [FLAGS] [OPTIONS] <SUBCOMMAND>
|
|
|
|
FLAGS:
|
|
-h, --help Prints help information
|
|
-V, --version Prints version information
|
|
-v, --verbose Verbose mode (-v, -vv, -vvv)
|
|
|
|
OPTIONS:
|
|
-p, --port <port>
|
|
The port for the daemon. Overwrites the port in the config file. Will force TCP mode
|
|
|
|
-u, --unix-socket-path <unix-socket-path>
|
|
The path to the unix socket. Overwrites the path in the config file. Will force Unix-socket mode
|
|
|
|
|
|
SUBCOMMANDS:
|
|
add Enqueue a task for execution
|
|
clean Remove all finished tasks from the list (also clears logs)
|
|
completions Generates shell completion files. This can be ignored during normal operations
|
|
edit Edit the command or path of a stashed or queued task.
|
|
This edits the command of the task by default.
|
|
enqueue Enqueue stashed tasks. They'll be handled normally afterwards
|
|
follow Follow the output of a currently running task. This command works like `tail -f`
|
|
group Manage groups. By default, this will simply display all known groups
|
|
help Prints this message or the help of the given subcommand(s)
|
|
kill Kill specific running tasks or various groups of tasks
|
|
log Display the log output of finished tasks. Prints either all logs or only the logs of specified
|
|
tasks
|
|
parallel Set the amount of allowed parallel tasks
|
|
pause Pause either running tasks or specific groups of tasks.
|
|
By default, pauses the default queue and all it's tasks.
|
|
A paused queue (group) won't start any new tasks.
|
|
Everything can be resumed with 'start'.
|
|
remove Remove tasks from the list. Running or paused tasks need to be killed first
|
|
reset Kill all running tasks, remove all tasks and reset max_task_id
|
|
restart Restart task(s). Identical tasks will be created and by default enqueued
|
|
send Send something to a task. Useful for sending confirmations such as 'y\n'
|
|
shutdown Remotely shut down the daemon. Should only be used if the daemon isn't started by a service
|
|
manager
|
|
start Resume operation of specific tasks or groups of tasks.
|
|
By default, this resumes the default queue and all it's tasks.
|
|
Can also be used force specific tasks to start.
|
|
stash Stashed tasks won't be automatically started. Either enqueue them, to be normally handled or
|
|
explicitly start them
|
|
status Display the current status of all tasks
|
|
switch Switches the queue position of two commands. Only works on queued and stashed commands
|
|
```
|
|
|
|
## Advantages over Using a Terminal Multiplexer
|
|
|
|
One of the most frequent questions is, why one should use Pueue, when there're terminal multiplexer such as Tmux or Screen.
|
|
|
|
My response is, that there're simply a lot of missing convenience features.\
|
|
Here are few examples of Pueue's basic functionality.
|
|
|
|
- The ability to queue commands and not start them all at once
|
|
- Specifying how many tasks should run in parallel
|
|
- Easy pausing/resuming of tasks
|
|
- Pretty and accessible task status overvies
|
|
- No need to attach to a multiple tmux sessions
|
|
|
|
There are a lot more built-in convenience features. You should read the [Wiki](https://github.com/Nukesor/pueue/wiki) for a detailed explanation.
|
|
|
|
Only using your shell's features is definitely possible!
|
|
However, in my opinion, having a tool that's specifically designed for managing tasks is just more efficient and fun.
|
|
|
|
One of my regular use cases is downloading lots of stuff. In this case I want:
|
|
|
|
- At most three parallel downloads, otherwise the other services on my service get no bandwidth.
|
|
- To see at first glance whether a download fails and easily edit and re-schedule it.
|
|
- An easy way to look at output.
|
|
- Everything to be in a uniform interface.
|
|
- It to look pretty and clear.
|
|
- To be able to pause/resume everything in case I need to some bandwidth right now.
|
|
|
|
I used tmux for this stuff all the time before writing Pueue.\
|
|
However, after using it for a really long time, it just kept feeling annoying and inconvenient.
|
|
Up to the point I could not bare it any longer and decided to write something that's better suited for such scenarios.
|
|
|
|
## Contributing
|
|
|
|
Feature requests and pull requests are very much appreciated and welcome!
|
|
|
|
Anyhow, please talk to me a bit about your ideas before you start hacking!
|
|
It's always nice to know what you're working on and I might have a few suggestions or tips :)
|
|
|
|
There's also the [Contribution Guide](https://github.com/Nukesor/pueue/blob/master/CHANGELOG.md), which is supposed to give you a brief overview and introduction into the project.
|
|
|
|
Copyright © 2019 Arne Beer ([@Nukesor](https://github.com/Nukesor))
|
|
|