From cdb71f2ce6bf0666ebb5abc6d68f8355216a0ab7 Mon Sep 17 00:00:00 2001 From: JMARyA Date: Tue, 2 Apr 2024 21:52:49 +0200 Subject: [PATCH] add soft-serve --- .../applications/development/soft-serve.md | 695 ++++++++++++++++++ 1 file changed, 695 insertions(+) create mode 100644 technology/applications/development/soft-serve.md diff --git a/technology/applications/development/soft-serve.md b/technology/applications/development/soft-serve.md new file mode 100644 index 0000000..5dd985a --- /dev/null +++ b/technology/applications/development/soft-serve.md @@ -0,0 +1,695 @@ +--- +obj: application +repo: https://github.com/charmbracelet/soft-serve +rev: 2024-04-01 +--- + +# soft-serve +A tasty, self-hostable [Git](../../dev/Git.md) server for the command line. 🍦 + +![SoftServe][SoftServe] + +![Soft Serve screencast](https://github.com/charmbracelet/soft-serve/assets/42545625/c754c746-dc4c-44a6-9c39-28649264cbf2) + +- Easy to navigate TUI available over [SSH](../network/SSH.md) +- Clone repos over [SSH](../network/SSH.md), [HTTP](../../internet/HTTP.md), or [Git](../../dev/Git.md) protocol +- Git LFS support with both [HTTP](../../internet/HTTP.md) and [SSH](../network/SSH.md) backends +- Manage repos with [SSH](../network/SSH.md) +- Create repos on demand with [SSH](../network/SSH.md) or `git push` +- Browse repos, files and commits with [SSH](../network/SSH.md)-accessible UI +- Print files over [SSH](../network/SSH.md) with or without syntax highlighting and line numbers +- Easy access control + - [SSH](../network/SSH.md) authentication using public keys + - Allow/disallow anonymous access + - Add collaborators with [SSH](../network/SSH.md) public keys + - Repos can be public or private + - User access tokens + +## Where can I see it? +Just run `ssh git.charm.sh` for an example. You can also try some of the following commands: + +```shell +# Jump directly to a repo in the TUI +ssh git.charm.sh -t soft-serve + +# Print out a directory tree for a repo +ssh git.charm.sh repo tree soft-serve + +# Print a specific file +ssh git.charm.sh repo blob soft-serve cmd/soft/main.go + +# Print a file with syntax highlighting and line numbers +ssh git.charm.sh repo blob soft-serve cmd/soft/main.go -c -l +``` + +Or you can use Soft Serve to browse local repositories using `soft browse [directory]` or running `soft` within a [Git](../../dev/Git.md) repository. + +## Docker +The official Soft Serve [Docker](../../tools/Docker.md) images are available at [charmcli/soft-serve](https://hub.docker.com/r/charmcli/soft-serve). Development and nightly builds are available at [ghcr.io/charmbracelet/soft-serve](https://github.com/charmbracelet/soft-serve/pkgs/container/soft-serve) + +```shell +docker pull charmcli/soft-serve:latest +``` + +Here’s how you might run `soft-serve` as a container. Keep in mind that repositories are stored in the `/soft-serve` directory, so you’ll likely want to mount that directory as a volume in order keep your repositories backed up. + +```shell +docker run \ + --name=soft-serve \ + --volume /path/to/data:/soft-serve \ + --publish 23231:23231 \ + --publish 23232:23232 \ + --publish 23233:23233 \ + --publish 9418:9418 \ + --restart unless-stopped \ + charmcli/soft-serve:latest +``` + +Or by using [docker compose](../../tools/Docker%20Compose.md): +```yaml +--- +version: "3.1" +services: + soft-serve: + image: charmcli/soft-serve:latest + container_name: soft-serve + volumes: + - /path/to/data:/soft-serve + ports: + - 23231:23231 + - 23232:23232 + - 23233:23233 + - 9418:9418 + restart: unless-stopped +``` + +## Setting up a server +Make sure `git` is installed, then run `soft serve`. That’s it. + +This will create a `data` directory that will store all the repos, ssh keys, and database. + +To change the default data path use `$SOFT_SERVE_DATA_PATH` environment variable. + +```shell +SOFT_SERVE_DATA_PATH=/var/lib/soft-serve soft serve +``` + +When you run Soft Serve for the first time, make sure you have the `$SOFT_SERVE_INITIAL_ADMIN_KEYS` environment variable is set to your ssh authorized key. Any added key to this variable will be treated as admin with full privileges. + +Using this environment variable, Soft Serve will create a new `admin` user that has full privileges. You can rename and change the user settings later. + +### Systemd +Most Linux OSes use [Systemd](../../linux/systemd/Systemd.md) as an init system and service management. You can use [Systemd](../../linux/systemd/Systemd.md) to manage Soft Serve as a service on your host machine. + +Our Soft Serve deb/rpm packages come with [Systemd](../../linux/systemd/Systemd.md) service files pre-packaged. You can install `soft-serve` from our Apt/Yum repositories. + +#### Writing a Systemd Service File +> **Note** you can skip this section if you are using our deb/rpm packages or installed Soft Serve from our Apt/Yum repositories. + +Start by writing a [Systemd](../../linux/systemd/Systemd.md) service file to define how your Soft Serve server should start. + +First, we need to specify where the data should live for our server. Here I will be choosing `/var/local/lib/soft-serve` to store the server's data. Soft Serve will look for this path in the `$SOFT_SERVE_DATA_PATH` environment variable. + +Make sure this directory exists before proceeding. +```shell +sudo mkdir -p /var/local/lib/soft-serve +``` + +We will also create a `/etc/soft-serve.conf` file for any extra server settings that we want to override. +``` +# Config defined here will override the config in /var/local/lib/soft-serve/config.yaml +# Keys defined in `SOFT_SERVE_INITIAL_ADMIN_KEYS` will be merged with +# the `initial_admin_keys` from /var/local/lib/soft-serve/config.yaml. +# +#SOFT_SERVE_GIT_LISTEN_ADDR=:9418 +#SOFT_SERVE_HTTP_LISTEN_ADDR=:23232 +#SOFT_SERVE_SSH_LISTEN_ADDR=:23231 +#SOFT_SERVE_SSH_KEY_PATH=ssh/soft_serve_host_ed25519 +#SOFT_SERVE_INITIAL_ADMIN_KEYS='ssh-ed25519 AAAAC3NzaC1lZDI1...' +``` + +> **Note** Soft Serve stores its server configuration and settings in `config.yaml` under its _data path_ directory specified using `$SOFT_SERVE_DATA_PATH` environment variable. + +Now, let's write a new `/etc/systemd/system/soft-serve.service` Systemd service file: + +``` +[Unit] +Description=Soft Serve git server 🍦 +Documentation=https://github.com/charmbracelet/soft-serve +Requires=network-online.target +After=network-online.target + +[Service] +Type=simple +Restart=always +RestartSec=1 +ExecStart=/usr/bin/soft serve +Environment=SOFT_SERVE_DATA_PATH=/var/local/lib/soft-serve +EnvironmentFile=-/etc/soft-serve.conf +WorkingDirectory=/var/local/lib/soft-serve + +[Install] +WantedBy=multi-user.target +``` + +Great, we now have a [Systemd](../../linux/systemd/Systemd.md) service file for Soft Serve. The settings defined here may vary depending on your specific setup. This assumes that you want to run Soft Serve as `root`. + +#### Start Soft Serve on boot +Now that we have our Soft Serve [Systemd](../../linux/systemd/Systemd.md) service file in-place, let's go ahead and enable and start Soft Serve to run on-boot. + +```shell +# Reload systemd daemon +sudo systemctl daemon-reload +# Enable Soft Serve to start on-boot +sudo systemctl enable soft-serve.service +# Start Soft Serve now!! +sudo systemctl start soft-serve.service +``` + +You can monitor the server logs using `journalctl -u soft-serve.service`. Use `-f` to _tail_ and follow the logs as they get written. + +### Server Configuration +Once you start the server for the first time, the settings will be in `config.yaml` under your data directory. The default `config.yaml` is self-explanatory and will look like this: + +```yaml +# Soft Serve Server configurations + +# The name of the server. +# This is the name that will be displayed in the UI. +name: "Soft Serve" + +# Log format to use. Valid values are "json", "logfmt", and "text". +log_format: "text" + +# The SSH server configuration. +ssh: + # The address on which the SSH server will listen. + listen_addr: ":23231" + + # The public URL of the SSH server. + # This is the address that will be used to clone repositories. + public_url: "ssh://localhost:23231" + + # The path to the SSH server's private key. + key_path: "ssh/soft_serve_host" + + # The path to the SSH server's client private key. + # This key will be used to authenticate the server to make git requests to + # ssh remotes. + client_key_path: "ssh/soft_serve_client" + + # The maximum number of seconds a connection can take. + # A value of 0 means no timeout. + max_timeout: 0 + + # The number of seconds a connection can be idle before it is closed. + idle_timeout: 120 + +# The Git daemon configuration. +git: + # The address on which the Git daemon will listen. + listen_addr: ":9418" + + # The maximum number of seconds a connection can take. + # A value of 0 means no timeout. + max_timeout: 0 + + # The number of seconds a connection can be idle before it is closed. + idle_timeout: 3 + + # The maximum number of concurrent connections. + max_connections: 32 + +# The HTTP server configuration. +http: + # The address on which the HTTP server will listen. + listen_addr: ":23232" + + # The path to the TLS private key. + tls_key_path: "" + + # The path to the TLS certificate. + tls_cert_path: "" + + # The public URL of the HTTP server. + # This is the address that will be used to clone repositories. + # Make sure to use https:// if you are using TLS. + public_url: "http://localhost:23232" + +# The database configuration. +db: + # The database driver to use. + # Valid values are "sqlite" and "postgres". + driver: "sqlite" + # The database data source name. + # This is driver specific and can be a file path or connection string. + # Make sure foreign key support is enabled when using SQLite. + data_source: "soft-serve.db?_pragma=busy_timeout(5000)&_pragma=foreign_keys(1)" + +# Git LFS configuration. +lfs: + # Enable Git LFS. + enabled: true + # Enable Git SSH transfer. + ssh_enabled: false + +# Cron job configuration +jobs: + mirror_pull: "@every 10m" + +# The stats server configuration. +stats: + # The address on which the stats server will listen. + listen_addr: ":23233" +# Additional admin keys. +#initial_admin_keys: +# - "ssh-rsa AAAAB3NzaC1yc2..." +``` + +You can also use [environment variables](../../linux/Environment%20Variables.md), to override these settings. All server settings [environment variables](../../linux/Environment%20Variables.md) start with `$SOFT_SERVE_` followed by the setting name all in uppercase. Here are some examples: + +- `$SOFT_SERVE_NAME`: The name of the server that will appear in the TUI +- `$SOFT_SERVE_SSH_LISTEN_ADDR`: SSH listen address +- `$SOFT_SERVE_SSH_KEY_PATH`: SSH host key-pair path +- `$SOFT_SERVE_HTTP_LISTEN_ADDR`: HTTP listen address +- `$SOFT_SERVE_HTTP_PUBLIC_URL`: HTTP public URL used for cloning +- `$SOFT_SERVE_GIT_MAX_CONNECTIONS`: The number of simultaneous connections to git daemon + +#### Database Configuration +Soft Serve supports both [SQLite](../../dev/programming/SQLite.md) and Postgres for its database. Like all other Soft Serve settings, you can change the database _driver_ and _data source_ using either `config.yaml` or [environment variables](../../linux/Environment%20Variables.md). The default config uses [SQLite](../../dev/programming/SQLite.md) as the default database driver. + +To use Postgres as your database, first create a Soft Serve database: +```shell +psql -h -p -U -c 'CREATE DATABASE soft_serve' +``` + +Then set the database _data source_ to point to your Postgres database. For instance, if you're running Postgres locally, using the default user `postgres` and using a database name `soft_serve`, you would have this config in your config file or environment variable: + +```yaml +db: + driver: "postgres" + data_source: "postgres://postgres@localhost:5432/soft_serve?sslmode=disable" +``` + +[Environment variables](../../linux/Environment%20Variables.md) equivalent: +```shell +SOFT_SERVE_DB_DRIVER=postgres \ +SOFT_SERVE_DB_DATA_SOURCE="postgres://postgres@localhost:5432/soft_serve?sslmode=disable" \ +soft serve +``` + +You can specify a database connection password in the _data source_ url. For example, `postgres://myuser:dbpass@localhost:5432/my_soft_serve_db`. + +#### LFS Configuration +Soft Serve supports both [Git](../../dev/Git.md) LFS [HTTP](../../internet/HTTP.md) and [SSH](../network/SSH.md) protocols out of the box, there is no need to do any extra set up. + +Use the `lfs` config section to customize your Git LFS server. + +> **Note**: The pure-[SSH](../network/SSH.md) transfer is disabled by default. + +## Server Access +Soft Serve at its core manages your server authentication and authorization. Authentication verifies the identity of a user, while authorization determines their access rights to a repository. + +To manage the server users, access, and repos, you can use the [SSH](../network/SSH.md) command line interface. + +Try `ssh localhost -i ~/.ssh/id_ed25519 -p 23231 help` for more info. Make sure you use your key here. + +For ease of use, instead of specifying the key, port, and hostname every time you [SSH](../network/SSH.md) into Soft Serve, add your own Soft Serve instance entry to your [SSH](../network/SSH.md) config. For instance, to use `ssh soft` instead of typing `ssh localhost -i ~/.ssh/id_ed25519 -p 23231`, we can define a `soft` entry in our [SSH](../network/SSH.md) config file `~/.ssh/config`. + +``` +Host soft + HostName localhost + Port 23231 + IdentityFile ~/.ssh/id_ed25519 +``` + +Now, we can do `ssh soft` to [SSH](../network/SSH.md) into Soft Serve. Since `git` is also aware of this config, you can use `soft` as the hostname for your clone commands. + +```shell +git clone ssh://soft/dotfiles +# make changes +# add & commit +git push origin main +``` + +> **Note** The `-i` part will be omitted in the examples below for brevity. You can add your server settings to your sshconfig for quicker access. + +### Authentication +Everything that needs authentication is done using [SSH](../network/SSH.md). Make sure you have added an entry for your Soft Serve instance in your `~/.ssh/config` file. + +By default, Soft Serve gives ready-only permission to anonymous connections to any of the above protocols. This is controlled by two settings `anon-access` and `allow-keyless`. + +- `anon-access`: Defines the access level for anonymous users. Available options are `no-access`, `read-only`, `read-write`, and `admin-access`. Default is `read-only`. +- `allow-keyless`: Whether to allow connections that doesn't use keys to pass. Setting this to `false` would disable access to [SSH](../network/SSH.md) keyboard-interactive, [HTTP](../../internet/HTTP.md), and [Git](../../dev/Git.md) protocol connections. Default is `true`. + +```shell +$ ssh -p 23231 localhost settings +Manage server settings + +Usage: + ssh -p 23231 localhost settings [command] + +Available Commands: + allow-keyless Set or get allow keyless access to repositories + anon-access Set or get the default access level for anonymous users + +Flags: + -h, --help help for settings + +Use "ssh -p 23231 localhost settings [command] --help" for more information about a command. +``` + +> **Note** These settings can only be changed by admins. + +When `allow-keyless` is disabled, connections that don't use [SSH](../network/SSH.md) Public Key authentication will get denied. This means cloning repos over [HTTP](../../internet/HTTP.md)(s) or git:// will get denied. + +Meanwhile, `anon-access` controls the access level granted to connections that use [SSH](../network/SSH.md) Public Key authentication but are not registered users. The default setting for this is `read-only`. This will grant anonymous connections that use [SSH](../network/SSH.md) Public Key authentication `read-only` access to public repos. + +`anon-access` is also used in combination with `allow-keyless` to determine the access level for [HTTP](../../internet/HTTP.md)(s) and git:// clone requests. + +#### SSH +Soft Serve doesn't allow duplicate [SSH](../network/SSH.md) public keys for users. A public key can be associated with one user only. This makes [SSH](../network/SSH.md) authentication simple and straight forward, add your public key to your Soft Serve user to be able to access Soft Serve. + +#### HTTP +You can generate user access tokens through the [SSH](../network/SSH.md) command line interface. Access tokens can have an optional expiration date. Use your access token as the basic auth user to access your Soft Serve repos through [HTTP](../../internet/HTTP.md). + +```shell +# Create a user token +ssh -p 23231 localhost token create 'my new token' +ss_1234abc56789012345678901234de246d798fghi + +# Or with an expiry date +ssh -p 23231 localhost token create --expires-in 1y 'my other token' +ss_98fghi1234abc56789012345678901234de246d7 +``` + +Now you can access to repos that require `read-write` access. + +```shell +git clone http://ss_98fghi1234abc56789012345678901234de246d7@localhost:23232/my-private-repo.git my-private-repo +# Make changes and push +``` + +### Authorization +Soft Serve offers a simple access control. There are four access levels, no-access, read-only, read-write, and admin-access. + +`admin-access` has full control of the server and can make changes to users and repos. + +`read-write` access gets full control of repos. + +`read-only` can read public repos. + +`no-access` denies access to all repos. + +## User Management +Admins can manage users and their keys using the `user` command. Once a user is created and has access to the server, they can manage their own keys and settings. + +To create a new user simply use `user create`: +```shell +# Create a new user +ssh -p 23231 localhost user create beatrice + +# Add user keys +ssh -p 23231 localhost user add-pubkey beatrice ssh-rsa AAAAB3Nz... +ssh -p 23231 localhost user add-pubkey beatrice ssh-ed25519 AAAA... + +# Create another user with public key +ssh -p 23231 localhost user create frankie '-k "ssh-ed25519 AAAATzN..."' + +# Need help? +ssh -p 23231 localhost user help +``` + +Once a user is created, they get `read-only` access to public repositories. They can also create new repositories on the server. + +Users can manage their keys using the `pubkey` command: +```shell +# List user keys +ssh -p 23231 localhost pubkey list + +# Add key +ssh -p 23231 localhost pubkey add ssh-ed25519 AAAA... + +# Wanna change your username? +ssh -p 23231 localhost set-username yolo + +# To display user info +ssh -p 23231 localhost info +``` + +## Repositories +You can manage repositories using the `repo` command. + +```shell +# Run repo help +$ ssh -p 23231 localhost repo help +Manage repositories + +Usage: + ssh -p 23231 localhost repo [command] + +Aliases: + repo, repos, repository, repositories + +Available Commands: + blob Print out the contents of file at path + branch Manage repository branches + collab Manage collaborators + create Create a new repository + delete Delete a repository + description Set or get the description for a repository + hide Hide or unhide a repository + import Import a new repository from remote + info Get information about a repository + is-mirror Whether a repository is a mirror + list List repositories + private Set or get a repository private property + project-name Set or get the project name for a repository + rename Rename an existing repository + tag Manage repository tags + tree Print repository tree at path + +Flags: + -h, --help help for repo + +Use "ssh -p 23231 localhost repo [command] --help" for more information about a command. +``` + +To use any of the above `repo` commands, a user must be a collaborator in the repository. More on this below. + +### Creating Repositories +To create a repository, first make sure you are a registered user. Use the `repo create ` command to create a new repository: + +```shell +# Create a new repository +ssh -p 23231 localhost repo create icecream + +# Create a repo with description +ssh -p 23231 localhost repo create icecream '-d "This is an Ice Cream description"' + +# ... and project name +ssh -p 23231 localhost repo create icecream '-d "This is an Ice Cream description"' '-n "Ice Cream"' + +# I need my repository private! +ssh -p 23231 localhost repo create icecream -p '-d "This is an Ice Cream description"' '-n "Ice Cream"' + +# Help? +ssh -p 23231 localhost repo create -h +``` + +Or you can add your Soft Serve server as a remote to any existing repo, given you have write access, and push to remote: +``` +git remote add origin ssh://localhost:23231/icecream +``` + +After you’ve added the remote just go ahead and push. If the repo doesn’t exist on the server it’ll be created. +``` +git push origin main +``` + +Repositories can be nested too: +```shell +# Create a new nested repository +ssh -p 23231 localhost repo create charmbracelet/icecream + +# Or ... +git remote add charm ssh://localhost:23231/charmbracelet/icecream +git push charm main +``` + +### Deleting Repositories +You can delete repositories using the `repo delete ` command. + +```shell +ssh -p 23231 localhost repo delete icecream +``` + +### Renaming Repositories +Use the `repo rename ` command to rename existing repositories. + +```shell +ssh -p 23231 localhost repo rename icecream vanilla +``` + +### Repository Collaborators +Sometimes you want to restrict write access to certain repositories. This can be achieved by adding a collaborator to your repository. + +Use the `repo collab ` command to manage repo collaborators. + +```shell +# Add collaborator to soft-serve +ssh -p 23231 localhost repo collab add soft-serve frankie + +# Add collaborator with a specific access level +ssh -p 23231 localhost repo collab add soft-serve beatrice read-only + +# Remove collaborator +ssh -p 23231 localhost repo collab remove soft-serve beatrice + +# List collaborators +ssh -p 23231 localhost repo collab list soft-serve +``` + +### Repository Metadata +You can also change the repo's description, project name, whether it's private, etc using the `repo ` command. + +```shell +# Set description for repo +ssh -p 23231 localhost repo description icecream "This is a new description" + +# Hide repo from listing +ssh -p 23231 localhost repo hidden icecream true + +# List repository info (branches, tags, description, etc) +ssh -p 23231 localhost repo icecream info +``` + +To make a repository private, use `repo private [true|false]`. Private repos can only be accessed by admins and collaborators. + +```shell +ssh -p 23231 localhost repo icecream private true +``` + +### Repository Branches & Tags +Use `repo branch` and `repo tag` to list, and delete branches or tags. You can also use `repo branch default` to set or get the repository default branch. + +### Repository Tree +To print a file tree for the project, just use the `repo tree` command along with the repo name as the [SSH](../network/SSH.md) command to your Soft Serve server: + +```shell +ssh -p 23231 localhost repo tree soft-serve +``` + +You can also specify the sub-path and a specific reference or branch. + +```shell +ssh -p 23231 localhost repo tree soft-serve server/config +ssh -p 23231 localhost repo tree soft-serve main server/config +``` + +From there, you can print individual files using the `repo blob` command: + +```shell +ssh -p 23231 localhost repo blob soft-serve cmd/soft/main.go +``` + +You can add the `-c` flag to enable syntax coloring and `-l` to print line numbers: + +```shell +ssh -p 23231 localhost repo blob soft-serve cmd/soft/main.go -c -l + +``` + +Use `--raw` to print raw file contents. This is useful for dumping binary data. + +### Repository webhooks +Soft Serve supports repository [webhooks](../../internet/Webhook.md) using the `repo webhook` command. You can create and manage webhooks for different repository events such as _push_, _collaborators_, and _branch_tag_create_ events. + +``` +Manage repository webhooks + +Usage: + ssh -p 23231 localhost repo webhook [command] + +Aliases: + webhook, webhooks + +Available Commands: + create Create a repository webhook + delete Delete a repository webhook + deliveries Manage webhook deliveries + list List repository webhooks + update Update a repository webhook + +Flags: + -h, --help help for webhook +``` + +## The Soft Serve TUI +Soft Serve TUI is mainly used to browse repos over [SSH](../network/SSH.md). You can also use it to browse local repositories with `soft browse` or running `soft` within a [Git](../../dev/Git.md) repository. + +```shell +ssh localhost -p 23231 +``` + +It's also possible to “link” to a specific repo: +```shell +ssh -p 23231 localhost -t soft-serve +``` + +You can copy text to your clipboard over [SSH](../network/SSH.md). For instance, you can press c on the highlighted repo in the menu to copy the clone command. + +## Hooks +Soft Serve supports git server-side hooks `pre-receive`, `update`, `post-update`, and `post-receive`. This means you can define your own hooks to run on repository push events. Hooks can be defined as a per-repository hook, and/or global hooks that run for all repositories. + +You can find per-repository hooks under the repository `hooks` directory. + +Globs hooks can be found in your `SOFT_SERVE_DATA_PATH` directory under `hooks`. Defining global hooks is useful if you want to run CI/CD for example. + +Here's an example of sending a message after receiving a push event. Create an executable file `/hooks/update`: + +```shell +#!/bin/sh +# +# An example hook script to echo information about the push +# and send it to the client. + +refname="$1" +oldrev="$2" +newrev="$3" + +# Safety check +if [ -z "$GIT_DIR" ]; then + echo "Don't run this script from the command line." >&2 + echo " (if you want, you could supply GIT_DIR then run" >&2 + echo " $0 )" >&2 + exit 1 +fi + +if [ -z "$refname" -o -z "$oldrev" -o -z "$newrev" ]; then + echo "usage: $0 " >&2 + exit 1 +fi + +# Check types +# if $newrev is 0000...0000, it's a commit to delete a ref. +zero=$(git hash-object --stdin