Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

sshwarm

k9s for ssh. Your Linux machines, their services, containers and logs, in one terminal: what's deployed where, what died overnight, what's crash-looping, and the logs of whatever is red a keypress away.

Quickstart

Install

From
Gogo install github.com/cmdoret/sshwarm@latest
Nixnix run github:cmdoret/sshwarm
Binaryreleases: Linux and macOS, amd64 and arm64
Containersee below

List the machines

In ~/.config/sshwarm/hosts.toml, one [[host]] per machine:

[[host]]
name = "db"                  # ssh db: the rest from ~/.ssh/config

[[host]]
name = "web-1"
host = "web-1.example.org"   # ssh -i ~/.ssh/web deploy@web-1.example.org
user = "deploy"
key  = "~/.ssh/web"

Using deploy-rs? Skip the file: sshwarm --flake ./nix. See configuration.

Run

sshwarm                       # browse
sshwarm status                # print the table, exit 1 on any problem
sshwarm status --format json  # for scripts

The machines need systemd, bash, and ssh without a password prompt. Logs and containers need more: see requirements.

Keys

Key
enteropen
l, dlogs, describe
/filter; in logs, search
:ctr webcontainers of web; :ctr all, of every machine. tab completes
?every key

Container

docker run --rm -it \
  -v ./hosts.toml:/hosts.toml:ro \
  -v ~/.ssh/id_ed25519:/root/.ssh/id_ed25519:ro \
  -v ~/.ssh/known_hosts:/root/.ssh/known_hosts:ro \
  ghcr.io/cmdoret/sshwarm --hosts /hosts.toml

No nix inside, so no --flake. ssh refuses a mounted ~/.ssh/config: the hosts file must set host, user and key.

Configuration

No config file: the machines, flags, and environment variables.

Machines

sshwarm takes the first of:

  1. --hosts <file> or --flake <dir>
  2. SSHWARM_HOSTS or SSHWARM_FLAKE
  3. $XDG_CONFIG_HOME/sshwarm/hosts.toml, by default ~/.config/sshwarm/hosts.toml

A hosts file and a flake together are an error.

Hosts file

[[host]]
name = "web-1"               # required: sshwarm's name for it
host = "web-1.example.org"   # default: name
user = "deploy"              # default: ~/.ssh/config's, else yours
key  = "~/.ssh/web"          # default: ~/.ssh/config's, or your agent

ssh looks host (or name) up in ~/.ssh/config for anything else: port, jump host… Unknown keys are an error.

Nix flake

sshwarm --flake ./nix

Reads the flake’s deploy.nodes, as deploy-rs does: each node’s hostname, and its sshUser and sshOpts, or deploy’s. Needs nix.

Flags

FlagVariableDefault
--hosts <file>SSHWARM_HOSTShosts file
--flake <dir>SSHWARM_FLAKEflake with deploy.nodes
--timeout <duration>SSHWARM_TIMEOUT20sprobe timeout, per machine
--refresh <duration>SSHWARM_REFRESH30show often the browser probes
--read-onlySSHWARM_READ_ONLYoffno actions, no shells

A flag beats its variable. SSHWARM_READ_ONLY is on unless empty, 0 or false.

Commands: status [--format text|json] [machine...], and probe <machine>, which prints a machine’s raw probe output. sshwarm <command> --help for more.

Requirements

Each machine runs Linux with systemd and bash. ssh must log in without a prompt: a key or an agent, and a known host key.

Tothe ssh user
see the machine and its unitsneeds nothing more
read logsis root, or in systemd-journal
see, start and stop compose stacks and containersruns docker: docker group, rootless, or passwordless sudo
start and stop units, rebootis root, or runs systemctl with passwordless sudo

Pending reboots are detected on NixOS, Debian, Ubuntu, Fedora, RHEL and Arch. On NixOS, sshwarm also shows the last deploy and its git revision, if the flake sets system.configurationRevision.

Connections

The browser keeps one ssh connection per machine (ControlMaster), its socket in ~/.ssh, closed a minute after you quit. It leaves ssh alone if your ssh config sets ControlPath or ControlMaster for the machine, or ~/.ssh isn’t writable.

Architecture

flowchart LR
    src["hosts.toml<br>or flake"] --> sw[sshwarm]
    sw -- "ssh, every --refresh:<br>probe.sh on stdin" --> m["each machine"]
    m -- "### sections:<br>OS, usage, units, stacks" --> sw
    sw --> ui["browser,<br>or status"]
    ui -. "logs, describe,<br>shell, actions" .-> m
  1. sshwarm reads the machines from a hosts file or a flake.
  2. It pipes probe.sh to every machine over ssh, in parallel, and waits for all. The browser does it again --refresh after the last round ended.
  3. The probe only reads: OS, uptime, usage, pending reboot, systemctl show of every service, docker ps and docker inspect of compose containers. It prints ### <name> sections; a command that fails prints ### error <name>.
  4. sshwarm parses each machine’s output into a report, and rates what’s in it: idle, fine, warning or broken. A machine’s problems are what’s broken, what restarted on its own in the last 10 minutes, a disk 90% full, and the probe’s errors. status exits 1 on any.
  5. Logs, descriptions, shells and actions each run their own ssh command, on demand.

In time

A refresh, a unit’s logs, then a container’s restart:

sequenceDiagram
    actor You
    participant UI as browser
    participant SW as swarm
    participant M as each machine

    loop every --refresh, from the end of the last round
        UI->>SW: probe
        SW->>M: ssh: bash, probe.sh on stdin, to all at once
        M-->>SW: OS, usage, units, containers, in sections
        SW-->>UI: a report per machine, rated
    end

    You->>UI: l, on a unit
    UI->>M: ssh: journalctl -u unit -n 300
    M-->>UI: its last lines
    Note over UI,M: s follows them: journalctl -f, with a terminal

    You->>UI: R, on a container
    UI-->>You: restart container web-1 on app-1? y/N
    You->>UI: y
    UI->>M: ssh: docker restart web-1
    UI->>SW: probe again

Code

main.goflags, commands
internal/inventoryhosts file, flake
internal/remotessh: to the end, as a stream, or with the terminal; shared connections
internal/swarmthe probe and its parsing, the report, the commands for logs, descriptions and actions
internal/uithe browser: screens, tables, cards, keys; status’s table and JSON

Choices

  • Agentless. A bash script over the user’s own ssh: nothing to install, and their ssh config, keys and agent just work.
  • Read-only probe. Actions ask first, and --read-only turns them off.
  • The machine’s clock. Ages are computed with the machine’s time, not yours.
  • Bounded waits. The probe stops itself on the machine before sshwarm gives up on it, and gives df 2 s, as a hung mount would hang it.

Contributing

The following kind of contributions are welcome:

  • New Issues (bug reports, questions, ideas, …)
  • Pull Requests (documentation improvements, code improvements, new features, …)

Setup

Setup Nix and direnv for your shell:

git clone https://github.com/cmdoret/sshwarm && cd sshwarm
direnv allow   # or: nix develop

The dev shell has Go and all the tooling to work in the repo.

Tasks

We use just as a task runner:

just testtests
just lintgolangci-lint, shellcheck, nix flake check
just formatformat everything
just buildbinary and Nix package, into .output/
just updateGo dependencies, flake lock, vendor hash
just docs::servethis documentation, live
justevery recipe

The git hooks format, lint, and update the Nix vendor hash when go.mod changes: the commit then fails; stage the new hash and commit again.

Tests

  • No real ssh: testutil.FakeBin puts a fake ssh first on PATH.
  • The probe fixtures in internal/swarm/testdata come from real machines: sshwarm probe <machine>.
  • The browser’s tests type keys and read the screen: internal/ui/tui_test.go.

Code

See architecture. Style: Google’s Go style guide. Comments say why, not what.

Release

just release::bump minor   # or patch, major, x.y.z: commits the version, tags it
git push --follow-tags

Then publish a GitHub release of the tag. CI uploads the binaries and pushes the images: x.y.z, and latest unless it’s a pre-release.

AI contributions

AI contributions are welcome, as long as:

  • the code passes all checks and follows the quality standards in place.
  • They are not unnecessarily verbose or overengineered
  • The PR descriptions issues and discussions are humen written

The use of samber/cc-skills-golang is recommended.