k9s for ssh. Your Linux machines, their services, containers and logs, in one terminal: what's deployed where, what died overnight, what's crash-looping, and the logs of whatever is red a keypress away.
Quickstart
Install
| From | |
|---|---|
| Go | go install github.com/cmdoret/sshwarm@latest |
| Nix | nix run github:cmdoret/sshwarm |
| Binary | releases: Linux and macOS, amd64 and arm64 |
| Container | see below |
List the machines
In ~/.config/sshwarm/hosts.toml, one [[host]] per machine:
[[host]]
name = "db" # ssh db: the rest from ~/.ssh/config
[[host]]
name = "web-1"
host = "web-1.example.org" # ssh -i ~/.ssh/web deploy@web-1.example.org
user = "deploy"
key = "~/.ssh/web"
Using deploy-rs? Skip the file: sshwarm --flake ./nix. See configuration.
Run
sshwarm # browse
sshwarm status # print the table, exit 1 on any problem
sshwarm status --format json # for scripts
The machines need systemd, bash, and ssh without a password prompt. Logs and containers need more: see requirements.
Keys
| Key | |
|---|---|
enter | open |
l, d | logs, describe |
/ | filter; in logs, search |
:ctr web | containers of web; :ctr all, of every machine. tab completes |
? | every key |
Container
docker run --rm -it \
-v ./hosts.toml:/hosts.toml:ro \
-v ~/.ssh/id_ed25519:/root/.ssh/id_ed25519:ro \
-v ~/.ssh/known_hosts:/root/.ssh/known_hosts:ro \
ghcr.io/cmdoret/sshwarm --hosts /hosts.toml
No nix inside, so no --flake. ssh refuses a mounted ~/.ssh/config: the hosts
file must set host, user and key.
Configuration
No config file: the machines, flags, and environment variables.
Machines
sshwarm takes the first of:
--hosts <file>or--flake <dir>SSHWARM_HOSTSorSSHWARM_FLAKE$XDG_CONFIG_HOME/sshwarm/hosts.toml, by default~/.config/sshwarm/hosts.toml
A hosts file and a flake together are an error.
Hosts file
[[host]]
name = "web-1" # required: sshwarm's name for it
host = "web-1.example.org" # default: name
user = "deploy" # default: ~/.ssh/config's, else yours
key = "~/.ssh/web" # default: ~/.ssh/config's, or your agent
ssh looks host (or name) up in ~/.ssh/config for anything else: port, jump
host… Unknown keys are an error.
Nix flake
sshwarm --flake ./nix
Reads the flake’s deploy.nodes, as deploy-rs
does: each node’s hostname, and its sshUser and sshOpts, or deploy’s. Needs nix.
Flags
| Flag | Variable | Default | |
|---|---|---|---|
--hosts <file> | SSHWARM_HOSTS | hosts file | |
--flake <dir> | SSHWARM_FLAKE | flake with deploy.nodes | |
--timeout <duration> | SSHWARM_TIMEOUT | 20s | probe timeout, per machine |
--refresh <duration> | SSHWARM_REFRESH | 30s | how often the browser probes |
--read-only | SSHWARM_READ_ONLY | off | no actions, no shells |
A flag beats its variable. SSHWARM_READ_ONLY is on unless empty, 0 or false.
Commands: status [--format text|json] [machine...], and probe <machine>, which
prints a machine’s raw probe output. sshwarm <command> --help for more.
Requirements
Each machine runs Linux with systemd and bash. ssh must log in without a prompt: a key or an agent, and a known host key.
| To | the ssh user |
|---|---|
| see the machine and its units | needs nothing more |
| read logs | is root, or in systemd-journal |
| see, start and stop compose stacks and containers | runs docker: docker group, rootless, or passwordless sudo |
| start and stop units, reboot | is root, or runs systemctl with passwordless sudo |
Pending reboots are detected on NixOS, Debian, Ubuntu, Fedora, RHEL and Arch. On
NixOS, sshwarm also shows the last deploy and its git revision, if the flake sets
system.configurationRevision.
Connections
The browser keeps one ssh connection per machine (ControlMaster), its socket in
~/.ssh, closed a minute after you quit. It leaves ssh alone if your ssh config
sets ControlPath or ControlMaster for the machine, or ~/.ssh isn’t writable.
Architecture
flowchart LR
src["hosts.toml<br>or flake"] --> sw[sshwarm]
sw -- "ssh, every --refresh:<br>probe.sh on stdin" --> m["each machine"]
m -- "### sections:<br>OS, usage, units, stacks" --> sw
sw --> ui["browser,<br>or status"]
ui -. "logs, describe,<br>shell, actions" .-> m
- sshwarm reads the machines from a hosts file or a flake.
- It pipes
probe.shto every machine over ssh, in parallel, and waits for all. The browser does it again--refreshafter the last round ended. - The probe only reads: OS, uptime, usage, pending reboot,
systemctl showof every service,docker psanddocker inspectof compose containers. It prints### <name>sections; a command that fails prints### error <name>. - sshwarm parses each machine’s output into a report, and rates what’s in it: idle,
fine, warning or broken. A machine’s problems are what’s broken, what restarted on
its own in the last 10 minutes, a disk 90% full, and the probe’s errors.
statusexits 1 on any. - Logs, descriptions, shells and actions each run their own ssh command, on demand.
In time
A refresh, a unit’s logs, then a container’s restart:
sequenceDiagram
actor You
participant UI as browser
participant SW as swarm
participant M as each machine
loop every --refresh, from the end of the last round
UI->>SW: probe
SW->>M: ssh: bash, probe.sh on stdin, to all at once
M-->>SW: OS, usage, units, containers, in sections
SW-->>UI: a report per machine, rated
end
You->>UI: l, on a unit
UI->>M: ssh: journalctl -u unit -n 300
M-->>UI: its last lines
Note over UI,M: s follows them: journalctl -f, with a terminal
You->>UI: R, on a container
UI-->>You: restart container web-1 on app-1? y/N
You->>UI: y
UI->>M: ssh: docker restart web-1
UI->>SW: probe again
Code
main.go | flags, commands |
internal/inventory | hosts file, flake |
internal/remote | ssh: to the end, as a stream, or with the terminal; shared connections |
internal/swarm | the probe and its parsing, the report, the commands for logs, descriptions and actions |
internal/ui | the browser: screens, tables, cards, keys; status’s table and JSON |
Choices
- Agentless. A bash script over the user’s own ssh: nothing to install, and their ssh config, keys and agent just work.
- Read-only probe. Actions ask first, and
--read-onlyturns them off. - The machine’s clock. Ages are computed with the machine’s time, not yours.
- Bounded waits. The probe stops itself on the machine before sshwarm gives up
on it, and gives
df2 s, as a hung mount would hang it.
Contributing
The following kind of contributions are welcome:
- New Issues (bug reports, questions, ideas, …)
- Pull Requests (documentation improvements, code improvements, new features, …)
Setup
Setup Nix and direnv for your shell:
git clone https://github.com/cmdoret/sshwarm && cd sshwarm
direnv allow # or: nix develop
The dev shell has Go and all the tooling to work in the repo.
Tasks
We use just as a task runner:
just test | tests |
just lint | golangci-lint, shellcheck, nix flake check |
just format | format everything |
just build | binary and Nix package, into .output/ |
just update | Go dependencies, flake lock, vendor hash |
just docs::serve | this documentation, live |
just | every recipe |
The git hooks format, lint, and update the Nix vendor hash when go.mod changes:
the commit then fails; stage the new hash and commit again.
Tests
- No real ssh:
testutil.FakeBinputs a fakesshfirst onPATH. - The probe fixtures in
internal/swarm/testdatacome from real machines:sshwarm probe <machine>. - The browser’s tests type keys and read the screen:
internal/ui/tui_test.go.
Code
See architecture. Style: Google’s Go style guide. Comments say why, not what.
Release
just release::bump minor # or patch, major, x.y.z: commits the version, tags it
git push --follow-tags
Then publish a GitHub release of the tag. CI uploads the binaries and pushes the
images: x.y.z, and latest unless it’s a pre-release.
AI contributions
AI contributions are welcome, as long as:
- the code passes all checks and follows the quality standards in place.
- They are not unnecessarily verbose or overengineered
- The PR descriptions issues and discussions are humen written
The use of samber/cc-skills-golang is recommended.