Home-Assistant-Watchdog/README.md

# ha-watchdog

A small Go program that watches selected Home Assistant apps (formerly add-ons) and starts them again if they stop. It ships as a single static binary with no dependencies.

> **Status:** written but not yet compile-tested. Expect to fix a typo or two on the first build.

## Features

- Lists all installed apps, numbered, with state, version, available updates, and whether each is watched
- Add and remove apps from a persistent watchlist by number
- Automatically starts watched apps that are not running
- Retry cap: gives up on an app after repeated failed starts, so a broken app doesn't loop forever
- Home Assistant persistent notifications when an app is restarted or the watchdog gives up
- Skips apps that are unavailable, detached, or not installed
- Interactive menu or headless `-daemon` mode
- Runs on the Home Assistant host itself, or remotely from a separate monitoring machine over SSH
- Version and build date embedded in the binary

## Requirements

- Home Assistant OS or Supervised (anything with a Supervisor; the `ha` CLI and Supervisor API are not available in Container or Core installs)
- For remote monitoring: an SSH server on the Home Assistant host where the `ha` command works (see [Remote monitoring](#remote-monitoring)), and the `ssh` client on the monitoring machine
- To build: Go 1.21+ (or Docker)

## Build

```bash
make                    # build for this machine
make arm64              # Raspberry Pi / ARM  -> ha-watchdog-arm64
make amd64              # x86 mini PC / VM    -> ha-watchdog-amd64
make VERSION=1.2.0      # override the version
```

Not sure which architecture your Home Assistant host uses? Run `uname -m` on it: `aarch64` means arm64, `x86_64` means amd64.

Builds are static (`CGO_ENABLED=0`), so they run on HA OS and Alpine without extra libraries.

**No Go installed? Build with Docker:**

```bash
docker run --rm -v "$PWD":/src -w /src \
  -e CGO_ENABLED=0 -e GOOS=linux -e GOARCH=arm64 \
  golang:1.22 \
  go build -ldflags "-s -w -X main.buildDate=$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
  -o ha-watchdog-arm64 .
```

## Install on Home Assistant

```bash
scp ha-watchdog-arm64 [email protected]:/config/ha-watchdog
ssh [email protected] "chmod +x /config/ha-watchdog && /config/ha-watchdog -version"
```

This needs the Terminal & SSH app (or host SSH on port 22222). Keeping the binary in `/config` means it survives reboots.

## Usage

### Interactive

```bash
./ha-watchdog -file /config/watchlist.json
```

```
=== HA App Watchdog 1.1.0 | monitor: OFF | 0 watched ===
1) List installed apps
2) Add apps to watchlist
3) Remove apps from watchlist
4) Show watchlist
5) Start/stop monitoring
6) Reset retry counters / give-up state
v) Version info
q) Quit
```

Typical flow: press `2`, enter the numbers of the apps to watch (for example `1,4,7`), then press `5` to start monitoring. Monitoring only runs while the program is open in this mode.

### Headless

```bash
./ha-watchdog -daemon -file /config/watchlist.json
```

Monitors the saved watchlist until stopped with Ctrl+C or SIGTERM. Build the watchlist first using the interactive menu: if the file is missing, the daemon starts with an empty list and watches nothing. The watchlist is read once at startup, so restart the daemon after changing it.

### Flags

| Flag | Default | Description |
|------|---------|-------------|
| `-daemon` | off | Run headless, no menu |
| `-version` | | Print version and build date, then exit |
| `-interval` | `30s` | How often to check app state |
| `-cooldown` | `60s` | Minimum time between start attempts for one app |
| `-max-retries` | `5` | Give up on an app after this many start attempts within the retry window |
| `-retry-window` | `10m` | Time window for counting start attempts |
| `-file` | see below | Watchlist file path |
| `-ssh` | | Monitor a remote host over SSH, e.g. `[email protected]` |
| `-ssh-port` | `22` | SSH port on the Home Assistant host |
| `-ssh-key` | | SSH private key file to use |
| `-ha-url` | | Home Assistant URL for remote notifications, e.g. `http://homeassistant.local:8123` |
| `-ha-token` | `$HA_TOKEN` | Long-lived access token for remote notifications |

### Environment variables

| Variable | Description |
|----------|-------------|
| `SUPERVISOR_TOKEN` | If set, the Supervisor REST API is used instead of the `ha` CLI. Provided automatically inside HA apps |
| `WATCHDOG_FILE` | Watchlist path (overridden by `-file`) |
| `HA_TOKEN` | Long-lived access token for remote notifications (overridden by `-ha-token`) |

Watchlist location when `-file` and `WATCHDOG_FILE` are not set: `/data/watchlist.json` if `/data` exists (inside an HA app), otherwise `./watchlist.json`.

## How it works

Every `-interval`, the monitor lists all apps and checks each watched one:

- **Running** (`started`): nothing to do. Any give-up state is cleared.
- **Not running** (`stopped`, `error`, etc.): sends a start command, subject to the cooldown, and raises a notification.
- **Unavailable, detached, or not installed:** skipped, and logged once.
- **Too many failures:** after `-max-retries` starts within `-retry-window`, the watchdog stops retrying, logs it, and raises a notification. Use menu option 6 to reset, or it clears when the app is seen running again. A flapping app that keeps crashing still trips the cap, because the attempt history is kept when it recovers.

### Backends

| Condition | Backend | Notifications |
|-----------|---------|---------------|
| `-ssh` given | `ha` CLI on the remote host, run over SSH | Yes, if `-ha-url` and `-ha-token` are set |
| `SUPERVISOR_TOKEN` set | Supervisor REST API | Yes |
| Otherwise | Local `ha` CLI (`ha addons --raw-json`, `ha addons start`) | No, logged only |

Notifications use the `persistent_notification.create` service through the Supervisor's Core API proxy. This requires `homeassistant_api: true` in the app's `config.yaml`. If notifications don't appear, check the log for a `notify:` error line.

## Remote monitoring

The watchdog can run on a separate machine (a server, NAS, Raspberry Pi, or VM) and manage apps on your Home Assistant host over SSH. This is useful because the monitor keeps working even if Home Assistant itself is struggling. It uses the same watchlist, retry cap, and menu as local mode.

### 1. Set up SSH access on Home Assistant

Install the **Advanced SSH & Web Terminal** or **Terminal & SSH** app and add the monitoring machine's public key to its authorized keys. Then check that this works from the monitoring machine with no password prompt and prints JSON:

```bash
ssh [email protected] "ha addons --raw-json"
```

If you use a non-default port or key, test with `-p` and `-i` the same way. The watchdog connects in batch mode, so password prompts will fail rather than hang. If that command doesn't work, the watchdog won't either.

### 2. (Optional) Enable notifications

In Home Assistant, open your user profile, then Security, and create a long-lived access token. Pass it with the URL:

```bash
export HA_TOKEN="your-token"
```

Without these, remote mode still restarts apps but only logs events.

### 3. Run it

```bash
# Build the watchlist interactively
./ha-watchdog -ssh [email protected] -file ./watchlist.json

# Then run headless
./ha-watchdog -daemon -ssh [email protected] \
  -ssh-key ~/.ssh/ha_watchdog -ha-url http://homeassistant.local:8123 \
  -file ./watchlist.json
```

### Keep it running with systemd

A service has no keyboard, so everything it needs comes from the unit file, an environment file, and files on disk. Build the watchlist interactively first, because `-daemon` only reads the saved file.

**Files involved, and who creates them**

| File | Purpose | Created by |
|------|---------|------------|
| `watchlist.json` | Apps to monitor | The program, the first time you add or remove an app in the menu. The folder must already exist |
| `/etc/systemd/system/ha-watchdog.service` | The service definition | You |
| `/etc/ha-watchdog/env` | Holds `HA_TOKEN` (optional) | You |
| `/etc/ha-watchdog/id_ed25519` | SSH private key for the HA host | You |

If the watchlist file is missing when `-daemon` starts, the program does not fail or create it. It starts with an empty list and watches nothing.

**1. Install the binary and set up files**

```bash
sudo cp ha-watchdog-amd64 /usr/local/bin/ha-watchdog      # match the machine's architecture
sudo mkdir -p /var/lib/ha-watchdog /etc/ha-watchdog
sudo cp watchlist.json /var/lib/ha-watchdog/
sudo cp ~/.ssh/ha_watchdog /etc/ha-watchdog/id_ed25519    # key that can SSH into HA
sudo chmod 600 /etc/ha-watchdog/id_ed25519
```

**2. (Optional) Store the notification token**

Keep the token out of the unit file, where anyone can read it with `systemctl show`:

```bash
sudo sh -c 'echo "HA_TOKEN=your-token" > /etc/ha-watchdog/env'
sudo chmod 600 /etc/ha-watchdog/env
```

**3. Accept the SSH host key once**

The program connects in batch mode, so it cannot answer the "trust this host?" prompt. Run this once as the same user the service will run as (root by default), type `yes`, and confirm it prints output:

```bash
sudo ssh -i /etc/ha-watchdog/id_ed25519 [email protected] "ha info"
```

Skip this and every check will fail with a host key error.

**4. Create the unit file**

```ini
# /etc/systemd/system/ha-watchdog.service
[Unit]
Description=Home Assistant app watchdog
After=network-online.target
Wants=network-online.target

[Service]
EnvironmentFile=-/etc/ha-watchdog/env
ExecStart=/usr/local/bin/ha-watchdog -daemon -ssh [email protected] \
  -ssh-key /etc/ha-watchdog/id_ed25519 -ha-url http://homeassistant.local:8123 \
  -file /var/lib/ha-watchdog/watchlist.json
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
```

Where the service gets its input:

- **Command-line flags in `ExecStart`:** `-daemon` skips the menu, `-ssh` and `-ssh-key` say how to reach Home Assistant, and `-file` points at the watchlist. Add `-interval`, `-max-retries` and the other flags here if you want non-default values.
- **`EnvironmentFile`:** supplies `HA_TOKEN`. The leading `-` makes the file optional.
- **Files on disk:** the watchlist and SSH key.

Optionally add `User=hawatchdog` to `[Service]` to avoid running as root. If you do, create that user, make the key and watchlist owned by it, and do step 3 as that user.

**5. Enable and check**

```bash
sudo systemctl daemon-reload
sudo systemctl enable --now ha-watchdog
systemctl status ha-watchdog
journalctl -u ha-watchdog -f
```

After changing the watchlist, run `sudo systemctl restart ha-watchdog`, since the daemon reads the file only at startup.

### Remote mode caveats

- The first SSH connection must be accepted by hand as the service user (see step 3 above), or the service cannot connect.
- If the Home Assistant host is unreachable, each check fails and is logged. The watchdog does not alert on this, since a notification can't be delivered to a host that is down.
- The SSH key gives the monitor full access to the host's shell, so use a dedicated key and keep it private.
- Build the binary for the monitoring machine's architecture, not Home Assistant's.

## Things to know

- **Apps you stop on purpose will be restarted.** Remove an app from the watchlist before stopping it deliberately.
- Home Assistant has its own per-app Watchdog toggle, which restarts apps that fail a health check. It's worth enabling alongside this, but it only covers apps that define a health check.
- An app stuck in `error` for a persistent reason (bad config, missing hardware) won't be fixed by restarts. Check `ha addons logs <slug>`.
- The watchdog only ever starts apps. It never stops, updates, or changes them.

## Running permanently

Running `-daemon` over SSH works for testing but stops when the session ends. To run it on the monitoring machine, use the systemd unit above. To run it on Home Assistant itself, package the binary as a custom Home Assistant app (`config.yaml` with `hassio_api: true` and `homeassistant_api: true`, plus a small Dockerfile) so the Supervisor starts it at boot. Inside an app, the watchlist is stored in `/data` automatically.

## Project layout

```
ha-watchdog/
├── main.go      # the whole program
├── go.mod
├── Makefile     # build targets, injects version and build date
└── README.md
```

## Versioning

The version defaults to the value in `main.go` and is overridden at build time with `-X main.version=...`. The build date is injected with `-X main.buildDate=...`; a plain `go build` falls back to the git commit time if available, otherwise shows `unknown`.