pve-backup-watcher/README.md

# proxmox-backup-watch

Wait for a Proxmox VE backup to finish, then restart the guest.

`proxmox-backup-watch` is a small, dependency-free Go CLI that runs on a Proxmox VE node. You give it one or more VMIDs. It watches the node's task list until the vzdump backup for each guest has finished, then restarts that guest. You don't need to know when the backup job will run or how long it will take.

## Contents

- [Features](#features)
- [Requirements](#requirements)
- [Install](#install)
- [Usage](#usage)
- [Options](#options)
- [How it works](#how-it-works)
- [Exit codes](#exit-codes)
- [Examples](#examples)
- [Things to know](#things-to-know)
- [Versioning](#versioning)

## Features

- Acts only on the VMIDs you name, never on the whole node
- Works with both QEMU VMs (`qm`) and LXC containers (`pct`)
- Restarts after any result by default, or only after a failed backup with `-only-failed`
- Ignores old results, so a failure from last week never triggers a restart
- Dry-run mode to preview exactly what would happen
- Configurable poll interval and timeout
- Stops cleanly on Ctrl-C or SIGTERM
- Meaningful exit codes for use in scripts and cron
- Single static binary, standard library only

## Requirements

- A Proxmox VE node (run the tool on the node that hosts the guest)
- Root, or a user allowed to run `pvesh`, `qm` and `pct`
- `pvesh`, `qm` and `pct` on the `PATH` (standard on Proxmox VE)
- Go 1.21 or newer, only to build from source

## Install

Build from source and copy the binary to the node:

```bash
make
scp proxmox-backup-watch root@your-node:/usr/local/bin/
```

`make` builds a static `linux/amd64` binary and embeds the version, git commit and build date. If you build on the node itself, or want another architecture, run `go build` directly:

```bash
go build -o proxmox-backup-watch .
```

A plain `go build` reports the version as `dev`.

## Usage

```
proxmox-backup-watch [options] VMID [VMID...]
```

Options can appear before, between or after the VMIDs. With several VMIDs, they are handled one after another in the order given.

## Options

| Option | Default | Description |
|---|---|---|
| `-n`, `-dry-run` | off | Print the restart commands instead of running them. It still waits for the backup. |
| `-t`, `-timeout` | `6h` | Maximum time to wait per VMID. `0` waits forever. On timeout, that guest is not restarted. |
| `-i`, `-interval` | `30s` | How often to poll the task list. |
| `-f`, `-only-failed` | off | Restart only if the backup did not finish with status `OK`. |
| `-w`, `-warn-ok` | off | With `-only-failed`, treat `WARNINGS: n` as success. |
| `-node` | short hostname | Proxmox node name, if it differs from the hostname. |
| `-v`, `-version` | | Print version, commit and build date, then exit. |

Durations use Go syntax: `30s`, `15m`, `6h`.

## How it works

1. Reads the node's recent vzdump tasks with `pvesh get /nodes/<node>/tasks --typefilter vzdump`.
2. Picks the newest task whose `id` matches the VMID and that is either still running or finished after the tool was started.
3. Polls until that task has an end time, then reads its `status` (`OK`, `WARNINGS: n`, or an error message).
4. Decides whether to restart (always by default, or only on failure with `-only-failed`).
5. Restarts the guest:
   1. `qm` or `pct` `unlock`, to clear a leftover backup lock (errors are ignored)
   2. `shutdown --timeout 120 --forceStop 1`, a graceful shutdown with a forced stop after 120 seconds
   3. `start`

Temporary errors from `pvesh` or JSON parsing don't abort the wait. A warning is printed and the next poll tries again.

## Exit codes

| Code | Meaning |
|---|---|
| `0` | Every VMID was handled without error |
| `1` | At least one VMID failed: timeout, guest not found, or a failed shutdown or start |
| `2` | Invalid usage, such as no VMID given or a non-numeric VMID |

## Examples

Preview what would happen for VM 123:

```bash
proxmox-backup-watch -n 123
```

Restart VM 123 as soon as its backup finishes, whatever the result:

```bash
proxmox-backup-watch 123
```

Restart only if the backup failed, treating warnings as success:

```bash
proxmox-backup-watch -f -w 123
```

Watch several guests, checking every 15 seconds and giving up after 2 hours each:

```bash
proxmox-backup-watch -i 15s -t 2h 123 124 200
```

Run in the background and keep a log:

```bash
nohup proxmox-backup-watch 123 &> /var/log/proxmox-backup-watch-123.log &
```

## Things to know

- **Start it before or during the backup.** It only reacts to a backup that is running or starts after the tool launches. If you start it after the backup has already finished, it waits for the next one (or times out).
- **vzdump `stop` mode restarts the guest itself.** In that mode, this tool causes a second restart. It is most useful with `snapshot` or `suspend` mode, or when you want a restart after the backup regardless.
- **A restart will not fix a broken backup.** If backups fail because of storage or network problems, using `-only-failed` on a schedule reboots the guest every time without solving the cause.
- **Local node only.** It looks at tasks on the node it runs on, and the guest must be on that node.
- **Recent tasks only.** It reads the latest 500 vzdump tasks on the node, which is plenty for a backup that is running now.
- **Timeouts never restart.** If the backup doesn't finish in time, the guest is left alone and the tool exits with code `1`.

## Versioning

This project follows [Semantic Versioning](https://semver.org). The current version is in the `VERSION` file and is embedded in the binary at build time. Check it with:

```bash
proxmox-backup-watch -version
```

The tool also prints its version at the start of every run, so log files show which version produced them. See `CHANGELOG.md` for the history.

To release a change, update `VERSION`, add an entry to `CHANGELOG.md`, and rebuild with `make`.