Skip to content

Commit 3ead2a1

Browse files
shyimshyim
authored
Merge pull request #14 from xterr/feature/clickhouse-database
feature: add clickhouse database support
2 parents 4202d8d + 64548c8 commit 3ead2a1

10 files changed

Lines changed: 1110 additions & 6 deletions

File tree

‎CLAUDE.md‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,7 @@ npm run watch:css
4141
- `api/` - Unix socket API server for backup triggers
4242
- `backup/` - Backup type interface, registry, and orchestration manager
4343
- `backuptypes/` - Backup type implementations
44+
- `clickhouse/` - ClickHouse backup using native BACKUP/RESTORE SQL (requires ClickHouse 22.8+)
4445
- `mysql/` - MySQL/MariaDB backup using mysqldump
4546
- `postgres/` - PostgreSQL backup using pg_dump
4647
- `volume/` - Volume backup for container mount points
@@ -283,6 +284,31 @@ Each named config requires a unique name (e.g., "db", "files") and supports:
283284

284285
The `notify` label at container level applies to all backup configs unless overridden per-config. Notifications are opt-in - if not specified, no notifications will be sent.
285286

287+
#### ClickHouse Example
288+
289+
```yaml
290+
services:
291+
clickhouse:
292+
image: clickhouse/clickhouse-server:latest
293+
environment:
294+
CLICKHOUSE_USER: admin
295+
CLICKHOUSE_PASSWORD: secret
296+
CLICKHOUSE_DB: analytics
297+
labels:
298+
- docker-backup.enable=true
299+
- docker-backup.db.type=clickhouse
300+
- docker-backup.db.schedule=0 3 * * *
301+
- docker-backup.db.retention=7
302+
- docker-backup.db.storage=s3
303+
```
304+
305+
**ClickHouse requirements:**
306+
- ClickHouse 22.8+ (for native `BACKUP`/`RESTORE` SQL support)
307+
- `clickhouse-client` must be available in the container (included in official images)
308+
- Supported env vars: `CLICKHOUSE_USER`, `CLICKHOUSE_PASSWORD`, `CLICKHOUSE_DB`
309+
- If no env vars are set, defaults to user `default` with empty password
310+
- If `CLICKHOUSE_DB` is not set, all user databases are backed up automatically
311+
286312
## CLI Commands
287313

288314
The daemon exposes a Unix socket API at `/var/run/docker-backup.sock` (configurable via `--socket`).

‎README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -93,6 +93,7 @@ labels:
9393
9494
| Type | Description |
9595
|------|-------------|
96+
| `clickhouse` | ClickHouse database backup using native `BACKUP`/`RESTORE` SQL (requires ClickHouse 22.8+) |
9697
| `postgres` | PostgreSQL database backup using `pg_dump` |
9798
| `mysql` | MySQL/MariaDB database backup using `mysqldump` |
9899
| `volume` | Backup all mounted volumes as compressed tarball |

‎docs/backup-types/clickhouse.md‎

Lines changed: 227 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,227 @@
1+
---
2+
icon: simple/clickhouse
3+
---
4+
5+
# ClickHouse Backup
6+
7+
The `clickhouse` backup type creates backups of ClickHouse databases using the native `BACKUP`/`RESTORE` SQL commands available in ClickHouse 22.8+.
8+
9+
## Overview
10+
11+
- **Backup Method**: Native `BACKUP DATABASE` SQL via `clickhouse-client`
12+
- **Compression**: zstd compression
13+
- **Output Format**: `.tar.zst` containing the ClickHouse backup directory
14+
- **Restore Method**: Native `RESTORE` SQL via `clickhouse-client`
15+
- **Compatibility**: ClickHouse 22.8+
16+
17+
## Configuration
18+
19+
```yaml
20+
labels:
21+
- docker-backup.enable=true
22+
- docker-backup.db.type=clickhouse
23+
- docker-backup.db.schedule=0 3 * * *
24+
- docker-backup.db.retention=7
25+
```
26+
27+
## Requirements
28+
29+
### Environment Variables
30+
31+
| Variable | Required | Default | Description |
32+
|----------|----------|---------|-------------|
33+
| `CLICKHOUSE_USER` | No | `default` | ClickHouse username |
34+
| `CLICKHOUSE_PASSWORD` | No | *(empty)* | ClickHouse password |
35+
| `CLICKHOUSE_DB` | No | *(all user DBs)* | Specific database to back up |
36+
37+
ClickHouse works with default credentials out of the box, so no environment variables are strictly required.
38+
39+
### Container Requirements
40+
41+
- ClickHouse server version **22.8 or later** (for native `BACKUP`/`RESTORE` support)
42+
- `clickhouse-client` must be available in the container (included in official `clickhouse/clickhouse-server` images)
43+
44+
## How It Works
45+
46+
### Backup Process
47+
48+
1. **Version Check**: Verifies `clickhouse-client` exists and ClickHouse version is >= 22.8
49+
2. **Configure Path**: Writes a config snippet to allow `/tmp/docker-backup/` as a backup path
50+
3. **Discover Databases**: If `CLICKHOUSE_DB` is set, backs up that database only. Otherwise, queries `system.databases` and backs up all user databases (excluding `system`, `INFORMATION_SCHEMA`, `information_schema`, `default`)
51+
4. **Execute Backup**: Runs `BACKUP DATABASE ... TO File('/tmp/docker-backup/<uuid>/')` inside the container
52+
5. **Stream Out**: Tars the backup directory and streams it through zstd compression to storage
53+
6. **Cleanup**: Removes temporary backup files from the container
54+
55+
### Backup Contents
56+
57+
The backup file (`.tar.zst`) contains a ClickHouse native backup directory:
58+
59+
```
60+
backup.tar.zst
61+
└── <uuid>/
62+
├── .backup # Backup metadata
63+
└── data/ # Table data parts and schema
64+
```
65+
66+
### Restore Process
67+
68+
1. **Decompress**: Reads the zstd-compressed tar archive
69+
2. **Extract**: Pipes the tar stream into the container, recreating the backup directory
70+
3. **Execute Restore**: Runs `RESTORE ALL FROM File('/tmp/docker-backup/<uuid>/') SETTINGS allow_non_empty_tables=true`
71+
4. **Cleanup**: Removes temporary files from the container
72+
73+
## Example Configurations
74+
75+
### Basic Setup
76+
77+
```yaml
78+
services:
79+
clickhouse:
80+
image: clickhouse/clickhouse-server:latest
81+
environment:
82+
CLICKHOUSE_USER: admin
83+
CLICKHOUSE_PASSWORD: secret
84+
CLICKHOUSE_DB: analytics
85+
labels:
86+
- docker-backup.enable=true
87+
- docker-backup.db.type=clickhouse
88+
- docker-backup.db.schedule=0 3 * * *
89+
- docker-backup.db.retention=7
90+
```
91+
92+
### Default Credentials
93+
94+
ClickHouse uses `default` user with no password by default. No env vars needed:
95+
96+
```yaml
97+
services:
98+
clickhouse:
99+
image: clickhouse/clickhouse-server:latest
100+
labels:
101+
- docker-backup.enable=true
102+
- docker-backup.db.type=clickhouse
103+
- docker-backup.db.schedule=0 3 * * *
104+
- docker-backup.db.retention=7
105+
```
106+
107+
### Multiple Backup Schedules
108+
109+
```yaml
110+
services:
111+
clickhouse:
112+
image: clickhouse/clickhouse-server:latest
113+
environment:
114+
CLICKHOUSE_USER: admin
115+
CLICKHOUSE_PASSWORD: secret
116+
CLICKHOUSE_DB: analytics
117+
labels:
118+
- docker-backup.enable=true
119+
- docker-backup.notify=telegram
120+
121+
# Hourly backups (short retention)
122+
- docker-backup.hourly.type=clickhouse
123+
- docker-backup.hourly.schedule=0 * * * *
124+
- docker-backup.hourly.retention=24
125+
- docker-backup.hourly.storage=local-fast
126+
127+
# Daily backups (long retention)
128+
- docker-backup.daily.type=clickhouse
129+
- docker-backup.daily.schedule=0 2 * * *
130+
- docker-backup.daily.retention=30
131+
- docker-backup.daily.storage=s3
132+
```
133+
134+
### With S3 Storage
135+
136+
```yaml
137+
services:
138+
docker-backup:
139+
image: ghcr.io/shyim/docker-backup:latest
140+
command:
141+
- daemon
142+
- --storage=s3.type=s3
143+
- --storage=s3.bucket=my-backups
144+
- --storage=s3.region=us-east-1
145+
- --storage=s3.access-key=AKIA...
146+
- --storage=s3.secret-key=secret
147+
- --default-storage=s3
148+
volumes:
149+
- /var/run/docker.sock:/var/run/docker.sock:ro
150+
151+
clickhouse:
152+
image: clickhouse/clickhouse-server:latest
153+
environment:
154+
CLICKHOUSE_USER: admin
155+
CLICKHOUSE_PASSWORD: secret
156+
CLICKHOUSE_DB: analytics
157+
labels:
158+
- docker-backup.enable=true
159+
- docker-backup.db.type=clickhouse
160+
- docker-backup.db.schedule=0 3 * * *
161+
- docker-backup.db.retention=7
162+
- docker-backup.db.storage=s3
163+
```
164+
165+
## Manual Operations
166+
167+
### Trigger Backup
168+
169+
```bash
170+
docker-backup backup run clickhouse
171+
```
172+
173+
### List Backups
174+
175+
```bash
176+
docker-backup backup list clickhouse
177+
```
178+
179+
Output:
180+
```
181+
KEY SIZE DATE
182+
clickhouse/db/2024-01-15/030000.tar.zst 2.1 MB 2024-01-15 03:00:00
183+
clickhouse/db/2024-01-14/030000.tar.zst 2.0 MB 2024-01-14 03:00:00
184+
```
185+
186+
### Restore Backup
187+
188+
```bash
189+
docker-backup backup restore clickhouse "clickhouse/db/2024-01-15/030000.tar.zst"
190+
```
191+
192+
!!! warning "Restore Behavior"
193+
Restoring will:
194+
195+
- Recreate databases and tables included in the backup
196+
- Use `allow_non_empty_tables=true`, which may mix existing data with restored data
197+
- Not affect databases not included in the backup
198+
199+
## Troubleshooting
200+
201+
### "clickhouse-client not found" Error
202+
203+
Ensure you're using an official ClickHouse Docker image (`clickhouse/clickhouse-server`), which includes the client binary. Minimal or custom images may not have it.
204+
205+
### "version X.Y is too old" Error
206+
207+
Native `BACKUP`/`RESTORE` requires ClickHouse 22.8 or later. Upgrade your ClickHouse image:
208+
209+
```yaml
210+
image: clickhouse/clickhouse-server:latest
211+
```
212+
213+
### "Path is not allowed for backups" Error
214+
215+
docker-backup automatically configures the backup path on first run. If you see this error, it may indicate the config reload failed. Verify the ClickHouse server can write to `/tmp/docker-backup/`:
216+
217+
```bash
218+
docker exec clickhouse ls -la /etc/clickhouse-server/config.d/
219+
```
220+
221+
### Large Databases
222+
223+
For very large ClickHouse databases, consider:
224+
225+
1. Setting `CLICKHOUSE_DB` to back up a specific database instead of all databases
226+
2. Backing up during low-traffic periods to minimize I/O impact
227+
3. Ensuring sufficient temporary disk space inside the container (backup is staged at `/tmp/docker-backup/` before streaming)

‎docs/backup-types/index.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ docker-backup supports multiple backup types, each designed for specific applica
1010

1111
| Type | Description | Output |
1212
|------|-------------|--------|
13+
| `clickhouse` | ClickHouse database backup (22.8+) | `.tar.zst` |
1314
| `postgres` | PostgreSQL database backup | `.tar.zst` |
1415
| `mysql` | MySQL/MariaDB database backup | `.tar.zst` |
1516
| `volume` | Docker volume backup | `.tar.zst` |
@@ -29,6 +30,7 @@ Select the backup type that matches your application:
2930

3031
| Application | Backup Type |
3132
|-------------|-------------|
33+
| ClickHouse | `clickhouse` |
3234
| PostgreSQL | `postgres` |
3335
| MySQL | `mysql` |
3436
| MariaDB | `mysql` |
@@ -38,6 +40,14 @@ Select the backup type that matches your application:
3840

3941
<div class="grid cards" markdown>
4042

43+
- :simple-clickhouse: **ClickHouse**
44+
45+
---
46+
47+
Backup ClickHouse databases using native `BACKUP`/`RESTORE` SQL
48+
49+
[:octicons-arrow-right-24: ClickHouse](clickhouse.md)
50+
4151
- :simple-postgresql: **PostgreSQL**
4252

4353
---

‎docs/configuration/container-labels.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -50,7 +50,7 @@ These labels define individual backup configurations. Replace `<name>` with your
5050

5151
| Label | Required | Default | Description |
5252
|-------|----------|---------|-------------|
53-
| `docker-backup.<name>.type` | Yes | - | Backup type (e.g., `postgres`, `mysql`) |
53+
| `docker-backup.<name>.type` | Yes | - | Backup type (`clickhouse`, `postgres`, `mysql`, `volume`) |
5454
| `docker-backup.<name>.schedule` | Yes | - | Cron expression for scheduling |
5555
| `docker-backup.<name>.retention` | No | `7` | Number of backups to keep |
5656
| `docker-backup.<name>.storage` | No | Default pool | Storage pool name |

‎go.mod‎

Lines changed: 16 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,10 @@ require (
3232
dario.cat/mergo v1.0.2 // indirect
3333
filippo.io/edwards25519 v1.1.1 // indirect
3434
github.com/Azure/go-ansiterm v0.0.0-20250102033503-faa5f7b0171c // indirect
35+
github.com/ClickHouse/ch-go v0.71.0 // indirect
36+
github.com/ClickHouse/clickhouse-go/v2 v2.45.0 // indirect
3537
github.com/Microsoft/go-winio v0.6.2 // indirect
38+
github.com/andybalholm/brotli v1.2.0 // indirect
3639
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.8 // indirect
3740
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.21 // indirect
3841
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.21 // indirect
@@ -67,10 +70,12 @@ require (
6770
github.com/felixge/httpsnoop v1.0.4 // indirect
6871
github.com/gabriel-vasile/mimetype v1.4.13 // indirect
6972
github.com/gin-contrib/sse v1.1.1 // indirect
73+
github.com/go-faster/city v1.0.1 // indirect
74+
github.com/go-faster/errors v0.7.1 // indirect
7075
github.com/go-jose/go-jose/v4 v4.1.4 // indirect
7176
github.com/go-logr/logr v1.4.3 // indirect
7277
github.com/go-logr/stdr v1.2.2 // indirect
73-
github.com/go-ole/go-ole v1.2.6 // indirect
78+
github.com/go-ole/go-ole v1.3.0 // indirect
7479
github.com/go-playground/locales v0.14.1 // indirect
7580
github.com/go-playground/universal-translator v0.18.1 // indirect
7681
github.com/go-playground/validator/v10 v10.30.1 // indirect
@@ -87,7 +92,7 @@ require (
8792
github.com/json-iterator/go v1.1.12 // indirect
8893
github.com/klauspost/cpuid/v2 v2.3.0 // indirect
8994
github.com/leodido/go-urn v1.4.0 // indirect
90-
github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0 // indirect
95+
github.com/lufia/plan9stats v0.0.0-20250317134145-8bc96cf8fc35 // indirect
9196
github.com/magiconair/properties v1.8.10 // indirect
9297
github.com/mattn/go-isatty v0.0.20 // indirect
9398
github.com/moby/docker-image-spec v1.3.1 // indirect
@@ -102,17 +107,22 @@ require (
102107
github.com/morikuni/aec v1.1.0 // indirect
103108
github.com/opencontainers/go-digest v1.0.0 // indirect
104109
github.com/opencontainers/image-spec v1.1.1 // indirect
110+
github.com/paulmach/orb v0.12.0 // indirect
105111
github.com/pelletier/go-toml/v2 v2.3.0 // indirect
112+
github.com/pierrec/lz4/v4 v4.1.25 // indirect
106113
github.com/pkg/errors v0.9.1 // indirect
107114
github.com/pmezard/go-difflib v1.0.0 // indirect
108-
github.com/power-devops/perfstat v0.0.0-20210106213030-5aafc221ea8c // indirect
115+
github.com/power-devops/perfstat v0.0.0-20240221224432-82ca36839d55 // indirect
109116
github.com/quic-go/qpack v0.6.0 // indirect
110117
github.com/quic-go/quic-go v0.59.0 // indirect
118+
github.com/segmentio/asm v1.2.1 // indirect
111119
github.com/shirou/gopsutil/v4 v4.25.6 // indirect
120+
github.com/shopspring/decimal v1.4.0 // indirect
112121
github.com/sirupsen/logrus v1.9.3 // indirect
113122
github.com/spf13/pflag v1.0.10 // indirect
114-
github.com/tklauser/go-sysconf v0.3.12 // indirect
115-
github.com/tklauser/numcpus v0.6.1 // indirect
123+
github.com/testcontainers/testcontainers-go/modules/clickhouse v0.40.0 // indirect
124+
github.com/tklauser/go-sysconf v0.3.15 // indirect
125+
github.com/tklauser/numcpus v0.10.0 // indirect
116126
github.com/twitchyliquid64/golang-asm v0.15.1 // indirect
117127
github.com/ugorji/go/codec v1.3.1 // indirect
118128
github.com/yusufpapurcu/wmi v1.2.4 // indirect
@@ -125,6 +135,7 @@ require (
125135
go.opentelemetry.io/otel/sdk v1.43.0 // indirect
126136
go.opentelemetry.io/otel/sdk/metric v1.43.0 // indirect
127137
go.opentelemetry.io/otel/trace v1.43.0 // indirect
138+
go.yaml.in/yaml/v3 v3.0.4 // indirect
128139
golang.org/x/arch v0.25.0 // indirect
129140
golang.org/x/net v0.52.0 // indirect
130141
golang.org/x/sync v0.20.0 // indirect

0 commit comments

Comments
 (0)