- A public, richly browseable 3D star map + worldbuilding layer grounded in real astronomy. It prioritizes fun exploration and factual, engaging descriptions while keeping core science data immutable and provenance‑clean.
- Gaia-first canonical datasets: systems, stars, planets (Gaia DR3 backbone + NASA Exoplanet Archive), with multiplicity and science side catalogs.
- AT-HYG is transitional crosswalk support for naming/identifier recovery, not canonical inventory authority.
- Optional packs and side catalogs (v1.2+): compact/remnant, superstellar, eclipsing, and lifecycle support artifacts.
- Disc layer derivatives: expositions, reference links, snapshots, scores.
- A browser 3D map (v2) with filters and overlays.
- data/raw/ immutable upstream artifacts.
- data/cooked/ normalized, catalog‑shaped (no joins).
- data/served/ queryable outputs; data/served/current points to the promoted data/out/<build_id>/.
- Strong provenance required in all rows.
- Default state lives under
./dataso it can be mounted as a single volume in containers.
- Core artifacts: DuckDB + Parquet, sorted by Morton Z‑order spatial_index.
- Stable object keys for systems/stars/planets; strict provenance fields required 100%.
- Planet → host matching prioritized by Gaia DR3 ID, then HIP, HD, then hostname.
- Display naming precedence: common/human names first, then survey/mission host labels (TRAPPIST/Kepler/TOI/WASP family), Gaia ID last fallback.
- Separate databases for layer boundaries:
- core: immutable served canonical astronomy inventory/projection
- Gaia-first systems, stars, and planets
- accepted object identity, accepted host/membership links, and selected hot-path scalar facts
- deterministic public slices tuned for interactive search, detail, and map traffic
- AT-HYG may contribute transitional alias/crosswalk enrichment, but not canonical inventory rows
- arm: immutable science evidence/support
- source-native support rows outside core hot paths
- multiplicity hierarchy and orbit graph evidence from NSS/WDS/ORB6/MSC/SBX and related catalogs
- normalized orbital solutions, derived science candidates, diagnostics, and future adjudication proposals
- confidence, provenance, source lineage, and transform lineage for support rows
- disc: reproducible presentation and enrichment derivatives
- coolness scoring, snapshots, factsheets, expositions, source links, and generated media metadata
- clearly labeled simulation/presentation assumptions that never become science claims by themselves
- regenerated from core/arm plus explicit generator versions
- rim: editable fiction and worldbuilding overlays
- namespace-scoped user/project entities and relationships
- optional map overlays such as gates, routes, factions, stations, and cultural references
- references canonical science by stable object key without mutating core, arm, or disc
- core: immutable served canonical astronomy inventory/projection
- Retired layer names:
- galaxy and halo are no longer active served/Admin layers in the current architecture. Older helper scripts and metadata may still reference them during migration or experiments, but the authoritative layer contract is core/arm/disc/rim.
- Pack schema requires stable_object_key, object type, coordinates, and full provenance.
- Discovered via packs_manifest.json.
Spacegate uses environment variables to locate persistent data and runtime state.
- export SPACEGATE_STATE_DIR=./data
- export SPACEGATE_CACHE_DIR=./data/cache
- export SPACEGATE_LOG_DIR=./data/logs
These directories are ignored by git, but scientific state should only be
removed through the retention process. A prebuilt public runtime is much
smaller than a full source build. The July 2026 Evidence Lake requires more
than 1 TB of working capacity; use at least 2 TB of usable fast local storage
for a practical full E0-E7 build. See
docs/BUILD_STORAGE_REQUIREMENTS.mdfor measured artifacts, peak scratch, and capacity tiers. PlaceSPACEGATE_STATE_DIRon a separate volume from the root for source builds.
- /srv/spacegate # web and api servers
- /var/lib/spacegate
- /var/cache/spacegate
- /var/log/spacegate
- /etc/spacegate
On server hosts, keep /etc/spacegate owned by root:spacegate with mode
2750. The setgid bit makes files replaced inside the directory inherit the
spacegate group, which keeps /etc/spacegate/spacegate.env readable by
Spacegate host-side tooling after root editor saves. Keep spacegate.env
root:spacegate with mode 0640.
On Photon, source /srv/spacegate/photon.env before host-side Spacegate tasks.
The active development/build state is under /data/spacegate/state on the
/data volume; bulk research documents, papers, OCR text, and other large
dossier cache material should live under /mnt/space/spacegate when that mount
is available, with hashes and metadata kept in Spacegate state.
Running Spacegate (API + web UI) from scratch.
You need Python, Node (18+), and basic download tools available on your PATH:
python3+pipnode(v18+) +npmgit,curl,aria2c,gzip,7z
On Debian/Ubuntu, install base tools first, then install Node.js 20:
sudo apt-get update
sudo apt-get install -y python3 python3-venv python3-pip git curl aria2 gzip p7zip-full ca-certificates gnupg
curl -fsSL https://deb.nodesource.com/setup_20.x | sudo -E bash -
sudo apt-get install -y nodejsThe installer creates virtualenvs, installs Python/web dependencies, and if needed bootstraps the current prebuilt core DB from https://spacegates.org/dl/current.json (with fallback to local source build). Override with SPACEGATE_PUBLIC_BASE_URL or SPACEGATE_BOOTSTRAP_META_URL if your public host differs.
git clone https://github.com/galenmatson/spacegate.git
cd spacegate
./install_spacegate.shOptional flags:
--overwritere-downloads installer inputs even if present.--skip-webskips the web UI dependency install.--skip-buildskips the data build step.--skip-db-downloadskips prebuilt DB bootstrap and builds from catalogs instead.
If you used --skip-build, used --skip-db-download, or want to rebuild:
scripts/build_database.shBefore a forced full catalog refresh, run preflight:
scripts/preflight_full_refresh.shThen run a true full refresh (Gaia delta_mode=refresh + support catalog overwrite):
scripts/build_database.sh --full-refreshFor routine update runs with automatic differential/full routing, use:
scripts/refresh_core.shFor multiplicity contribution analysis with MSC fixed on (nss_off, nss_on, optional nss_on_wds_xmatch):
scripts/run_multiplicity_modes.shOptional experimental WDS->Gaia crosswalk (for WDS-linked grouping support alongside MSC):
SPACEGATE_ENABLE_WDS_GAIA_XMATCH=1 scripts/download_core.sh --non-interactive
SPACEGATE_ENABLE_WDS_GAIA_XMATCH=1 scripts/cook_core.sh
SPACEGATE_ENABLE_WDS_GAIA_XMATCH=1 scripts/ingest_core.shWhen enabled, ingest applies physical-consistency gates to multi-member WDS groups before using them for system grouping.
To fetch the currently published prebuilt DB manually:
scripts/bootstrap_core_db.shFor server deploys, pass the target state dir inline so files land where Docker mounts them:
SPACEGATE_STATE_DIR=/srv/spacegate/data \
SPACEGATE_CACHE_DIR=/srv/spacegate/data/cache \
scripts/bootstrap_core_db.shTo package the currently promoted build and update download metadata:
scripts/publish_db.shBy default this writes to SPACEGATE_DL_ROOT (auto-detected as /data/spacegate/dl when /data/spacegate exists, otherwise /srv/spacegate/dl):
- archive:
db/<build_id>.7z(or.tar.zstif7zis unavailable) - symlink:
current -> db/<archive> - metadata:
current.json - reports:
reports/<build_id>/{qc_report,match_report,identifier_report,alias_report,provenance_report,system_grouping_report,core_manifest}.jsonwhen present
current.json includes artifact checksum/size plus report links and summary metadata used by bootstrap clients.
To mirror catalog artifacts (raw + cooked) into $SPACEGATE_DL_ROOT/catalogs:
scripts/publish_catalog_mirror.pyOutputs:
catalogs/snapshots/<snapshot_id>/raw/...(upstream raw format, unchanged)catalogs/snapshots/<snapshot_id>/cooked/...(Spacegate-normalized artifacts)catalogs/snapshots/<snapshot_id>/index.jsoncatalogs/current -> snapshots/<snapshot_id>catalogs/current.json
Use --catalog <name> repeatedly to mirror a subset, or --raw-only to skip cooked artifacts.
To copy the published DB archive, current.json, referenced reports, and the
active catalog mirror snapshot when present, to a remote /dl tree:
scripts/push_published_db.sh --remote antiprotonThe script reads local current.json, transfers only the referenced DB/report
files, and preserves relative paths (db/..., reports/...). When
$SPACEGATE_DL_ROOT/catalogs/current.json exists locally, it also pushes:
catalogs/current.jsoncatalogs/current -> snapshots/<snapshot_id>catalogs/snapshots/<snapshot_id>/...
Real transfer runs print phase timestamps plus aggregate rsync progress/stats so large catalog mirror syncs do not look stalled.
By default it does not update the remote current symlink. If you still want that pointer on the remote host:
scripts/push_published_db.sh --remote antiproton --set-current-linkFor private deployment routes, set host-specific defaults in a local untracked env file. If your deploy key is not selected by ssh config automatically:
scripts/push_published_db.sh --remote <deploy-user>@<deploy-host> --ssh-key ~/.ssh/spacegate_deploy_keyIf your remote host keeps ufw limit enabled for SSH, add a short gap between
new connections:
scripts/push_published_db.sh --remote <deploy-user>@<deploy-host> --ssh-key ~/.ssh/spacegate_deploy_key --ssh-cooldown 3Use --skip-catalogs if you want to push only DB/reports and leave the remote
catalog mirror untouched.
After a database publish or API deployment, run the benchmark-system API check against the target service:
scripts/verify_known_systems_api.py https://coolstars.org/api/v1This verifies search/detail/hierarchy/simulation-scene behavior for Castor, Nu Sco, Alpha Centauri, Sirius, TRAPPIST-1, 55 Cnc, Sol, and 16 Cyg. Preview body-count mismatches are warnings by default because the live preview is still catching up with the richer hierarchy model.
To sync the app and restart containers without overwriting remote secrets:
scripts/deploy_antiproton.shThis deploy helper excludes these remote-local env files from rsync:
.spacegate.env.spacegate.local.env
Set host-specific deploy defaults through local config rather than tracked code:
export SPACEGATE_DEPLOY_REMOTE=deploy-user@your-public-host
export SPACEGATE_DEPLOY_SSH_KEY=$HOME/.ssh/spacegate_deploy_key
export SPACEGATE_DEPLOY_PUBLIC_URL=https://your-public-host.exampleUseful options:
--no-buildrestart without image rebuild--skip-public-checkskip public URL checks--ssh-cooldown 3space out fresh SSH sessions for hosts protected byufw limit--dry-runpreview sync/restart steps without changing remote files
Real deploy runs print phase timestamps plus aggregate rsync progress/stats so large app syncs do not look stalled.
The launcher verifies the database, then starts the API service:
scripts/run_spacegate.shDefaults:
- API:
http://0.0.0.0:8000
For local UI development, opt in to the Vite dev server:
scripts/run_spacegate.sh --web-devDev web default:
- Web UI:
http://0.0.0.0:5173
scripts/run_spacegate.sh --stop
scripts/run_spacegate.sh --restartscripts/spacegate_status.sh
scripts/spacegate_status.sh --watch 2scripts/ops_report.sh
scripts/ops_report.sh --public-url https://spacegates.orgscripts/spacegate_stress.sh --profile smoke --url http://192.168.1.102
scripts/spacegate_stress.sh --profile sustain --url http://192.168.1.102 --duration 600# Data locations (defaults to ./data)
export SPACEGATE_STATE_DIR=/var/lib/spacegate
export SPACEGATE_CACHE_DIR=/var/cache/spacegate
export SPACEGATE_LOG_DIR=/var/log/spacegate
# Optional alias (compose/scripts fallback to this when SPACEGATE_STATE_DIR is unset)
export SPACEGATE_DATA_DIR=/var/lib/spacegate
# DuckDB resources (otherwise auto-detected)
export SPACEGATE_DUCKDB_MEMORY_LIMIT=24GB
export SPACEGATE_DUCKDB_THREADS=4
# Multiplicity toggles
export SPACEGATE_ENABLE_GAIA_NSS=1
export SPACEGATE_ENABLE_MSC=1
export SPACEGATE_ENABLE_SBX=1
export SPACEGATE_ENABLE_PROXIMITY=0
export SPACEGATE_ENABLE_WDS_GAIA_XMATCH=0
export SPACEGATE_ENABLE_ECLIPSING_CATALOGS=1
export SPACEGATE_ENABLE_KEPLER_EB=0
export SPACEGATE_ENABLE_ATHYG_ALIAS_CROSSWALK=0
export SPACEGATE_ENABLE_ATHYG_SUPPLEMENT_MERGE=0
# MSC transport fallback (only if CTIO TLS chain fails in your environment)
# Preferred MSC mirror override:
# export SPACEGATE_MSC_MIRROR_URL=$SPACEGATE_PUBLIC_BASE_URL/dl/catalogs/current/raw/msc/newmsc-20240101.tar.gz
# Security note: HTTP is vulnerable to in-transit tampering.
# In HTTP mode, SHA pinning is required.
export SPACEGATE_MSC_ALLOW_INSECURE_HTTP=0
# export SPACEGATE_MSC_ALLOW_INSECURE_HTTP=1
# export SPACEGATE_MSC_FORCE_HTTP=1
# export SPACEGATE_MSC_SHA256=<expected_sha256>
# Optional cooker safety limits for MSC tar processing
# export SPACEGATE_MSC_MAX_ARCHIVE_BYTES=134217728
# export SPACEGATE_MSC_MAX_MEMBER_BYTES=67108864
# Auto-score coolness after promote_build.sh (default on)
export SPACEGATE_AUTO_SCORE_COOLNESS=1
# Multiplicity golden exam is on by default in verify_build.sh.
# Set to 0 only when intentionally bypassing this gate.
# export SPACEGATE_VERIFY_MULTIPLICITY_GOLDENS=0
# Gaia NSS fetch tuning (download_core.sh)
export SPACEGATE_GAIA_NSS_BUCKETS=53
export SPACEGATE_GAIA_NSS_TIMEOUT_S=240
export SPACEGATE_GAIA_NSS_RETRIES=4
# Gaia differential fetch controls (download_core.sh)
# resume: reuse local bucket parts whenever present (fastest, default behavior)
# delta: refresh only stale/missing buckets
# refresh: force full refetch of all buckets
export SPACEGATE_GAIA_DELTA_MODE=delta
export SPACEGATE_GAIA_DELTA_MAX_AGE_HOURS=720
# Optional per-source overrides:
# export SPACEGATE_GAIA_BACKBONE_DELTA_MODE=delta
# export SPACEGATE_GAIA_CLASSPROB_DELTA_MODE=delta
# export SPACEGATE_GAIA_NSS_DELTA_MODE=delta
# export SPACEGATE_SBX_DELTA_MODE=delta
# Optional WDS->Gaia crosswalk via CDS XMatch (download_core.sh)
export SPACEGATE_WDS_GAIA_XMATCH_DIST_ARCSEC=2.0
export SPACEGATE_WDS_GAIA_XMATCH_SELECTION=best
export SPACEGATE_WDS_GAIA_XMATCH_MAX_REC=2000000
# WDS->Gaia ingest-time gating (ingest_core.py)
export SPACEGATE_WDS_GAIA_MATCH_MAX_ARCSEC=2.0
export SPACEGATE_WDS_GAIA_GATE_MAX_DIST_SPREAD_LY=10.0
export SPACEGATE_WDS_GAIA_GATE_MAX_PM_DELTA_MASYR=25.0
# Core DB bootstrap controls
export SPACEGATE_BOOTSTRAP_DB=0
export SPACEGATE_PUBLIC_BASE_URL=https://spacegates.org
export SPACEGATE_BOOTSTRAP_META_URL=https://spacegates.org/dl/current.json
# Web runtime mode for scripts/run_spacegate.sh
# 0 = API only (default), 1 = API + Vite dev server
export SPACEGATE_WEB_ENABLE=1Instead of exporting variables for every command, create a local env file in the repo root:
cat > .spacegate.env <<'EOF'
SPACEGATE_STATE_DIR=/data/spacegate/data
SPACEGATE_CACHE_DIR=/data/spacegate/data/cache
SPACEGATE_LOG_DIR=/data/spacegate/data/logs
# Optional source-build knobs
SPACEGATE_ENABLE_PROXIMITY=0
SPACEGATE_ENABLE_GAIA_BACKBONE=1
SPACEGATE_ENABLE_GAIA_CLASSPROB=1
SPACEGATE_ENABLE_GAIA_NSS=1
SPACEGATE_ENABLE_MSC=1
SPACEGATE_ENABLE_SBX=1
SPACEGATE_ENABLE_WDS_GAIA_XMATCH=0
SPACEGATE_ENABLE_ECLIPSING_CATALOGS=1
SPACEGATE_ENABLE_KEPLER_EB=0
SPACEGATE_ENABLE_ATHYG_ALIAS_CROSSWALK=0
SPACEGATE_ENABLE_ATHYG_SUPPLEMENT_MERGE=0
SPACEGATE_GAIA_BACKBONE_BUCKETS=211
SPACEGATE_GAIA_CLASSPROB_BUCKETS=211
SPACEGATE_GAIA_NSS_BUCKETS=53
SPACEGATE_GAIA_DELTA_MODE=delta
SPACEGATE_GAIA_DELTA_MAX_AGE_HOURS=720
SPACEGATE_DUCKDB_MEMORY_LIMIT=24GB
SPACEGATE_DUCKDB_THREADS=12
EOFMost scripts now auto-load these files in this precedence (lowest to highest):
/etc/spacegate/spacegate.env./.spacegate.env./.spacegate.local.envSPACEGATE_ENV_FILE(if set)
Process env always wins (inline prefixes like SPACEGATE_STATE_DIR=... scripts/... override all files).
Note: .spacegate.env and .spacegate.local.env are ignored by git.
SPACEGATE_STATE_DIR is the canonical persistent runtime/build state root.
SPACEGATE_DATA_DIR remains a compatibility alias/fallback for older
compose/script paths.
Admin v2 is served from /admin/ and uses authenticated /api/v2/admin/*
endpoints. OAuth callbacks should target
/api/v2/auth/callback/google; successful logins should redirect to /admin/.
The older /api/v2/admin/ui page remains only as an embedded fallback while
migration work continues.
The Agency source allowlist ships as config/agent_source_allowlist.json.
Admin writes operator edits to
$SPACEGATE_STATE_DIR/config/agent_source_allowlist.json and snapshots previous
runtime versions under
$SPACEGATE_STATE_DIR/config/agent_source_allowlist.history/.
For release deployments, use nginx in front of the API and web containers (/api -> 127.0.0.1:8000, / -> 127.0.0.1:8081):
sudo scripts/setup_nginx_spacegate.shFor host-local deployment notes, use:
docs/RUNTIME_NOTES_TEMPLATE.mdas a template- copy it to
/srv/spacegate/RUNTIME.md(outside git) and fill host-specific values
Behavior:
- Uses port 80 if free or already owned by nginx.
- Falls back to port 8080 if port 80 is in use by a non‑nginx process.
- Writes
/etc/nginx/sites-available/spacegate.confwith provenance comments. - Symlinks to
/etc/nginx/sites-enabled/spacegate.conf(without touching other sites). - Proxies web UI to container upstream
http://127.0.0.1:8081by default. - Applies API abuse controls by default: per-IP rate limit, burst limit, and connection limit.
- Applies proxy timeouts on API upstream connections.
- Serves
/dl/fromSPACEGATE_DL_ALIAS_DIR(defaults toSPACEGATE_DL_ROOT;SPACEGATE_DL_ENABLE=0disables/dl/). - Runs
nginx -tbefore reload/start.
If you prefer host-served static files from srv/web/dist, use:
sudo scripts/setup_nginx_spacegate.sh --static-web --forceTip: if you access by IP or a specific hostname, set it explicitly:
sudo SPACEGATE_SERVER_NAME="192.168.1.102" scripts/setup_nginx_spacegate.sh --forceRate-limit and timeout tuning example:
sudo SPACEGATE_SERVER_NAME="your-public-host.example www.your-public-host.example" \
SPACEGATE_API_RATE_RPS=15 \
SPACEGATE_API_RATE_BURST=30 \
SPACEGATE_API_CONN_LIMIT=30 \
SPACEGATE_PROXY_READ_TIMEOUT=45s \
scripts/setup_nginx_spacegate.sh --forceOptional HTTPS enforcement from this script (requires cert files):
sudo SPACEGATE_SERVER_NAME="your-public-host.example www.your-public-host.example" \
SPACEGATE_TLS_ENABLE=1 \
SPACEGATE_TLS_CERT_FILE=/etc/letsencrypt/live/your-public-host.example/fullchain.pem \
SPACEGATE_TLS_KEY_FILE=/etc/letsencrypt/live/your-public-host.example/privkey.pem \
scripts/setup_nginx_spacegate.sh --forceYou can either:
- Use script-managed TLS via
SPACEGATE_TLS_ENABLE=1and cert/key paths (shown above), or - Keep script-managed HTTP and let certbot edit nginx:
sudo certbot --nginx -d your-public-host.example -d www.your-public-host.example
If you use certbot-managed TLS, re-running scripts/setup_nginx_spacegate.sh --force will overwrite certbot's nginx edits.
Install and start the Spacegate API as a systemd service:
sudo scripts/install_spacegate_systemd.shThis runs uvicorn with --proxy-headers and --forwarded-allow-ips="*" so nginx can forward scheme/client IP.
Spacegate can run in Docker with two containers: API + web. The web container serves the static UI and proxies /api to the API container.
scripts/compose_spacegate.sh up --buildThis exposes:
- API:
http://127.0.0.1:8000 - Web container:
http://127.0.0.1:8081
Note: compose now binds API/web container ports to loopback only (127.0.0.1) by default.
Public traffic should go through host nginx (80/443) only.
By default, compose bind-mounts ./data from the repo into /data inside the API container.
For consistent env-file behavior (.spacegate.env, .spacegate.local.env), use the wrapper script:
scripts/compose_spacegate.sh up --buildDirect compose still works, but use an inline env prefix (or --env-file) when you need a custom mount:
SPACEGATE_DATA_DIR=/data/spacegate/data docker compose up --buildOptional API DuckDB runtime caps for smaller hosts:
SPACEGATE_API_DUCKDB_MEMORY_LIMIT=6GB \
SPACEGATE_API_DUCKDB_THREADS=4 \
scripts/compose_spacegate.sh up --buildYou still need to build the core database (once). Easiest path:
- Run the build on the host (recommended), then start compose (the container sees your mounted state dir at
/data). - Or exec into the API container and run the build scripts there (requires build tools inside the image).
If you want a dedicated “builder” container in compose, say the word and I’ll add it.
-
pipmissing in venv
Installpython3-venv(Debian/Ubuntu), then rerun./install_spacegate.sh. -
npmnot found or Node is too old
Install/upgrade Node.js (v18+; v20 recommended), then rerun./install_spacegate.sh. -
7znot found
Installp7zip-full(Debian/Ubuntu), then rerun./install_spacegate.sh. -
Port already in use
Stop the running instance withscripts/run_spacegate.sh --stop, or change ports with:SPACEGATE_API_PORT=8001 scripts/run_spacegate.shIf using dev web mode, also set:SPACEGATE_WEB_PORT=5174 scripts/run_spacegate.sh --web-dev -
Build verification fails
Runscripts/verify_build.shdirectly to see details. If it references an old build, rebuild withscripts/build_database.sh --overwrite.
- v1.1: static snapshot generation (SVG) with deterministic rendering rules.
- v1.2: additional catalogs / packs foundation before heavy enrichment.
- v1.3-v1.5: reference links, facts → exposition, and image generation.
- v1.6: precomputed 10-nearest neighbor graph.
- v2: browser 3D map.
- v2.2: rim overlays, engagement, and community ranking overlays.