Merge pull request #11 from rywils/feature/cve-database-distribution

Add CVE snapshot distribution

Author: rywils

.github/workflows/update-cve-db.yml

@@ -0,0 +1,74 @@
+name: Update CVE Database
+
+on:
+  schedule:
+    - cron: "17 6 * * *"
+  workflow_dispatch:
+    inputs:
+      full_rebuild:
+        description: Ignore the existing snapshot and rebuild from NVD
+        required: false
+        type: boolean
+        default: false
+
+permissions:
+  contents: write
+
+concurrency:
+  group: cve-db-producer
+  cancel-in-progress: false
+
+jobs:
+  sync-and-publish:
+    runs-on: ubuntu-latest
+    timeout-minutes: 360
+    env:
+      NVD_API_KEY: ${{ secrets.NVD_API_KEY }}
+      GH_TOKEN: ${{ github.token }}
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          persist-credentials: false
+
+      - name: Configure data directory
+        run: echo "BITSENTRY_DATA_DIR=$RUNNER_TEMP/bitsentry-data" >> "$GITHUB_ENV"
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.13"
+
+      - name: Install dependencies
+        run: pip install -r requirements.txt
+
+      - name: Require NVD API key
+        run: test -n "$NVD_API_KEY"
+
+      - name: Restore previous snapshot
+        id: restore
+        if: ${{ !inputs.full_rebuild }}
+        run: |
+          if gh release view cve-db-latest >/dev/null 2>&1; then
+            mkdir -p previous
+            gh release download cve-db-latest --dir previous --pattern manifest.json --pattern cve_db.sqlite.gz
+            PYTHONPATH=bitprobe python -m scanner.cve_db_bootstrap --manifest previous/manifest.json --artifact previous/cve_db.sqlite.gz
+            echo "restored=true" >> "$GITHUB_OUTPUT"
+          else
+            echo "restored=false" >> "$GITHUB_OUTPUT"
+          fi
+
+      - name: Update canonical database
+        env:
+          FULL_REBUILD: ${{ inputs.full_rebuild }}
+          SNAPSHOT_RESTORED: ${{ steps.restore.outputs.restored }}
+        run: |
+          if [[ "$FULL_REBUILD" == "true" || "$SNAPSHOT_RESTORED" != "true" ]]; then
+            PYTHONPATH=bitprobe python bitprobe/bitprobe.py update-cve-db --full --no-snapshot
+          else
+            PYTHONPATH=bitprobe python bitprobe/bitprobe.py update-cve-db --no-snapshot
+          fi
+
+      - name: Build snapshot
+        run: python scripts/build_cve_snapshot.py
+
+      - name: Publish releases
+        run: bash scripts/update_cve_snapshot_release.sh dist

.gitignore

@@ -81,3 +81,6 @@ AGENTS.md
 
 # Personal AI workflow tooling -- not project convention, not for the public repo
 .claude/
+
+# Isolated feature worktrees
+.worktrees/

CLAUDE.md

@@ -30,9 +30,9 @@ Network scanning has a fallback chain in `bitprobe/scanner/engines/network/__ini
 
 ## Data
 
-- CVE data: `bitprobe/scanner/data/cve_db.sqlite` (primary) with `cve_db.json` as fallback/legacy (`cve_db.py`, `cve_db_manager.py`, `cve_updater.py`). Refresh via `bitprobe update-cve-db` (NVD API key recommended, see README).
+- CVE data: `~/.bitsentry/data/cve_db.sqlite` (override with `BITSENTRY_DATA_DIR`), with the source-tree database used only as a one-time legacy migration source. Refresh via `bitsentry update-cve-db`; the default path installs a verified release snapshot before incremental NVD catch-up.
 - ASN/IP intel DB: `asn_db_updater.py`, refreshed via `bitsentry update-db`.
-- Neither DB is checked into git (`bitprobe/data/cve_db.sqlite` is gitignored — a prior commit removed a large SQLite file from the repo for this reason).
+- Neither generated database is checked into git. Published CVE snapshots live in reserved `cve-db-*` GitHub Releases.
 
 ## Commands
 

README.md

@@ -10,6 +10,7 @@ BitSentry is a CLI-first security assessment suite. The public build focuses on
 - web-focused vulnerability scanning
 
 It is built to run cleanly in local shells, CI pipelines, and Docker.
+Supported hosts are Linux and macOS.
 > Use only on systems you own or are explicitly authorized to test.
 
 ## Current Product Status
@@ -54,14 +55,8 @@ Refresh local intelligence databases once so scans are useful. This is separate
 # 1) ASN database (fast; needed for ASN/IP intel plugins)
 bitsentry update-db
 
-# 2) CVE database (choose one bootstrap — required for technology/CVE correlation)
-export NVD_API_KEY="your-nvd-api-key"   # optional but strongly recommended
-
-# Recommended: full local mirror (slow once; best coverage)
-bitsentry update-cve-db --full
-
-# Alternative: smaller first-time bootstrap (~15 years of publications)
-# bitsentry update-cve-db --years 15
+# 2) CVE database: verified snapshot, then incremental NVD catch-up
+bitsentry update-cve-db
 
 # Check what was loaded
 bitsentry cve-stats
@@ -70,14 +65,14 @@ bitsentry cve-stats
 bitsentry scan example.com
 ```
 
-**Ongoing maintenance** (after the one-time bootstrap above):
+**Ongoing maintenance:**
 
 ```bash
 bitsentry update-db              # refresh ASN data when stale
-bitsentry update-cve-db          # incremental CVE sync (fast)
+bitsentry update-cve-db          # snapshot if needed, otherwise incremental sync
 ```
 
-If you skip CVE bootstrap, the first scan may still run but will only auto-fetch a **short recent-publication window**—not enough for historical product/CVE exposure. See [CVE database](#cve-database) below for details.
+If you skip this step, the first scan uses the same snapshot bootstrap automatically. See [CVE database](#cve-database) for direct-NVD and offline fallback behavior.
 
 ### Option 2: manual setup
 
@@ -88,8 +83,7 @@ pip install -r requirements.txt
 
 # Same post-install DB steps as Option 1 (use python bitsentry.py if bitsentry is not on PATH)
 python bitsentry.py update-db
-export NVD_API_KEY="your-nvd-api-key"   # optional
-python bitsentry.py update-cve-db --full   # or: --years 15
+python bitsentry.py update-cve-db
 python bitsentry.py cve-stats
 
 # Full workflow (default): BitScope discovery -> BitProbe scan
@@ -159,48 +153,49 @@ python bitsentry.py update-db          # alias: update-asn-db
 
 ### CVE database
 
-BitProbe stores CVEs in a local SQLite database (`bitprobe/data/cve_db.sqlite`) and matches them **by detected product and version** during scans—not by “CVEs published in the last N days.”
+BitProbe stores mutable CVE data in `~/.bitsentry/data/cve_db.sqlite` and matches CVEs by detected product and version. Set `BITSENTRY_DATA_DIR` to use a different data directory.
 
 | Phase | What happens |
 |---|---|
-| **Bootstrap** | Populates the local DB (one-time or after a wipe) |
+| **Bootstrap** | Downloads and verifies the published full-corpus snapshot |
 | **Incremental sync** | Fetches only NVD records modified since the last cursor (fast) |
 | **Scan** | Fingerprints the target, then queries the DB for that product/CPE |
 
-A short publication window (for example `--days 30`) only controls **what gets downloaded into the DB**. It does not limit scan logic. For real exposure coverage, bootstrap with a full or multi-year mirror first, then rely on incremental updates.
-
-**Recommended first-time setup:**
+The default command installs a verified snapshot when the database is missing or incomplete, then fetches changes made after the snapshot cursor:
 
 ```bash
-# Optional but strongly recommended (higher NVD rate limits)
-export NVD_API_KEY="your-nvd-api-key"
-
-# One-time: build a complete local mirror (slow; ~350k CVEs)
-python bitsentry.py update-cve-db --full
-
-# Alternative: compromise bootstrap (~15 years of publications)
-python bitsentry.py update-cve-db --years 15
-
-# Ongoing refresh (incremental when a sync cursor exists)
 python bitsentry.py update-cve-db
 
-# Inspect local store
+# Install the snapshot without an incremental NVD catch-up
+python bitsentry.py update-cve-db --snapshot-only
+
+# Inspect local coverage and counts
 python bitsentry.py cve-stats
 ```
 
-**Other options:**
+Direct-NVD modes skip the snapshot. BitSentry splits long NVD date ranges into 119-day windows:
 
 ```bash
-# Quick bootstrap only (~recent publications; not sufficient alone for deep history)
+# Rebuild the complete corpus directly from NVD
+python bitsentry.py update-cve-db --full
+
+# Raw unfiltered crawl (best-effort offset resumption)
+python bitsentry.py update-cve-db --raw-full
+
+# Build partial publication-window databases
 python bitsentry.py update-cve-db --days 30
+python bitsentry.py update-cve-db --years 15
+
+# Synchronize directly without downloading a snapshot
+python bitsentry.py update-cve-db --no-snapshot
 
 # Skip automatic CVE refresh at scan startup
 export BITSENTRY_SKIP_CVE_UPDATE=1
 ```
 
-On scan startup, if the DB is empty, BitProbe may run a **7-day publication bootstrap** so the tool stays usable without blocking on a full NVD download. Run `update-cve-db --full` or `--years 15` before relying on CVE findings in production assessments.
+Set `NVD_API_KEY` for the higher NVD request limit. Interrupted windowed updates resume from the last committed page. BitSentry checksum-verifies snapshots and installs them atomically. If the snapshot is unavailable on an empty installation, it falls back to a 30-day publication database and warns that coverage is partial.
 
-Direct product commands are also available via `python bitprobe/bitprobe.py ...` (same flags: `--full`, `--years`, `--days`).
+Direct product commands are also available via `python bitprobe/bitprobe.py ...` with the same flags.
 
 ### Other maintenance
 

bitprobe/bitprobe.py

@@ -16,6 +16,7 @@
 from scanner.config import ScanConfig, SCAN_PROFILES
 from scanner.asn_db_updater import update_asn_db
 from scanner.cve_db_manager import update_cve_database, get_stats
+from scanner.cve_db_bootstrap import update_with_snapshot_policy
 
 
 def cmd_scan(args) -> int:
@@ -191,8 +192,8 @@ def main() -> int:
     cve_parser.add_argument(
         "--days",
         type=int,
-        default=30,
-        help="Publication window for bootstrap when DB is empty (default: 30)",
+        default=None,
+        help="Build a publication-window mirror directly from NVD",
     )
     cve_parser.add_argument(
         "--years",
@@ -203,7 +204,22 @@ def main() -> int:
     cve_parser.add_argument(
         "--full",
         action="store_true",
-        help="Build full local NVD mirror (~350k CVEs; first-time setup)",
+        help="Rebuild the full local mirror directly from NVD",
+    )
+    cve_parser.add_argument(
+        "--raw-full",
+        action="store_true",
+        help="Best-effort unfiltered NVD crawl; offset resumption is not deterministic",
+    )
+    cve_parser.add_argument(
+        "--snapshot-only",
+        action="store_true",
+        help="Install the published snapshot without contacting NVD afterward",
+    )
+    cve_parser.add_argument(
+        "--no-snapshot",
+        action="store_true",
+        help="Use direct NVD synchronization without downloading a snapshot",
     )
 
     cve_stats_parser = subparsers.add_parser(
@@ -226,14 +242,31 @@ def main() -> int:
 
     elif args.command == "update-cve-db":
         try:
-            full_sync = getattr(args, "full", False)
-            count = update_cve_database(
-                days=args.days,
-                years=getattr(args, "years", None),
-                full_sync=full_sync,
-                force=full_sync,
-                verbose=verbose,
-            )
+            raw_full = getattr(args, "raw_full", False)
+            full_sync = getattr(args, "full", False) or raw_full
+            years = getattr(args, "years", None)
+            days = getattr(args, "days", None)
+            snapshot_only = getattr(args, "snapshot_only", False)
+            direct = full_sync or years is not None or days is not None or getattr(args, "no_snapshot", False)
+            if snapshot_only and direct:
+                raise ValueError("--snapshot-only cannot be combined with direct-NVD options")
+            if direct:
+                count = update_cve_database(
+                    days=days if days is not None else 30,
+                    years=years,
+                    full_sync=full_sync,
+                    raw_full_sync=raw_full,
+                    force=full_sync,
+                    verbose=verbose,
+                )
+            else:
+                count = update_with_snapshot_policy(
+                    snapshot_only=snapshot_only,
+                    verbose=verbose,
+                )
+            if snapshot_only:
+                print("[+] CVE database snapshot installed")
+                return 0
             print(f"[+] CVE database updated with {count} entries")
             return 0
         except Exception as e:
@@ -247,6 +280,8 @@ def main() -> int:
             print("=" * 40)
             print(f"Total CVEs: {stats.get('total_cves', 0)}")
             print(f"Total Products: {stats.get('total_products', 0)}")
+            print(f"Coverage: {stats.get('coverage_mode', 'unknown')}")
+            print(f"NVD Cursor: {stats.get('nvd_cursor', 'Never')}")
             print(f"Last Updated: {stats.get('last_updated', 'Never')}")
             print("\nBy Severity:")
             for sev, count in stats.get('severity_counts', {}).items():

bitprobe/scanner/cve_db.py

@@ -2,10 +2,10 @@
 import os
 from pathlib import Path
 from typing import List, Dict, Any
+from scanner.paths import CVE_DB_PATH as CVE_SQLITE_PATH
 
 _DATA_DIR = Path(__file__).resolve().parents[1] / "data"
 CVE_DB_PATH = str(_DATA_DIR / "cve_db.json")
-CVE_SQLITE_PATH = str(_DATA_DIR / "cve_db.sqlite")
 
 
 def sqlite_cve_db_available() -> bool: