feat(build_transformer): add Caddy file server support with JSON API #304

CharlVS · 2025-12-11T10:21:47Z

Summary

Add CaddyArtefactDownloader that uses Caddy's JSON directory listing API instead of HTML scraping
Update ArtefactDownloaderFactory to detect and route Caddy-hosted mirrors (e.g., devbuilds.gleec.com)
Add --source caddy option to update_api_config CLI tool for Caddy mirror support
Fix PowerShell Expand-Archive invocation on Windows (requires -Command flag)

Details

Caddy JSON API

Caddy provides structured JSON directory listings when the Accept: application/json header is included:

[
  {
    "name": "kdf_9e6f4e5-wasm.zip",
    "size": 11739893,
    "url": "./kdf_9e6f4e5-wasm.zip",
    "mod_time": "2025-12-10T20:37:05Z",
    "is_dir": false,
    "is_symlink": false
  }
]

This is more reliable than HTML scraping and provides structured metadata.

Windows PowerShell Fix

The existing PowerShell extraction code was incorrect - cmdlets like Expand-Archive must be passed via the -Command flag:

// Before (broken)
Process.run('powershell', ['Expand-Archive', '-Path', ...])

// After (fixed)
Process.run('powershell', ['-Command', 'Expand-Archive -Path "..." -DestinationPath "..." -Force'])

CLI Usage

dart run komodo_wallet_cli:update_api_config \
  --branch dev \
  --source caddy \
  --mirror-url https://devbuilds.gleec.com \
  --config path/to/build_config.json

Test plan

1. Static analysis

cd packages/komodo_wallet_build_transformer
dart analyze

No errors

2. Test CLI with Caddy source

dart run packages/komodo_wallet_cli/bin/update_api_config.dart \
  --branch tests-split-docker-tests \
  --commit 9e6f4e5 \
  --source caddy \
  --mirror-url https://devbuilds.gleec.com \
  --config packages/komodo_defi_framework/app_build/build_config.json \
  --output-dir packages/komodo_defi_framework/app_build/temp_downloads \
  --verbose \
  --no-strict

Verify all platforms download successfully (web, ios, macos, windows, android-armv7, android-aarch64, linux)
Verify checksums are calculated and config is updated

3. Verify build with downloaded artifacts

# Reset config to original values first, then:
flutter pub get
flutter build web  # or other platform

Verify build completes without KDF download errors

4. Test Windows extraction (manual)

Run the build transformer on Windows to verify PowerShell extraction works

Add a dedicated CaddyArtefactDownloader that uses Caddy's JSON directory listing API instead of HTML scraping. This provides more reliable and structured file discovery for KDF artifact downloads. Changes: - Add CaddyArtefactDownloader with JSON API support (Accept: application/json) - Update ArtefactDownloaderFactory to detect and route Caddy-hosted mirrors - Add --source caddy option to update_api_config CLI tool - Fix PowerShell Expand-Archive invocation on Windows (requires -Command flag) - Update BUILD_CONFIG_README with Caddy mirror documentation The Caddy JSON API returns structured directory listings which is more robust than parsing HTML. Recursive directory search is supported up to 3 levels deep.

github-actions · 2025-12-11T10:26:56Z

Visit the preview URL for this PR (updated for commit 6758fed):

https://komodo-playground--pr304-feat-caddy-mirror-su-j214jna9.web.app

_{(expires Thu, 18 Dec 2025 10:36:27 GMT)}

_{🔥 via Firebase Hosting GitHub Action 🌎}

_{Sign: 2bfedd77fdea45b25ba7c784416e81f177aa5c47}

Copilot

Pull request overview

This PR adds support for Caddy-based file servers with JSON API directory listings as an alternative to HTML-based mirror scraping and GitHub releases. It introduces a new CaddyArtefactDownloader class that leverages Caddy's structured JSON API for more reliable artifact discovery, routing for the devbuilds.gleec.com mirror, and fixes PowerShell extraction commands across all downloader implementations.

Key Changes:

Added CaddyArtefactDownloader that fetches directory listings via JSON API with recursive subdirectory search
Updated ArtefactDownloaderFactory to detect Caddy hosts and route appropriately
Fixed PowerShell Expand-Archive invocation to use -Command flag with single-string format across all downloaders

Reviewed changes

Copilot reviewed 5 out of 6 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
`packages/komodo_wallet_cli/bin/update_api_config.dart`	Added `caddy` source option to CLI, updated help text and examples, added `_fetchCaddyDownloadUrl` and `_searchCaddyDirectory` methods for JSON API interaction
`packages/komodo_wallet_build_transformer/lib/src/steps/defi_api_build_step/caddy_artefact_downloader.dart`	New file implementing `CaddyArtefactDownloader` with JSON API support, including `CaddyFileEntry` model and recursive directory search with hash matching
`packages/komodo_wallet_build_transformer/lib/src/steps/defi_api_build_step/artefact_downloader_factory.dart`	Added `_caddyHosts` whitelist and `_isCaddyServer` detection method, updated factory logic to route Caddy URLs to `CaddyArtefactDownloader`
`packages/komodo_wallet_build_transformer/lib/src/steps/defi_api_build_step/github_artefact_downloader.dart`	Fixed PowerShell extraction command to use `-Command` flag with properly quoted paths
`packages/komodo_wallet_build_transformer/lib/src/steps/defi_api_build_step/dev_builds_artefact_downloader.dart`	Fixed PowerShell extraction command to use `-Command` flag with properly quoted paths
`packages/komodo_defi_framework/app_build/BUILD_CONFIG_README.md`	Updated documentation to explain three mirror types (GLEEC DevBuilds with JSON API, Nebula HTML, Komodo DevBuilds HTML), added clarity about Caddy-based servers

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-12-11T10:29:10Z

...o_wallet_build_transformer/lib/src/steps/defi_api_build_step/github_artefact_downloader.dart

+          '-Command',
+          'Expand-Archive -Path "$filePath" -DestinationPath "$destinationFolder" -Force',


On Windows, filePath and destinationFolder are interpolated directly into a PowerShell -Command string in Expand-Archive -Path "$filePath" -DestinationPath "$destinationFolder" -Force, which allows PowerShell expression injection if an attacker can influence the archive filename or path (e.g., via $() in a downloaded asset name). Because these paths ultimately derive from remote URLs (GitHub release assets), a compromised or malicious source could craft a filename that executes arbitrary PowerShell code when this command runs. To fix this, avoid embedding untrusted values in a PowerShell script string and instead either use a non-shell unzip library in Dart or invoke PowerShell with parameters that are not subject to string interpolation (e.g., use a small wrapper script with param() and pass paths via -File/-ArgumentList).

Copilot · 2025-12-11T10:29:11Z

...llet_build_transformer/lib/src/steps/defi_api_build_step/dev_builds_artefact_downloader.dart

        final result = await Process.run('powershell', [
-          'Expand-Archive',
-          '-Path',
-          filePath,
-          '-DestinationPath',
-          destinationFolder,
+          '-Command',
+          'Expand-Archive -Path "$filePath" -DestinationPath "$destinationFolder" -Force',


On Windows, filePath and destinationFolder are interpolated directly into a PowerShell -Command string in Expand-Archive -Path "$filePath" -DestinationPath "$destinationFolder" -Force, which enables PowerShell expression injection if an attacker can control the archive filename or path (e.g., by including $() in a downloaded file name). Here those paths come from remote HTTP downloads (dev builds mirrors), so a compromised or malicious mirror could craft filenames that execute arbitrary PowerShell commands when extraction runs. Mitigate this by not constructing a script string from untrusted data—either use a native unzip library in Dart or a PowerShell wrapper that accepts paths as parameters (via -File/-ArgumentList) without double-quoted interpolation.

Copilot · 2025-12-11T10:29:11Z

...do_wallet_build_transformer/lib/src/steps/defi_api_build_step/caddy_artefact_downloader.dart

+        final result = await Process.run('powershell', [
+          '-Command',
+          'Expand-Archive -Path "$filePath" -DestinationPath "$destinationFolder" -Force',


On Windows, filePath and destinationFolder are interpolated directly into a PowerShell -Command string in Expand-Archive -Path "$filePath" -DestinationPath "$destinationFolder" -Force, which allows PowerShell expression injection if a Caddy-hosted artifact filename or path can be influenced by an attacker (e.g., including $() in the url/basename returned by the JSON listing). Since Caddy directory listings and download URLs are driven by remote data, a compromised or malicious Caddy server could supply filenames that execute arbitrary PowerShell code when this extraction step runs. To harden this, avoid embedding untrusted paths in a PowerShell script string and instead use a safer extraction mechanism (e.g., a Dart unzip library or a PowerShell script invoked via -File with paths passed as arguments rather than interpolated into double-quoted strings).

Copilot AI review requested due to automatic review settings December 11, 2025 10:21

Copilot started reviewing on behalf of CharlVS December 11, 2025 10:22 View session

CharlVS requested a review from DeckerSU December 11, 2025 10:25

CharlVS self-assigned this Dec 11, 2025

Copilot AI reviewed Dec 11, 2025

View reviewed changes

chore(formatting): add missing blank EOF line

6758fed

CharlVS mentioned this pull request Dec 15, 2025

fix nebula api source #306

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(build_transformer): add Caddy file server support with JSON API #304

feat(build_transformer): add Caddy file server support with JSON API #304

Uh oh!

CharlVS commented Dec 11, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Dec 11, 2025 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Dec 11, 2025

Uh oh!

Copilot AI Dec 11, 2025

Uh oh!

Copilot AI Dec 11, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		'-Command',
		'Expand-Archive -Path "$filePath" -DestinationPath "$destinationFolder" -Force',

feat(build_transformer): add Caddy file server support with JSON API #304

Are you sure you want to change the base?

feat(build_transformer): add Caddy file server support with JSON API #304

Uh oh!

Conversation

CharlVS commented Dec 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Details

Caddy JSON API

Windows PowerShell Fix

CLI Usage

Test plan

1. Static analysis

2. Test CLI with Caddy source

3. Verify build with downloaded artifacts

4. Test Windows extraction (manual)

Uh oh!

github-actions bot commented Dec 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Dec 11, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 11, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 11, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

CharlVS commented Dec 11, 2025 •

edited

Loading

github-actions bot commented Dec 11, 2025 •

edited

Loading