Reconcile stdin config support and developer docs with implementation (#5645)

## Bug Fix

The nightly documentation reconciliation found several drifts between
the documented config surface and the actual implementation. The main
gaps were missing JSON stdin support for `gateway.payloadPathPrefix`,
inconsistent stdin server timeout field naming, and contributor docs
that no longer matched the `internal/config` package layout or
validation rules.

### What was the bug?

- `AGENTS.md` implied a single `internal/config/config.go` entry point
even though the config package is split across multiple files.
- JSON stdin config could not set `gateway.payloadPathPrefix`, despite
TOML and env/flag paths supporting it.
- Per-server stdin timeout fields used `snake_case` while the rest of
the JSON gateway config uses `camelCase`.
- Docs/examples were missing or outdated for:
  - absolute-path requirements on `payload_dir`
  - OpenTelemetry vs legacy tracing key precedence
- integration-test env vars `AWMG_BINARY_PATH` and
`AWMG_WASM_GUARD_PATH`

### How did you fix it?

- **JSON stdin config parity**
- Added `payloadPathPrefix` to `StdinGatewayConfig` and wired it into
stdin-to-runtime config conversion.
- Standardized stdin server timeout fields on `connectTimeout` /
`toolTimeout`.
- Kept `connect_timeout` / `tool_timeout` as backward-compatible JSON
aliases during unmarshalling.
- Updated the stdin JSON schema and config tests to cover the new field
and alias behavior.

- **Developer docs reconciliation**
- Updated `AGENTS.md` to describe the real multi-file `internal/config`
layout, including `config_core.go`, `config_stdin.go`, feature-specific
config files, and split validation files.
- Documented that `payload_dir` / `MCP_GATEWAY_PAYLOAD_DIR` must be
absolute in examples and reference docs.
- Added migration guidance for `[gateway.opentelemetry]` vs legacy
`[gateway.tracing]`, including precedence when both are present.
- Added integration test environment variable docs to `CONTRIBUTING.md`.

- **Reference/example cleanup**
- Updated config examples and docs to reflect the supported JSON field
names and current behavior.
- Added `payloadPathPrefix` to the configuration reference alongside
existing payload settings.

### Testing

- Added/updated focused config tests for:
  - `gateway.payloadPathPrefix` in stdin JSON
  - camelCase per-server timeout fields
  - backward-compatible snake_case timeout aliases

```json
{
  "gateway": {
    "apiKey": "${MCP_GATEWAY_API_KEY}",
    "payloadPathPrefix": "/workspace/payloads"
  },
  "mcpServers": {
    "repo-mind": {
      "type": "http",
      "url": "http://127.0.0.1:8000/mcp",
      "connectTimeout": 45,
      "toolTimeout": 600
    }
  }
}
```

Author: lpcox

AGENTS.md

@@ -24,8 +24,11 @@ Quick reference for AI agents working with MCP Gateway (Go-based MCP proxy serve
 
 - `internal/auth/` - Authentication header parsing and middleware
 - `internal/cmd/` - CLI (Cobra)
-- `internal/config/` - Config parsing (TOML/JSON) with validation
-  - `validation.go` - Variable expansion and fail-fast validation
+- `internal/config/` - Config parsing (TOML/JSON) with validation; the package is split across multiple files
+  - `config_core.go` - Core `Config`, `GatewayConfig`, and `ServerConfig` types plus TOML loading
+  - `config_stdin.go` - JSON stdin structs and stdin→internal config conversion
+  - `config_env.go`, `config_feature.go`, `config_tracing.go` - Environment- and feature-specific config helpers
+  - `validation.go`, `validation_env.go`, `validation_schema.go` - Fail-fast field, environment, and schema validation
   - `validation_test.go` - Comprehensive validation tests
 - `internal/difc/` - Decentralized Information Flow Control
 - `internal/envutil/` - Environment variable utilities
@@ -64,7 +67,7 @@ Quick reference for AI agents working with MCP Gateway (Go-based MCP proxy serve
 [gateway]
 port = 3000
 api_key = "your-api-key"
-payload_dir = "/tmp/jq-payloads"  # Optional: directory for large payload storage
+payload_dir = "/tmp/jq-payloads"  # Optional: directory for large payload storage (must be absolute)
 
 [servers.github]
 command = "docker"
@@ -383,8 +386,8 @@ DEBUG_COLORS=0 DEBUG=* ./awmg --config config.toml
 - `DEBUG_COLORS` - Control colored output (0 to disable, auto-disabled when piping)
 - `MCP_GATEWAY_LOG_DIR` - Log file directory (sets default for `--log-dir` flag, default: `/tmp/gh-aw/mcp-logs`)
 - `MCP_GATEWAY_WASM_CACHE_DIR` - Disk-backed wazero compilation cache directory (sets default for `--wasm-cache-dir`, default: `<log-dir>/wazero-cache`)
-- `MCP_GATEWAY_PAYLOAD_DIR` - Large payload storage directory (sets default for `--payload-dir` flag, default: `/tmp/jq-payloads`)
-- `MCP_GATEWAY_PAYLOAD_PATH_PREFIX` - Path prefix for remapping payloadPath returned to clients (sets default for `--payload-path-prefix` flag, default: empty)
+- `MCP_GATEWAY_PAYLOAD_DIR` - Large payload storage directory (sets default for `--payload-dir` flag, default: `/tmp/jq-payloads`). Must be an absolute path.
+- `MCP_GATEWAY_PAYLOAD_PATH_PREFIX` - Path prefix for remapping payloadPath returned to clients (sets default for `--payload-path-prefix` flag, default: empty). In JSON stdin config use `gateway.payloadPathPrefix`.
 - `MCP_GATEWAY_PAYLOAD_SIZE_THRESHOLD` - Size threshold in bytes for payload storage; payloads larger than this are stored to disk (sets default for `--payload-size-threshold` flag, default: `524288`)
 - `MCP_GATEWAY_SESSION_TIMEOUT` - Session timeout for stateful sessions in both unified mode (`/mcp`) and routed mode (`/mcp/<server>`). Accepts Go duration strings (e.g., `30m`, `1h`, `2h30m`). (default: `6h`)
 - `MCP_GATEWAY_TOOL_TIMEOUT` - Tool invocation timeout in seconds. Fallback when `gateway.toolTimeout` is not set in stdin JSON config. Accepts any integer ≥ 10 (no upper bound). Priority: stdin config > env var > built-in default. (default: `60`)
@@ -426,7 +429,7 @@ DEBUG_COLORS=0 DEBUG=* ./awmg --config config.toml
 
 **Large Payload Handling:**
 - Large tool response payloads are stored in the configured payload directory
-- Default payload directory: `/tmp/jq-payloads` (configurable via `--payload-dir` flag, `MCP_GATEWAY_PAYLOAD_DIR` env var, or `payload_dir` in config)
+- Default payload directory: `/tmp/jq-payloads` (configurable via `--payload-dir` flag, `MCP_GATEWAY_PAYLOAD_DIR` env var, or `payload_dir` in config). The configured path must be absolute.
 - Payloads are organized by session ID: `{payload_dir}/{sessionID}/{queryID}/payload.json`
 - This allows agents to mount their session-specific subdirectory to access full payloads
 - The jq middleware returns: preview (first `PayloadPreviewSize` chars, default 500), schema, payloadPath, queryID, originalSize, truncated flag

CONTRIBUTING.md

@@ -107,6 +107,11 @@ make test-container-proxy
 
 This target builds a Docker image and tests proxy mode with TLS. It requires a GitHub token available via the `gh` CLI (`gh auth login`) or the `GITHUB_TOKEN`/`GH_TOKEN` environment variable.
 
+#### Testing Environment Variables
+
+- `AWMG_BINARY_PATH` — Override the binary path used by integration tests when you want to run tests against a prebuilt `awmg` binary.
+- `AWMG_WASM_GUARD_PATH` — Override the GitHub guard WASM path used by proxy integration tests when the default build output path is not available.
+
 #### Race Detection Tests
 Run unit tests with Go's race detector to catch concurrent data races:
 ```bash
@@ -170,8 +175,8 @@ echo '{"mcpServers": {...}}' | ./awmg --config-stdin
 # Increase verbosity
 ./awmg --config config.toml -v
 
-# Custom payload directory and size threshold
-./awmg --config config.toml --payload-dir ./payloads --payload-size-threshold 1048576
+# Custom payload directory and size threshold (payload dir must be absolute)
+./awmg --config config.toml --payload-dir /tmp/payloads --payload-size-threshold 1048576
 ```
 
 See [docs/ENVIRONMENT_VARIABLES.md](docs/ENVIRONMENT_VARIABLES.md) for the full list of environment variable overrides.

README.md

@@ -47,6 +47,7 @@ The gateway starts in routed mode on `http://0.0.0.0:8000`, proxying MCP request
 - `-i`: Enables stdin for passing JSON configuration
 - `-v /var/run/docker.sock`: Required for spawning backend MCP servers
 - `-p 8000:8000`: Port mapping must match `MCP_GATEWAY_PORT`
+- If you configure `payloadDir` / `MCP_GATEWAY_PAYLOAD_DIR`, use an absolute path (for example `/tmp/jq-payloads`)
 
 ## Guard Policies
 

config.example-payload-threshold.toml

@@ -12,6 +12,7 @@ port = 3000
 api_key = "your-api-key-here"
 
 # Payload directory for storing large tool responses
+# Must be an absolute path
 # Can also be set via:
 # - Flag: --payload-dir /custom/path
 # - Env:  MCP_GATEWAY_PAYLOAD_DIR=/custom/path

config.example.toml

@@ -31,6 +31,7 @@ startup_timeout = 60
 tool_timeout = 120
 
 # Directory for storing large payload files (default: /tmp/jq-payloads)
+# Must be an absolute path
 # Payloads are segmented by session ID: {payload_dir}/{sessionID}/{queryID}/payload.json
 # Can also be set via MCP_GATEWAY_PAYLOAD_DIR environment variable or --payload-dir flag
 # payload_dir = "/tmp/jq-payloads"
@@ -39,6 +40,11 @@ tool_timeout = 120
 # Prevents remote servers from expiring idle sessions by sending periodic pings.
 # Set to -1 to disable keepalive pings entirely.
 # keepalive_interval = 1500
+#
+# OpenTelemetry TOML key migration:
+# - Prefer [gateway.opentelemetry] for new configs.
+# - Legacy [gateway.tracing] is still accepted for TOML migrations.
+# - If both sections are present, [gateway.opentelemetry] takes precedence.
 
 # ============================================================================
 # MCP Server Configurations

docs/CONFIGURATION.md

@@ -185,11 +185,11 @@ Run `./awmg --help` for full CLI options. Key flags:
   - Enables per-server DIFC guard assignment independent of `guard-policies`
   - Example: `guard = "github"` (uses the guard named `github` from `[guards.github]`)
 
-- **`connect_timeout`** (optional, HTTP servers only): Per-transport connection timeout in seconds for connecting to HTTP backends. The gateway tries streamable HTTP, then SSE, then plain JSON-RPC over HTTP POST in sequence; this timeout applies to each attempt. It does **not** set the end-to-end `tools/call` execution timeout. Default: `30`.
+- **`connectTimeout`** (preferred JSON) / **`connect_timeout`** (legacy JSON alias, HTTP servers only): Per-transport connection timeout in seconds for connecting to HTTP backends. The gateway tries streamable HTTP, then SSE, then plain JSON-RPC over HTTP POST in sequence; this timeout applies to each attempt. It does **not** set the end-to-end `tools/call` execution timeout. Default: `30`.
 
-- **`tool_timeout`** (optional): Per-server tool invocation timeout in seconds. When set to a positive value, overrides the global tool timeout for `tools/call` requests to this specific server. This allows reusable shared workflow components wrapping long-running HTTP MCP servers to set an appropriate timeout once, without requiring every consumer to configure `MCP_GATEWAY_TOOL_TIMEOUT`. Minimum: `10`. Omit the field (or set to `0`) to fall back to the global timeout.
+- **`toolTimeout`** (preferred JSON) / **`tool_timeout`** (legacy JSON alias, optional): Per-server tool invocation timeout in seconds. When set to a positive value, overrides the global tool timeout for `tools/call` requests to this specific server. This allows reusable shared workflow components wrapping long-running HTTP MCP servers to set an appropriate timeout once, without requiring every consumer to configure `MCP_GATEWAY_TOOL_TIMEOUT`. Minimum: `10`. Omit the field (or set to `0`) to fall back to the global timeout.
   - Global timeout field name: **`toolTimeout`** in stdin JSON (`gateway.toolTimeout`), **`tool_timeout`** in TOML (`[gateway]` → `tool_timeout`)
-  - Example (stdin JSON): `"tool_timeout": 600` on an HTTP MCP server that may take up to 10 minutes
+  - Example (stdin JSON): `"toolTimeout": 600` on an HTTP MCP server that may take up to 10 minutes
   - Example (TOML): `tool_timeout = 600` under a `[servers.my-server]` section
 
 - **`rate_limit_threshold`** (optional, TOML/JSON file configs only): Number of consecutive rate-limit errors from this backend that will trip the circuit breaker (transition CLOSED → OPEN). When OPEN, requests are immediately rejected until the breaker is eligible to transition to HALF-OPEN again; this is normally controlled by `rate_limit_cooldown`, but if the gateway knows an upstream rate-limit reset time (for example from response headers or parsed tool error text), that reset time takes precedence. **Not available in JSON stdin format.** Default: `3`.
@@ -394,7 +394,8 @@ The `customSchemas` top-level field allows you to define custom server types bey
 | `domain` | Gateway domain (`"localhost"`, `"host.docker.internal"`, or `"${VAR}"`) | (unset) |
 | `startupTimeout` | Seconds to wait for backend startup | `30` |
 | `toolTimeout` | Maximum seconds for a single tool call, enforced as a context deadline on all backend requests (stdio and HTTP) | `60` |
-| `payloadDir` | Directory for large payload files | `/tmp/jq-payloads` |
+| `payloadDir` | Directory for large payload files. Must be an absolute path. | `/tmp/jq-payloads` |
+| `payloadPathPrefix` | Optional path prefix used when returning `payloadPath` values to clients (for example when the host payload directory is mounted at a different in-container path) | (empty - use actual filesystem path) |
 | `payloadSizeThreshold` (JSON) / `payload_size_threshold` (TOML) | Size threshold in bytes; responses larger than this are stored to disk and returned as a `payloadPath` reference | `524288` (512 KB) |
 | `trustedBots` (JSON) / `trusted_bots` (TOML) | Optional list of additional bot usernames to trust with "approved" integrity level. Additive to the built-in trusted bot list. When specified, must be a non-empty array with non-empty string entries (spec §4.1.3.4); omit the field entirely if not needed. Example: `["my-bot[bot]", "org-automation"]` | (disabled) |
 | `keepaliveInterval` (JSON) / `keepalive_interval` (TOML) | Interval (seconds) between keepalive pings sent to HTTP backends. Prevents remote servers from expiring idle sessions. Set to `-1` to disable keepalive pings entirely. | `1500` (25 min) |

docs/ENVIRONMENT_VARIABLES.md

@@ -23,7 +23,7 @@ When running locally (`run.sh`), these variables are optional (warnings shown if
 | `MCP_GATEWAY_API_KEY` | Informational only — not read directly by the binary; must be referenced in your config via `"${MCP_GATEWAY_API_KEY}"` to enable authentication | (disabled) |
 | `MCP_GATEWAY_LOG_DIR` | Log file directory (sets default for `--log-dir` flag) | `/tmp/gh-aw/mcp-logs` |
 | `MCP_GATEWAY_WASM_CACHE_DIR` | Disk-backed wazero compilation cache directory (sets default for `--wasm-cache-dir`; defaults to `<log-dir>/wazero-cache`) | `/tmp/gh-aw/mcp-logs/wazero-cache` |
-| `MCP_GATEWAY_PAYLOAD_DIR` | Large payload storage directory (sets default for `--payload-dir` flag) | `/tmp/jq-payloads` |
+| `MCP_GATEWAY_PAYLOAD_DIR` | Large payload storage directory (sets default for `--payload-dir` flag). Must be an absolute path. | `/tmp/jq-payloads` |
 | `MCP_GATEWAY_PAYLOAD_PATH_PREFIX` | Path prefix for remapping payloadPath returned to clients (sets default for `--payload-path-prefix` flag) | (empty - use actual filesystem path) |
 | `MCP_GATEWAY_PAYLOAD_SIZE_THRESHOLD` | Size threshold in bytes for payload storage (sets default for `--payload-size-threshold` flag) | `524288` |
 | `MCP_GATEWAY_SESSION_TIMEOUT` | Session timeout for stateful sessions in both unified (`/mcp`) and routed (`/mcp/<server>`) modes. Accepts Go duration strings (e.g., `30m`, `1h`). Default is 6 hours to match the GitHub Actions default timeout. | `6h` |

internal/config/config_stdin.go

@@ -39,6 +39,7 @@ type StdinGatewayConfig struct {
 	ToolTimeout          *int                      `json:"toolTimeout,omitempty"`
 	KeepaliveInterval    *int                      `json:"keepaliveInterval,omitempty"`
 	PayloadDir           string                    `json:"payloadDir,omitempty"`
+	PayloadPathPrefix    *string                   `json:"payloadPathPrefix,omitempty"`
 	PayloadSizeThreshold *int                      `json:"payloadSizeThreshold,omitempty"`
 	TrustedBots          []string                  `json:"trustedBots,omitempty"`
 	OpenTelemetry        *StdinOpenTelemetryConfig `json:"opentelemetry,omitempty"`
@@ -129,17 +130,19 @@ type StdinServerConfig struct {
 
 	// ConnectTimeout is the per-transport timeout (in seconds) for connecting to HTTP backends.
 	// Only applies to HTTP server types. Default: 30 seconds.
-	ConnectTimeout *int `json:"connect_timeout,omitempty"`
+	ConnectTimeout *int `json:"connectTimeout,omitempty"`
 
 	// ToolTimeout is the per-server maximum time (seconds) to wait for a single tool invocation.
 	// When set to a positive value, this overrides the global gateway.toolTimeout for calls to
 	// this server only. Minimum: 10. Omit the field (or set to 0) to fall back to the global
 	// gateway.toolTimeout (or MCP_GATEWAY_TOOL_TIMEOUT env fallback).
-	ToolTimeout *int `json:"tool_timeout,omitempty"`
+	ToolTimeout *int `json:"toolTimeout,omitempty"`
 
 	// AdditionalProperties stores any extra fields for custom server types
 	// This allows custom schemas to define their own fields beyond the standard ones
 	AdditionalProperties map[string]interface{} `json:"-"`
+
+	toolTimeoutFieldName string `json:"-"`
 }
 
 // UnmarshalJSON implements custom JSON unmarshaling to capture additional properties
@@ -157,6 +160,25 @@ func (s *StdinServerConfig) UnmarshalJSON(data []byte) error {
 		return err
 	}
 
+	var rawFields map[string]json.RawMessage
+	if err := json.Unmarshal(data, &rawFields); err != nil {
+		return err
+	}
+
+	s.toolTimeoutFieldName = "toolTimeout"
+	if _, ok := rawFields["toolTimeout"]; !ok {
+		if _, legacyOK := rawFields["tool_timeout"]; legacyOK {
+			s.toolTimeoutFieldName = "tool_timeout"
+		}
+	}
+
+	if err := assignLegacyIntAlias(rawFields, "connect_timeout", &s.ConnectTimeout); err != nil {
+		return err
+	}
+	if err := assignLegacyIntAlias(rawFields, "tool_timeout", &s.ToolTimeout); err != nil {
+		return err
+	}
+
 	// Now unmarshal into a map to capture all fields
 	var allFields map[string]interface{}
 	if err := json.Unmarshal(data, &allFields); err != nil {
@@ -180,7 +202,9 @@ func (s *StdinServerConfig) UnmarshalJSON(data []byte) error {
 		"guard-policies":        true,
 		"guard":                 true,
 		"auth":                  true,
+		"connectTimeout":        true,
 		"connect_timeout":       true,
+		"toolTimeout":           true,
 		"tool_timeout":          true,
 	}
 
@@ -195,6 +219,30 @@ func (s *StdinServerConfig) UnmarshalJSON(data []byte) error {
 	return nil
 }
 
+func (s *StdinServerConfig) toolTimeoutField() string {
+	if s.toolTimeoutFieldName != "" {
+		return s.toolTimeoutFieldName
+	}
+	return "toolTimeout"
+}
+
+func assignLegacyIntAlias(rawFields map[string]json.RawMessage, alias string, target **int) error {
+	if *target != nil {
+		return nil
+	}
+	raw, ok := rawFields[alias]
+	if !ok {
+		return nil
+	}
+
+	var value int
+	if err := json.Unmarshal(raw, &value); err != nil {
+		return fmt.Errorf("invalid %s value: %w", alias, err)
+	}
+	*target = &value
+	return nil
+}
+
 // intPtrOrDefault returns the value of the int pointer if not nil, otherwise returns the default value.
 // This helper reduces code duplication when handling optional integer fields with defaults.
 func intPtrOrDefault(ptr *int, defaultValue int) int {
@@ -333,6 +381,9 @@ func convertStdinConfig(stdinCfg *StdinConfig) (*Config, error) {
 		if stdinCfg.Gateway.PayloadDir != "" {
 			cfg.Gateway.PayloadDir = stdinCfg.Gateway.PayloadDir
 		}
+		if stdinCfg.Gateway.PayloadPathPrefix != nil {
+			cfg.Gateway.PayloadPathPrefix = *stdinCfg.Gateway.PayloadPathPrefix
+		}
 		if stdinCfg.Gateway.PayloadSizeThreshold != nil {
 			cfg.Gateway.PayloadSizeThreshold = *stdinCfg.Gateway.PayloadSizeThreshold
 		}
@@ -518,7 +569,7 @@ func buildStdioServerConfig(name string, server *StdinServerConfig) *ServerConfi
 	logStdin.Printf("Server %q: configured stdio container=%s, env_vars=%d, mounts=%d", name, server.Container, len(server.Env), len(server.Mounts))
 	logConfig.Printf("Configured stdio MCP server: name=%s, container=%s", name, server.Container)
 
-	return &ServerConfig{
+	serverCfg := &ServerConfig{
 		Type:                "stdio",
 		Command:             "docker",
 		Args:                args,
@@ -529,6 +580,10 @@ func buildStdioServerConfig(name string, server *StdinServerConfig) *ServerConfi
 		GuardPolicies:       server.GuardPolicies,
 		Guard:               server.Guard,
 	}
+	if server.ToolTimeout != nil {
+		serverCfg.ToolTimeout = *server.ToolTimeout
+	}
+	return serverCfg
 }
 
 // normalizeLocalType normalizes "local" type to "stdio" for backward compatibility.