fix(containers): tolerate OTel bootstrap conflicts when backfilling old releases (#6063)

## What

Adds an `OTEL_BEST_EFFORT` build arg. When set to `1`, a failure of
`opentelemetry-bootstrap -a install` is logged and tolerated instead of
failing the image build. The base
`opentelemetry-distro`/`opentelemetry-exporter-otlp` install remains
required either way.

The publish workflow sets `OTEL_BEST_EFFORT=1` **only** when bundling a
non-`ogx` package (a pre-rename backfill, e.g. `llama-stack`). `ogx`
release images keep the strict behavior.

## Why

`opentelemetry-bootstrap -a install` auto-selects the *latest*
per-library instrumentation packages. When backfilling an older release,
those conflict with the release's pinned deps and the build fails:

```
RuntimeError: Dependency conflict found: opentelemetry-instrumentation-openai-v2 2.4b0
  requires opentelemetry-util-genai>=0.4b0.dev, but you have opentelemetry-util-genai 0.2b0
ERROR: ... "opentelemetry-bootstrap -a install" ... exit code: 1
```

Hit while backfilling `llama-stack` 0.5.2 images. The per-library
bootstrap is inherently best-effort; the core OTel exporter/distro (the
part needed for OTLP export) still installs.

## Effect

- **`ogx` release builds:** unchanged — bootstrap stays strict (fatal on
conflict).
- **Pre-rename backfills (`package_name != ogx`):** if the latest
auto-instrumentation packages can't be installed cleanly, the image is
built **without that per-library auto-instrumentation**, but with the
base OTel exporter/distro intact. `OTEL_*`-triggered
`opentelemetry-instrument` still works; some library-specific spans may
be missing.

## Test plan

- `pypi.yml` validated as YAML.
- Verified derivation (`ogx` → strict, `llama-stack` → best-effort) and
the bootstrap-failure branch logic.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/ogx-ai/ogx/pull/6063"
target="_blank">
  <picture>
<source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-open-in-devin-review-dark.svg?v=1">
<img
src="https://static.devin.ai/assets/gh-open-in-devin-review-light.svg?v=1"
alt="Open in Devin Review">
  </picture>
</a>
<!-- devin-review-badge-end -->

Signed-off-by: Charlie Doern <cdoern@redhat.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

Author: cdoern

.github/workflows/pypi.yml

@@ -973,9 +973,15 @@ jobs:
           # suffix ("llama-stack" -> "llama", "ogx" -> "ogx").
           PACKAGE_NAME="${{ inputs.package_name || 'ogx' }}"
           CLI_NAME="${PACKAGE_NAME%-stack}"
+          # Backfilling a non-ogx (pre-rename) package pins old dependencies that
+          # can conflict with the latest OTel auto-instrumentation packages, so
+          # make the bootstrap best-effort for those builds. The ogx path stays
+          # strict.
+          if [ "$PACKAGE_NAME" == "ogx" ]; then OTEL_BEST_EFFORT=""; else OTEL_BEST_EFFORT="1"; fi
           {
             echo "package_name=${PACKAGE_NAME}"
             echo "cli_name=${CLI_NAME}"
+            echo "otel_best_effort=${OTEL_BEST_EFFORT}"
           } >> "$GITHUB_OUTPUT"
 
           if [ "$EVENT" == "release" ] || [ "$DRY_RUN" == "off" ]; then
@@ -1032,4 +1038,5 @@ jobs:
             INSTALL_MODE=${{ steps.meta.outputs.install_mode }}
             PACKAGE_NAME=${{ steps.meta.outputs.package_name }}
             CLI_NAME=${{ steps.meta.outputs.cli_name }}
+            OTEL_BEST_EFFORT=${{ steps.meta.outputs.otel_best_effort }}
             ${{ steps.meta.outputs.version_arg }}

containers/Containerfile

@@ -31,6 +31,10 @@ ARG DISTRO_NAME="starter"
 ARG PACKAGE_NAME="ogx"
 # CLI binary the package provides (e.g. "ogx", or "llama" for llama-stack).
 ARG CLI_NAME="ogx"
+# Tolerate failures of the OpenTelemetry per-library bootstrap. Used when
+# backfilling older releases whose pinned deps conflict with the latest
+# auto-instrumentation packages. The default ("") keeps the bootstrap strict.
+ARG OTEL_BEST_EFFORT=""
 ARG RUN_CONFIG_PATH=""
 ARG UV_HTTP_TIMEOUT=500
 ARG UV_EXTRA_INDEX_URL=""
@@ -153,10 +157,22 @@ RUN set -eux; \
         printf '%s\n' "$deps" | xargs -L1 uv pip install --no-cache; \
     fi
 
-# Install OpenTelemetry auto-instrumentation support
+# Install OpenTelemetry auto-instrumentation support.
+# The base distro/exporter install is required. The per-library bootstrap
+# (opentelemetry-bootstrap -a install) selects the latest instrumentation
+# packages, which can conflict with the pinned dependencies of older releases
+# being backfilled. With OTEL_BEST_EFFORT=1 a bootstrap failure is logged and
+# tolerated instead of failing the build.
 RUN set -eux; \
     pip install --no-cache opentelemetry-distro opentelemetry-exporter-otlp; \
-    opentelemetry-bootstrap -a install
+    if ! opentelemetry-bootstrap -a install; then \
+        if [ "$OTEL_BEST_EFFORT" = "1" ]; then \
+            echo "opentelemetry-bootstrap failed; continuing without full auto-instrumentation (OTEL_BEST_EFFORT=1)" >&2; \
+        else \
+            echo "opentelemetry-bootstrap failed" >&2; \
+            exit 1; \
+        fi; \
+    fi
 
 # Pre-cache tiktoken cl100k_base encoding at build time so the server never
 # attempts a runtime download to openaipublic.blob.core.windows.net.