Skip to content

Commit e15e7bb

Browse files
sjarmakclaude
andcommitted
fix: clone all oracle repos locally in MCP-unique task Dockerfiles
All 12 MCP-unique tasks now clone every repo referenced in their oracle into /workspace/, giving baseline equal local access to the codebases that SG_full reaches via Sourcegraph MCP. Previously, oracle repos marked access_mode="mcp_only" were only reachable by the SG_full config, making the baseline comparison trivial (structurally unable to find the answer). Now both configs have the same information available — just through different retrieval mechanisms. Repos added per fixture: - kubernetes-ecosystem: +kubernetes/client-go, +kubernetes/api, +etcd-io/etcd - grafana-observability: +grafana/loki - multi-org-go: +etcd-io/etcd, +grafana/grafana - nodejs-web-stack: +expressjs/express - python-ml-stack: +numpy/numpy, +pandas-dev/pandas, +scipy/scipy Updated: 12 Dockerfiles, 9 instruction.md files, 5 fixture JSONs. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1 parent fa591a4 commit e15e7bb

File tree

25 files changed

+114
-101
lines changed

25 files changed

+114
-101
lines changed

benchmarks/ccb_mcp_crossorg/ccx-crossorg-061/environment/Dockerfile

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,8 @@ WORKDIR /workspace
1515

1616
# Clone local checkout repos (baseline config: agent has local access to these)
1717
RUN git clone --depth 1 --branch v1.32.0 https://github.com/kubernetes/kubernetes /workspace/kubernetes
18+
RUN git clone --depth 1 --branch v11.4.0 https://github.com/grafana/grafana /workspace/grafana
19+
RUN git clone --depth 1 --branch v3.5.17 https://github.com/etcd-io/etcd /workspace/etcd
1820

1921
# Initialize git identity for agent commits
2022
RUN git config --global user.email "agent@example.com" && \

benchmarks/ccb_mcp_crossorg/ccx-crossorg-061/instruction.md

Lines changed: 4 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -25,11 +25,10 @@ by projects from entirely different organizations.
2525

2626
## Available Resources
2727

28-
The local `/workspace/` directory contains: kubernetes/kubernetes.
29-
30-
**Note:** Additional repositories are accessible via Sourcegraph MCP tools:
31-
- `etcd-io/etcd` (distributed-kv-store)
32-
- `grafana/grafana` (observability-platform)
28+
The local `/workspace/` directory contains all repositories:
29+
- `kubernetes/kubernetes` at v1.32.0 → `/workspace/kubernetes`
30+
- `etcd-io/etcd` at v3.5.17 → `/workspace/etcd`
31+
- `grafana/grafana` at v11.4.0 → `/workspace/grafana`
3332

3433
## Output Format
3534

benchmarks/ccb_mcp_crossorg/ccx-crossorg-066/environment/Dockerfile

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,8 @@ WORKDIR /workspace
1515

1616
# Clone local checkout repos (baseline config: agent has local access to these)
1717
RUN git clone --depth 1 --branch v1.32.0 https://github.com/kubernetes/kubernetes /workspace/kubernetes
18+
RUN git clone --depth 1 --branch v11.4.0 https://github.com/grafana/grafana /workspace/grafana
19+
RUN git clone --depth 1 --branch v3.5.17 https://github.com/etcd-io/etcd /workspace/etcd
1820

1921
# Initialize git identity for agent commits
2022
RUN git config --global user.email "agent@example.com" && \

benchmarks/ccb_mcp_crossorg/ccx-crossorg-066/instruction.md

Lines changed: 4 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -26,11 +26,10 @@ maintained.
2626

2727
## Available Resources
2828

29-
The local `/workspace/` directory contains: kubernetes/kubernetes.
30-
31-
**Note:** Additional repositories are accessible via Sourcegraph MCP tools:
32-
- `etcd-io/etcd` (distributed-kv-store)
33-
- `grafana/grafana` (observability-platform)
29+
The local `/workspace/` directory contains all repositories:
30+
- `kubernetes/kubernetes` at v1.32.0 → `/workspace/kubernetes`
31+
- `etcd-io/etcd` at v3.5.17 → `/workspace/etcd`
32+
- `grafana/grafana` at v11.4.0 → `/workspace/grafana`
3433

3534
## Output Format
3635

benchmarks/ccb_mcp_crossrepo_tracing/ccx-config-trace-010/environment/Dockerfile

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,8 @@ WORKDIR /workspace
1515

1616
# Clone local checkout repos (baseline config: agent has local access to these)
1717
RUN git clone --depth 1 --branch v1.32.0 https://github.com/kubernetes/kubernetes /workspace/kubernetes
18+
RUN git clone https://github.com/kubernetes/client-go /workspace/client-go --no-checkout && \
19+
cd /workspace/client-go && git fetch --depth 1 origin 8020fc4fcf89965904a5f43689f169d6e01d1e80 && git checkout FETCH_HEAD
1820

1921
# Initialize git identity for agent commits
2022
RUN git config --global user.email "agent@example.com" && \

benchmarks/ccb_mcp_crossrepo_tracing/ccx-config-trace-010/instruction.md

Lines changed: 5 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -26,12 +26,11 @@ directories, but the authoritative source lives in separate repositories accessi
2626

2727
## Available Resources
2828

29-
The local `/workspace/` directory contains: kubernetes/kubernetes.
30-
31-
**Note:** Additional repositories are accessible via Sourcegraph MCP tools:
32-
- `sg-benchmarks/kubernetes-client-go` (go-client-library)
33-
- `sg-benchmarks/kubernetes-api` (api-type-definitions)
34-
- `etcd-io/etcd` (distributed-kv-store)
29+
The local `/workspace/` directory contains all repositories:
30+
- `kubernetes/kubernetes` at v1.32.0 → `/workspace/kubernetes`
31+
- `kubernetes/client-go` at 8020fc4 → `/workspace/client-go`
32+
- `kubernetes/api` at fa23dd3 → `/workspace/api`
33+
- `etcd-io/etcd` at v3.5.17 → `/workspace/etcd`
3534

3635
## Output Format
3736

benchmarks/ccb_mcp_crossrepo_tracing/ccx-dep-trace-001/environment/Dockerfile

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,8 @@ WORKDIR /workspace
1515

1616
# Clone local checkout repos (baseline config: agent has local access to these)
1717
RUN git clone --depth 1 --branch v1.32.0 https://github.com/kubernetes/kubernetes /workspace/kubernetes
18+
RUN git clone https://github.com/kubernetes/client-go /workspace/client-go --no-checkout && \
19+
cd /workspace/client-go && git fetch --depth 1 origin 8020fc4fcf89965904a5f43689f169d6e01d1e80 && git checkout FETCH_HEAD
1820

1921
# Initialize git identity for agent commits
2022
RUN git config --global user.email "agent@example.com" && \

benchmarks/ccb_mcp_crossrepo_tracing/ccx-dep-trace-001/instruction.md

Lines changed: 3 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -20,12 +20,9 @@ that directly imports it (not just subpackages) will be affected by a breaking A
2020

2121
## Available Resources
2222

23-
The local `/workspace/` directory contains: kubernetes/kubernetes.
24-
25-
**Note:** Additional repositories are accessible via Sourcegraph MCP tools:
26-
- `sg-benchmarks/kubernetes-client-go` (go-client-library)
27-
- `sg-benchmarks/kubernetes-api` (api-type-definitions)
28-
- `etcd-io/etcd` (distributed-kv-store)
23+
The local `/workspace/` directory contains all repositories:
24+
- `kubernetes/kubernetes` at v1.32.0 → `/workspace/kubernetes`
25+
- `kubernetes/client-go` at 8020fc4 → `/workspace/client-go`
2926

3027
## Output Format
3128

benchmarks/ccb_mcp_crossrepo_tracing/ccx-dep-trace-004/environment/Dockerfile

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,8 @@ WORKDIR /workspace
1515

1616
# Clone local checkout repos (baseline config: agent has local access to these)
1717
RUN git clone --depth 1 --branch v11.4.0 https://github.com/grafana/grafana /workspace/grafana
18+
RUN git clone https://github.com/grafana/loki /workspace/loki --no-checkout && \
19+
cd /workspace/loki && git fetch --depth 1 origin a3af38d4da899032d3ee46a30932d072d37e1b9c && git checkout FETCH_HEAD
1820

1921
# Initialize git identity for agent commits
2022
RUN git config --global user.email "agent@example.com" && \

benchmarks/ccb_mcp_crossrepo_tracing/ccx-dep-trace-004/instruction.md

Lines changed: 3 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -22,11 +22,9 @@ adding observability, or extending the query pipeline.
2222

2323
## Available Resources
2424

25-
The local `/workspace/` directory contains: grafana/grafana.
26-
27-
**Note:** Additional repositories are accessible via Sourcegraph MCP tools:
28-
- `sg-benchmarks/grafana-loki` (log-aggregation)
29-
- `sg-benchmarks/grafana-mimir` (metrics-backend)
25+
The local `/workspace/` directory contains all repositories:
26+
- `grafana/grafana` at v11.4.0 → `/workspace/grafana`
27+
- `grafana/loki` at a3af38d → `/workspace/loki`
3028

3129
## Output Format
3230

0 commit comments

Comments
 (0)