Skip to content

Commit 7b66842

Browse files
author
LoCoBench Bot
committed
Merge remote-tracking branch 'origin/ralph/gapfill-onboarding'
# Conflicts: # configs/selected_benchmark_tasks.json
2 parents 9d7ef6e + 99c1e9e commit 7b66842

File tree

44 files changed

+4135
-137
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

44 files changed

+4135
-137
lines changed

‎benchmarks/ccb_onboarding/onboard-handoff-001/environment/Dockerfile‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
FROM ubuntu:22.04
2+
3+
# Install dependencies
4+
RUN apt-get update && apt-get install -y \
5+
git \
6+
python3 \
7+
python3-pip \
8+
&& rm -rf /var/lib/apt/lists/*
9+
10+
# Clone Envoy at pinned commit (v1.32.1 stable release)
11+
RUN git clone https://github.com/envoyproxy/envoy.git /workspace \
12+
&& cd /workspace \
13+
&& git checkout v1.32.1
14+
15+
WORKDIR /workspace
16+
17+
# Create logs directory for output
18+
RUN mkdir -p /logs/agent

‎benchmarks/ccb_onboarding/onboard-handoff-001/instruction.md‎

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
# Team Handoff: Envoy ext_authz Filter
2+
3+
You are taking over ownership of the **ext_authz HTTP filter** from a departing team member. Your task is to produce a comprehensive handoff document that will help you (and future maintainers) understand this component.
4+
5+
## Context
6+
7+
The ext_authz filter is one of Envoy's authorization extensions. Your predecessor maintained it for the past year but is moving to a different project. You need to quickly build a working mental model of this component.
8+
9+
## Your Task
10+
11+
Explore the Envoy codebase and produce a handoff document at `/logs/agent/onboarding.md` covering the following sections:
12+
13+
### 1. Purpose
14+
What does the ext_authz filter do? What problems does it solve? When would someone use it?
15+
16+
### 2. Dependencies
17+
- **Upstream dependencies**: What other Envoy components or libraries does ext_authz depend on?
18+
- **Downstream consumers**: What uses or integrates with ext_authz? How is it configured in Envoy's filter chain?
19+
20+
### 3. Key Files
21+
List the most important source files for this filter. For each file, explain its role (e.g., "configuration parsing", "request handling", "gRPC client implementation").
22+
23+
### 4. Failure Modes
24+
What can go wrong? What are common failure scenarios? How does the filter handle errors (e.g., authorization service unavailable, timeout, network failure)?
25+
26+
### 5. Testing
27+
How is this filter tested? Where are the tests located? Are there integration tests, unit tests, or both?
28+
29+
### 6. Debugging
30+
If something goes wrong with ext_authz in production, how would you debug it? What logs, metrics, or traces would you look at?
31+
32+
## Output Format
33+
34+
Write your findings to `/logs/agent/onboarding.md` using the following structure:
35+
36+
```markdown
37+
# ext_authz Filter Handoff Document
38+
39+
## 1. Purpose
40+
[Your findings here]
41+
42+
## 2. Dependencies
43+
### Upstream Dependencies
44+
[What ext_authz depends on]
45+
46+
### Downstream Consumers
47+
[What depends on ext_authz]
48+
49+
## 3. Key Files
50+
- **path/to/file.cc**: [Role/responsibility]
51+
- **path/to/file.h**: [Role/responsibility]
52+
...
53+
54+
## 4. Failure Modes
55+
[Failure scenarios and error handling]
56+
57+
## 5. Testing
58+
[Test locations, test types, how to run tests]
59+
60+
## 6. Debugging
61+
[Debugging strategies, observability]
62+
```
63+
64+
## Guidelines
65+
66+
- Be specific. Reference actual file paths, class names, function names.
67+
- Focus on understanding the **system's behavior**, not just describing the code.
68+
- This is analysis only — do not modify any code.
69+
- Use MCP tools (list_files, nls_search, read_file, find_references) to explore the codebase efficiently.

‎benchmarks/ccb_onboarding/onboard-handoff-001/task.toml‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
[task]
2+
name = "onboard-handoff-001"
3+
category = "team_handoff"
4+
language = "cpp"
5+
difficulty = "hard"
6+
time_limit_sec = 1800
7+
environment = "docker"
8+
9+
[task.metadata]
10+
description = "Team handoff for Envoy ext_authz filter ownership"
11+
repo = "envoyproxy/envoy"
12+
target_component = "ext_authz HTTP filter"
13+
output_path = "/logs/agent/onboarding.md"
14+
15+
[task.metadata.mcp_benefit]
16+
context_complexity = 0.9
17+
cross_file_deps = 0.8
18+
semantic_search_potential = 0.85
19+
tool_chain_weight = 0.7

‎benchmarks/ccb_onboarding/onboard-handoff-001/tests/ground_truth.json‎

Lines changed: 158 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,158 @@
1+
{
2+
"weights": {
3+
"required_findings": 0.25,
4+
"file_references": 0.30,
5+
"causal_chain": 0.30,
6+
"negative_checks": 0.15
7+
},
8+
"required_findings": [
9+
{
10+
"id": "purpose_authz",
11+
"patterns": ["external authorization", "ext_?authz", "authorization service", "delegat(e|ing) auth"],
12+
"weight": 1.0,
13+
"description": "Identifies that ext_authz delegates authorization decisions to an external service",
14+
"section": "purpose"
15+
},
16+
{
17+
"id": "purpose_grpc_http",
18+
"patterns": ["gRPC", "HTTP.*protocol", "protocol.*service"],
19+
"weight": 1.0,
20+
"description": "Mentions that ext_authz supports gRPC or HTTP protocol for authz service",
21+
"section": "purpose"
22+
},
23+
{
24+
"id": "upstream_grpc_async",
25+
"patterns": ["Grpc::AsyncClient", "AsyncClient", "grpc.*client"],
26+
"weight": 1.0,
27+
"description": "Identifies gRPC async client as upstream dependency",
28+
"section": "dependencies"
29+
},
30+
{
31+
"id": "upstream_http_filter_chain",
32+
"patterns": ["http.*filter.*chain", "StreamDecoderFilter", "filter.*interface"],
33+
"weight": 1.0,
34+
"description": "Identifies HTTP filter chain/interface dependency",
35+
"section": "dependencies"
36+
},
37+
{
38+
"id": "failure_mode_timeout",
39+
"patterns": ["timeout", "request.*timeout", "TTL"],
40+
"weight": 1.0,
41+
"description": "Mentions timeout as a failure mode",
42+
"section": "failure_modes"
43+
},
44+
{
45+
"id": "failure_mode_unavailable",
46+
"patterns": ["service.*unavailable", "connection.*fail", "network.*error"],
47+
"weight": 1.0,
48+
"description": "Mentions service unavailability as failure mode",
49+
"section": "failure_modes"
50+
},
51+
{
52+
"id": "testing_integration",
53+
"patterns": ["integration.*test", "ext_authz.*integration", "test/integration"],
54+
"weight": 1.0,
55+
"description": "Identifies integration tests for ext_authz",
56+
"section": "testing"
57+
}
58+
],
59+
"file_references": [
60+
{
61+
"id": "file_ext_authz_h",
62+
"patterns": ["source/extensions/filters/http/ext_authz/ext_authz\\.h"],
63+
"weight": 1.5,
64+
"description": "Main header file for ext_authz filter",
65+
"section": "key_files"
66+
},
67+
{
68+
"id": "file_ext_authz_cc",
69+
"patterns": ["source/extensions/filters/http/ext_authz/ext_authz\\.cc"],
70+
"weight": 1.5,
71+
"description": "Main implementation file for ext_authz filter",
72+
"section": "key_files"
73+
},
74+
{
75+
"id": "file_config",
76+
"patterns": ["source/extensions/filters/http/ext_authz/config\\.(h|cc)", "source/extensions/filters/http/ext_authz/config\\.h", "source/extensions/filters/http/ext_authz/config\\.cc"],
77+
"weight": 1.0,
78+
"description": "Configuration parsing for ext_authz",
79+
"section": "key_files"
80+
},
81+
{
82+
"id": "file_ext_authz_grpc",
83+
"patterns": ["source/extensions/filters/http/ext_authz/ext_authz_grpc"],
84+
"weight": 1.0,
85+
"description": "gRPC client implementation for ext_authz",
86+
"section": "key_files"
87+
},
88+
{
89+
"id": "file_ext_authz_http",
90+
"patterns": ["source/extensions/filters/http/ext_authz/ext_authz_http"],
91+
"weight": 1.0,
92+
"description": "HTTP client implementation for ext_authz",
93+
"section": "key_files"
94+
},
95+
{
96+
"id": "test_integration",
97+
"patterns": ["test/integration/filters/ext_authz.*test\\.cc", "test/integration.*ext_authz"],
98+
"weight": 1.0,
99+
"description": "Integration test files",
100+
"section": "testing"
101+
},
102+
{
103+
"id": "test_unit",
104+
"patterns": ["test/extensions/filters/http/ext_authz/.*test\\.cc", "test.*ext_authz.*test"],
105+
"weight": 1.0,
106+
"description": "Unit test files",
107+
"section": "testing"
108+
}
109+
],
110+
"causal_chain": [
111+
{
112+
"id": "chain_request_flow",
113+
"patterns": ["decodeHeaders", "check", "authorization service"],
114+
"weight": 1.5,
115+
"description": "Describes request flow from filter decoding to authorization service call",
116+
"section": "purpose"
117+
},
118+
{
119+
"id": "chain_failure_handling",
120+
"patterns": ["timeout|failure|error", "fail.*mode", "deny|allow"],
121+
"weight": 1.5,
122+
"description": "Describes failure handling chain: error detection -> failure mode policy -> response generation",
123+
"section": "failure_modes"
124+
},
125+
{
126+
"id": "chain_config_bootstrap",
127+
"patterns": ["bootstrap|envoy\\.yaml|config", "http_filters", "ext_authz"],
128+
"weight": 1.0,
129+
"description": "Describes configuration flow from bootstrap to filter chain to ext_authz config",
130+
"section": "dependencies"
131+
},
132+
{
133+
"id": "chain_authz_response",
134+
"patterns": ["CheckResponse|authorization.*response", "ok|success", "continue"],
135+
"weight": 1.0,
136+
"description": "Describes successful authorization response flow",
137+
"section": "purpose"
138+
}
139+
],
140+
"negative_checks": [
141+
{
142+
"id": "neg_wrong_location",
143+
"patterns": ["source/common/http/ext_authz"],
144+
"weight": 1.0,
145+
"description": "Incorrect location (ext_authz is in extensions/, not common/)",
146+
"section": "key_files",
147+
"should_not_match": true
148+
},
149+
{
150+
"id": "neg_single_protocol",
151+
"patterns": ["ext_authz (only supports|is limited to) (gRPC|HTTP)(?!.*and)"],
152+
"weight": 1.0,
153+
"description": "Incorrectly claims ext_authz only supports one protocol (it supports both gRPC and HTTP)",
154+
"section": "purpose",
155+
"should_not_match": true
156+
}
157+
]
158+
}

0 commit comments

Comments
 (0)