enhance: add IP aggregation option for blocklists#122
Merged
Conversation
Add `aggregate: true` configuration option that merges individual IP
decisions into minimal CIDR blocks, reducing blocklist size.
Feature:
- New `aggregate` field in blocklist config works with any format
- Adjacent IPs are merged into larger CIDR ranges (e.g., .0 + .1 → /31)
- Overlapping prefixes are deduplicated (larger block absorbs smaller)
- Supports both IPv4 and IPv6 with optimized bit operations
Performance:
- Aggregation is pre-computed when decisions are added/deleted from LAPI
- HTTP requests read from cached aggregated view (no per-request cost)
- Only enabled if at least one blocklist has aggregate: true
- Uses RWMutex for concurrent read access during updates
Filter behavior:
- ipv4only/ipv6only: work normally on aggregated results
- origin/supported_decisions_types: skipped for aggregated results
(aggregated ranges may contain IPs from multiple origins/types)
Example config:
blocklists:
- format: plain_text
endpoint: /security/blocklist
aggregate: true
Files:
- pkg/aggregate/aggregate.go: sort-and-merge algorithm
- pkg/aggregate/aggregate_test.go: comprehensive tests
- pkg/registry/registry.go: stores pre-computed aggregated view
- pkg/cfg/config.go: Aggregate bool field
- pkg/server/server.go: passes aggregate flag to registry
- cmd/root.go: enables aggregation before LAPI connection
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
LaurenceJJones
force-pushed
the
feature/ip-aggregation
branch
from
e6cbbe7 to
ac43787
Compare
There was a problem hiding this comment.
Pull request overview
This pull request adds an IP aggregation feature that merges individual IP decisions into minimal CIDR blocks to reduce blocklist sizes. Aggregation is configured per-blocklist via an aggregate: true option and pre-computed when decisions are added/deleted to avoid per-request overhead.
Changes:
- New
aggregatepackage implementing sort-and-merge algorithm for IPv4/IPv6 CIDR optimization - Registry modifications to store pre-computed aggregated decisions with RWMutex for concurrent access
- Per-blocklist aggregate configuration flag passed through middleware chain
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
| pkg/aggregate/aggregate.go | Core aggregation algorithm with CIDR merging and deduplication logic |
| pkg/aggregate/aggregate_test.go | Comprehensive unit tests for aggregation functionality |
| pkg/registry/registry.go | Added aggregation storage, locking, and dual decision views (aggregated/non-aggregated) |
| pkg/server/server.go | Modified middleware to accept and pass aggregate flag per blocklist |
| pkg/cfg/config.go | Added Aggregate boolean field to BlockListConfig |
| cmd/root.go | Enables aggregation on registry if any blocklist requires it |
| .golangci.yml | Increased complexity thresholds to accommodate new code |
| go.mod | Added testify dependency for testing |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Prevent potential panic if a decision has nil Value pointer. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add ProcessDecisions method that handles both additions and deletions with a single lock and only recomputes aggregation once, instead of potentially twice when both new and deleted decisions arrive together. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Always output CIDR notation (e.g., 10.0.0.1/32) with 'range' scope
for all aggregated decisions, preventing formatter issues
- Add placeholder values for Scenario ('aggregated') and Duration ('24h')
to prevent panics in formatters that require these fields
- Add defensive nil check for Scenario in F5 formatter
- Performance optimizations: pre-allocate slices, reuse string constants,
use IndexByte instead of Contains for CIDR detection
- Replace custom stringPtr with ptr.Of() from go-cs-lib/ptr
Fixes issues where:
- F5 formatter would panic on nil Scenario
- Juniper formatter would generate invalid output like 10.0.0.1/32/32
- Mikrotik templates would show <nil> for Scenario/Duration
blotus
requested changes
Jan 27, 2026
blotus
approved these changes
Jan 27, 2026
LaurenceJJones
added a commit
to crowdsecurity/crowdsec-docs
that referenced
this pull request
Jan 27, 2026
* feat: clean up bl-mirror docs and add aggregate option See crowdsecurity/cs-blocklist-mirror#122 * enhance: add other 0.0.6 option
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add
aggregate: trueconfiguration option that merges individual IP decisions into minimal CIDR blocks, reducing blocklist size.Feature:
aggregatefield in blocklist config works with any formatPerformance:
Example config:
Files: