Skip to content

DESIGN.md

Latest commit

 

History

History
333 lines (280 loc) · 17 KB

DESIGN.md

File metadata and controls

333 lines (280 loc) · 17 KB
Raw

Felix — Architecture & Design Index

Felix is Calico's per-node agent. It watches the datastore for configuration, computes per-endpoint state through its calculation graph, and programs one or more dataplanes (BPF, iptables, nftables, Windows) to enforce policy and route traffic.

This document has two parts:

  1. Architecture overview — the shape of Felix as a whole. Read this first; it sets the context every sub-design depends on.
  2. Sub-design index — pointers to per-topic design docs under felix/design/ with a path-to-doc mapping. Invariants and review criteria live in the sub-designs, not here.

Operational guidance (how to build, test, debug, use tooling) is separate and lives in felix/CLAUDE.md.

1. Architecture overview

Data flow

Datastore syncer
   → AsyncCalcGraph
   → CalcGraph (dispatcher + calculation nodes)
   → EventSequencer
   → InternalDataplane
   → dataplane-specific managers → kernel objects
  1. Datastore syncer receives updates from the Calico datastore (Kubernetes CRDs or etcd).
  2. CalcGraph processes them through a graph of calculation nodes — policy resolution, route resolution, IP set indexing, service-endpoint synthesis, VTEP calculation, encapsulation mode.
  3. EventSequencer buffers and coalesces the outputs and flushes them in dependency-safe order (e.g. IP sets before the policies that reference them).
  4. The dataplane driver (dataplane/driver.go) selects a dataplane implementation based on configuration.
  5. InternalDataplane (Linux; dataplane/linux/int_dataplane.go) or the Windows equivalent (dataplane/windows/win_dataplane.go) fans updates out to managers, each owning a slice of dataplane state.

Key orientation files: daemon/daemon.go, calc/calc_graph.go, dataplane/linux/int_dataplane.go, dataplane/driver.go.

Calculation graph (the "brain")

The calc graph in felix/calc/ is an event-processing pipeline that transforms raw datastore updates into dataplane-ready instructions. Key calculation nodes:

Node Role
AllUpdDispatcher Fans out datastore updates by resource type to downstream nodes
ActiveRulesCalculator Tracks which policies/profiles are active based on endpoint labels
RuleScanner Scans rules for selector references; feeds the IP-set index
PolicyResolver Resolves per-endpoint policy ordering (tiers, priorities)
L3RouteResolver Computes routes from IP pools, workload endpoints, and host IPs
VXLANResolver Computes VTEP (VXLAN tunnel endpoint) entries
EncapsulationResolver Determines encapsulation mode from IP-pool config
IstioCalculator Marks WEPs in the Istio ambient mesh (see bpf-observability "Istio ambient mode integration")

PipelineCallbacks (calc/calc_graph.go) is the composite interface the graph emits through. EventSequencer (calc/event_sequencer.go) is the primary implementation — it buffers updates in pending* maps/sets and flushes via Flush() in dependency-safe order, coalescing rapid updates so the dataplane only sees the final state.

Full invariants and per-node review notes — the node contract, the upstream syncer contract, inter-node ordering, the label indexes, the EventSequencer flush order, and the calc-graph FV testing framework — are in calc-graph.md.

Dataplane: managers and drivers

The Linux dataplane is a single codebase (InternalDataplane, dataplane/linux/) switched between iptables, nftables and eBPF modes; all three share the layering, event loop and restart-and-resync doctrine sketched here. Windows (dataplane/windows/) is a separate dataplane of the same overall shape.

The dataplane is split into two layers:

  • Managers (dataplane/linux/*_mgr.go) take the calc graph's Calico-internal desired state (local WEPs, abstract policy rules, resolved IP sets) and convert it into this dataplane's terms — iptables/nftables rules, IP-set contents and routes; BPF map entries; or Windows HNS policy. This conversion is the manager layer's defining job.
  • Drivers (iptables/, nftables/, ipsets/, routetable/, routerule/, vxlanfdb/) bring actual kernel state into sync with the desired state — read back what's there, compute a minimal delta, apply it. Some managers reconcile directly; others delegate to a driver and stay declarative.

Beyond convert and reconcile, the dataplane also reacts to expected kernel changes (interfaces coming and going, via ifacemonitor/), detects unexpected drift (periodic full resyncs), and reports programming status back to the datastore and to the CNI plugin.

Each manager implements a two-method Manager interface — OnUpdate to receive desired state cheaply, CompleteDeferredWork to program the kernel during a throttled apply() cycle. Key managers:

Manager Handles
endpointManager / bpfEndpointManager Workload/host endpoint programming
policyManager / rawEgressPolicyManager Policy chain/rule generation
ipsetsManager IP-set synchronisation
noEncapManager / vxlanManager Route management for encap modes
ipipManager IPIP tunnel interfaces
wireguardManager WireGuard tunnel setup
masqManager IP masquerade rules
hostIPManager Host-IP tracking
floatingIPManager Floating-IP NAT
dscpManager DSCP marking
serviceLoopManager Service-loop prevention
failsafeMgr BPF failsafe port programming

IPv4 and IPv6 each get their own manager instances; dataplane/driver.go is the factory that constructs and wires the dataplane.

The Manager contract and extended interfaces, the apply() ordering, the restart/resync (mark-and-sweep) doctrine, the *tables Table abstraction, IP sets, and the calc-graph→dataplane proto contract are all detailed in dataplane.md. eBPF mode reuses this architecture; its mode-specific managers, maps and packet path are in the bpf-* family.

Dataplane backends

Felix runs against one dataplane at a time (selected by BPFEnabled and NFTablesMode config):

  • BPF — eBPF programs on TC and cgroup hooks, BPF maps for NAT / conntrack / policy. See the bpf-* sub-designs under design/, starting with bpf-overview.md.
  • iptables — legacy netfilter via iptables-restore. Code in felix/iptables/.
  • nftables — modern netfilter via the nftables API. Code in felix/nftables/.
  • Windows (HNS/HCN) — separate dataplane in dataplane/windows/.

NFTablesMode=Auto resolves by following the detected kube-proxy mode: an nftables-mode kube-proxy selects the nftables dataplane. That signal is about coexistence — Felix must use the same netfilter generation as kube-proxy — not host capability, so it must not be replaced by a capability probe on cluster hosts. See useNftables() in dataplane/linux/int_dataplane.go. The per-host escape hatch is NFTablesMode=Disabled/Enabled set locally (env var or config file), which overrides any datastore-inherited value.

The iptables and nftables backends share a common rule- generation layer in felix/rules/ and a common table-abstraction interface in felix/generictables/. Backend-neutral rule generation: dispatch.go (per-endpoint dispatch chains), policy.go (policy chain generation), endpoints.go (endpoint chain setup), static.go (boilerplate filter/NAT/mangle chains), nat.go. A PR adding policy semantics usually touches felix/rules/ and needs matching changes on both backends.

Shared networking subsystems

Used by more than one dataplane:

Package Purpose
routetable/ Linux route-table management via netlink
routerule/ Policy-based routing rules
vxlanfdb/ VXLAN forwarding-database management
wireguard/ WireGuard tunnel setup and key management
ifacemonitor/ Interface state monitoring (link up/down, address changes)
nfnetlink/ Conntrack and nflog via netfilter netlink
netlinkshim/ Netlink abstraction layer for testing and portability

The route-sync drivers (routetable/, routerule/, vxlanfdb/) fit the dataplane manager/driver architecture and resync doctrine covered in dataplane.md; their deeper netlink-level design (resync grace periods, conntrack cleanup on IP moves) is reserved for a future route-sync.md sub-design. flow-logs-collector.md is likewise still to be written.

Cross-component designs Felix takes part in

Some subsystems are split between Felix and another component, so their design lives at the repo level rather than under felix/design/:

Design What it covers in Felix
design/cluster-route-programming/DESIGN.md Whether Felix or confd/BIRD programs the routes to workloads on other nodes, per encapsulation type. Covers ipipManager, noEncapManager, EncapsulationResolver.NoEncapNeeded, and the ProgramClusterRoutes config parameter.
design/ipam/DESIGN.md Felix is a read-only consumer of IPAM state (IPAM blocks feed the L3RouteResolver).

2. Sub-design index

Per-topic design docs under felix/design/. Each is the authoritative source for its area's architecture, invariants, and review notes.

A PR that touches files across multiple "applies to" scopes must load every matching sub-design before acting. The applies to column is the authoritative mapping from source path to design doc.

The bpf-* rows form a single sub-design family for the BPF dataplane, deliberately split so a PR touching one area pulls only the relevant knowledge. The bpf-overview umbrella row is the always-pulled foundation (packet-path mental model, fast-path cost rule, cross-cutting review notes); the others have tight applyTo globs scoped to their topic. Load each of them either when you touch a matched file or when you're working on the related topic — the globs cover the common cases, but a change in a central file (e.g. tc.c, bpf.h) may legitimately need a sub-design even if the immediate edit site doesn't match its glob narrowly, and conversely a PR description that says "this fixes the conntrack scanner" should pull bpf-conntrack-flowstate.md even if the edit happens to land in code paths the glob doesn't list. Other sub-designs should split the same way once they grow large enough to bloat AI-tool context.

Topic Applies to Status
bpf-overview felix/bpf/**, felix/bpf-gpl/**, felix/dataplane/linux/bpf_*.go, felix/dataplane/linux/vxlan_mgr.go (umbrella — pulled by every BPF change) ✅ exists
bpf-tc-programs felix/bpf-gpl/tc.c, tc_preamble.c, xdp_preamble.c, jump.h, bpf.h, globals.h, types.h, felix/bpf/hook/**, felix/bpf/tc/**, felix/bpf/jump/**, felix/bpf/ifstate/** ✅ exists
bpf-xdp felix/bpf-gpl/xdp.c, xdp_preamble.c, metadata.h, felix/bpf/xdp/** ✅ exists
bpf-services felix/bpf/proxy/**, felix/bpf/nat/**, felix/bpf/consistenthash/**, felix/bpf-gpl/connect*.{c,h}, nat*.h, nat_lookup.h, maglev.h, ctlb*.h, sendrecv.h, felix/dataplane/linux/bpf_ep_mgr.go ✅ exists
bpf-host-networking felix/dataplane/linux/bpf_ep_mgr.go, dataplanedefs/dataplane_defs.go, felix/bpf-gpl/fib_co_re.h ✅ exists
bpf-conntrack-flowstate felix/bpf/conntrack/**, felix/bpf-gpl/conntrack*.{c,h}, rpf.h, felix/bpf/allowsources/**, felix/rules/static.go ✅ exists
bpf-encap-fragments-icmp felix/bpf/ipfrags/**, felix/bpf-gpl/ip_v4_fragment.h, tc_ip_frag.c, icmp*.h, fib*.h, felix/bpf/routes/**, felix/dataplane/linux/vxlan_mgr.go ✅ exists
bpf-observability felix/bpf/filter/**, events/**, ringbuf/**, qos/**, felix/bpf-gpl/log.h, events*.h, qos.h, ringbuf.h ✅ exists
bpf-tests felix/bpf/ut/**, felix/fv/bpf_*_test.go ✅ exists
dataplane felix/dataplane/linux/** (the shared loop/manager/resync architecture, all modes — BPF-specific files here are also matched by the bpf-* rows, intentionally), felix/iptables/**, felix/nftables/**, felix/generictables/**, felix/ipsets/**, felix/markbits/**, felix/rules/**; also the manager/driver architecture & resync doctrine for felix/routetable/**, felix/routerule/**, felix/vxlanfdb/** ✅ exists
calc-graph felix/calc/**, felix/labelindex/**, felix/dispatcher/** ✅ exists
route-sync (deep netlink design only) felix/routetable/**, felix/routerule/**, felix/vxlanfdb/** — architecture covered by dataplane.md; this row reserved for the deeper netlink-level resync design not yet written
flow-logs-collector felix/collector/** not yet written
config-engine felix/config/** not yet written
windows-dataplane felix/dataplane/windows/** not yet written

A missing sub-design means the area's invariants have not been written down yet — not that the area has no constraints. Treat absence as "read the code and ask"; do not assume anything goes.

3. For coding agents and reviewers

  • Follow links. Every sub-design may reference sibling sub-designs, .github/instructions/*.instructions.md files, code, or external references. Load them. A design is a graph, not a single node.
  • Load what applies — by path or by topic. The applies to globs above are the path-based trigger: if a PR touches both BPF and route-sync code, both sub-designs are needed. The topic of the change matters too — a PR described as "fixing the conntrack scanner" should pull bpf-conntrack-flowstate.md even if the edit happens to land only in a central file the glob covers under a broader umbrella. When in doubt, pull the topic-relevant sub-design.
  • Review notes are the checklist. Each sub-design embeds per-section review notes describing the invariants a PR must respect. At write-time, respect them; at review-time, apply them.
  • Update rule. A change to how Felix works in a given area must update the relevant file under felix/design/ in the same PR — typically the sub-design covering the area. This index (felix/DESIGN.md) is also updated when the sub-design table, a applies to scope, or §1's architecture overview changes. Exemptions: (a) a bug fix that restores behaviour the doc already describes, (b) a mechanical refactor with no observable change, (c) comment or log-message edits, (d) dependency bumps. If in doubt, update. The path-scoped .github/instructions/*.instructions.md files wire this rule into Copilot's automated review.

4. Adding a new sub-design

When a topic above graduates from not yet written to a real doc:

  1. Create felix/design/<topic>.md. Follow the shape of an existing sub-design (e.g. the BPF family): narrative prose, architecture, per-section review notes at the end of each section, and a "keep this in sync" tail.
  2. Update the sub-design index above: replace not yet written with a link to the new file and the ✅ exists marker.
  3. Move any orientation content that belongs to the new sub-design out of this file into the new doc (most of the §1 architecture overview is cross-cutting and stays here).
  4. Create a matching .github/instructions/<topic>.instructions.md with the applyTo globs from the table above plus a pointer to the new design doc. Keep it thin — see any of the bpf-*.instructions.md files as the template.

When a sub-design grows too big, split it

A single sub-design that grows large enough to bloat AI-tool context for narrow PRs (in practice: ~2000+ lines, ~25k+ tokens) should split into a family of focused files the way the BPF dataplane did:

  • A short always-pulled <topic>-overview.md containing the mental model, the cost / discipline rules, and cross-cutting review notes.
  • Per-area files (<topic>-<area>.md) covering specific feature groupings, each with a tight applyTo glob in its own .github/instructions/<topic>-<area>.instructions.md.

The multi-file family appears in this index as multiple rows, all sharing the topic prefix. A PR touching multiple areas matches multiple instruction files; only the union of the matched sub-designs loads, not the whole family.