[Broker/NATS] Follow-ups after Option D (chart-vendored NATS): decisions + deferred work

[leecalcote]

Tracking issue for the open decisions and deferred work after adopting Option D — the operator now consumes the official NATS Helm chart's vendored manifests (build-time helm template → go:embed → SSA), with token-via-Secret auth and no committed JWT.

• Proposal: docs: proposal — consume the official NATS project instead of hand-authoring the broker #799 (docs/proposals/broker-nats-direct-consumption.md)
• Implementation: feat(broker): consume the official NATS Helm chart (Option D) — drop hand-authored NATS + committed JWT #800 (stacked on feat: networking-capable BrokerSpec + declarative service reconfiguration (WS-2, WS-4, WS-5) #798 → feat: SSA reconcile, watches, pure non-blocking endpoint (WS-3, WS-6, WS-7) #797)
• Modernization epic: [WS-1] SDK & scaffolding refresh: Kubebuilder v4 layout, current operator-sdk, OLM bundle #783

Decisions needed from maintainers

• 1. Default Service type. The chart renders ClusterIP; the overlay currently preserves the historical LoadBalancer default when spec.service.type is unset, to avoid an unflagged behavioral change. Flip the default to ClusterIP (works on every cluster; the plan's recommendation)? If so, Meshery Server must set spec.service.type explicitly for any environment that relied on the LB default, and it needs a release note. (Proposal §7 Q1, plan §8.)
• 2. Broker CRD longevity. Is the Broker CRD a permanent part of the Meshery API, or is eventual Option C (Meshery-Server-driven Helm install, deprecate the CRD) on the table? Gates how much further to invest in the thin-CR wrapper. (Proposal §7 Q5.)
• 3. Chart pin & refresh cadence. Chart is pinned to NATS_CHART_VERSION = 2.14.2 in the Makefile; a CI drift gate (.github/workflows/nats-chart-drift.yml) fails if pkg/broker/manifests/nats.gen.yaml drifts from a fresh make nats-manifests. Who owns the periodic bump + drift-gate review, and on what cadence? (Proposal §7 Q6.)
• 4. Service/release naming. The vendored objects use the fixed chart release name meshery-nats (single-broker-per-namespace assumption). Keep it, or adopt chart-default <release> naming with an alias? (Proposal §7 Q7.)

Deferred work (scoped out of Option D — you chose core pub/sub + token auth)

• Verify MeshSync↔broker connectivity end-to-end in a full Meshery deployment. The standalone integration-tests only assert "deployment available"; MeshSync (distroless, silent standalone) doesn't reach its NATS-connect path there. The operator provides the correct nats://<token>@host:port URL (independently verified to authenticate + publish via the nats CLI, same nats.go client MeshSync uses), but a real Meshery run should confirm MeshSync authenticates and publishes.
• TLS via cert-manager for the NATS listeners (chart *.tls values) and for the operator's own metrics endpoint (cert-manager-issued cert instead of the current self-signed SecureServing). The cert-manager API is already on v1.
• JetStream (chart config.jetstream.enabled + file-store PVC) — only if MeshSync ever needs persistent streaming. Add BrokerSpec.JetStream (additive) when needed.
• NACK (jetstream.nats.io CRDs: Stream/Consumer/KeyValue) — only as a follow-on to JetStream, to declaratively manage JetStream objects on the running server.
• NATS Prometheus exporter (promExporter.enabled) for broker metrics, if broker observability is wanted.

Notes

• Helm stays a build-time-only tool; there is no Helm/NATS runtime dependency in go.mod (WS-6 preserved).
• Meshery Server contracts are unchanged by Option D: the v1alpha1 CRDs, the typed client, and Broker.Status.Endpoint all behave as before.

[CodeAhmedJamil]

Decision (1) — default Service type — is resolved as ClusterIP (maintainer call: breaking change sanctioned). Landed on the #800 branch: an unset spec.service.type now keeps the chart's ClusterIP default instead of implicitly acquiring a LoadBalancer; explicit LoadBalancer/NodePort behave as before, and CRD docs were regenerated.

Also landed in the same push, relevant to the deferred items list:

• NATS pods hardened (non-root 65532, RuntimeDefault seccomp, dropped capabilities, read-only rootfs, full requests/limits on both containers) — the Trivy findings on the vendored manifests are fixed at the source.
• MeshSyncSpec.version is now wired (was dead code): maps to meshery/meshsync:<tag>, moving -latest tags keep PullAlways, pinned tags use IfNotPresent so kind side-loading works.
• The e2e now asserts a live MeshSync connection on the NATS server (connz), which exposed a real cross-repo break: meshsync's startup ping derives its host by splitting BROKER_URL on :", so any scheme-prefixed URL makes it ping http://nats:8222/connz` forever, silently, at debug level. Fix in progress in meshery/meshsync (proper URL parsing + bounded retry + /healthz).