* fix(tls): cilium-gateway-cert STAGING/PROD issuer selectable via tofu
clusters/_template/sovereign-tls/cilium-gateway-cert.yaml hardcoded
letsencrypt-dns01-prod-powerdns regardless of qa_test_session_enabled.
On high-cadence QA reprov cycles this hits the LE PROD 5/168h rate
limit (caught on prov #76 at 13:45 UTC, retry-after 16:49 UTC) and
the wildcard Certificate sticks Ready=False — Cilium Gateway has no
valid TLS secret → envoy listener never binds → public TLS handshake
to console.<fqdn> dies with SSL_ERROR_SYSCALL.
Add tofu local.wildcard_cert_issuer = qa_test_session_enabled ?
staging : prod. Thread WILDCARD_CERT_ISSUER through the sovereign-
tls Kustomization postBuild.substitute. cilium-gateway-cert.yaml
references it as ${WILDCARD_CERT_ISSUER}.
Default behaviour unchanged for non-QA (production) Sovereigns —
they still resolve to letsencrypt-dns01-prod-powerdns.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(cilium-gateway): allow world ingress to Cilium Gateway reserved:ingress endpoint
When Cilium Gateway API runs with gatewayAPI.hostNetwork.enabled=true and
a default-deny CCNP is present, every public request to a Sovereign host
(console, auth, gitea, registry, api, ...) hits the gateway listener and
gets DENIED at envoy's cilium.l7policy filter with:
cilium.l7policy: Ingress from 1 policy lookup for endpoint X for port 30443: DENY
Public response: HTTP/1.1 403 Forbidden, body "Access denied", server: envoy.
Root cause: Cilium creates a special endpoint with identity reserved:ingress (8)
representing the gateway listener. By default this endpoint has
policy-enabled=both with allowed-ingress-identities=[1 (host)] and empty
L4 rules — so no port is permitted. The default-deny CCNP's NotIn-namespace
endpointSelector does NOT cover this endpoint (it has no
io.kubernetes.pod.namespace label), and our qa-fixtures didn't ship a
matching allow-template for it. Net effect: TLS handshake succeeds, HTTPRoutes
are Programmed, backends are healthy in-cluster, but every request 403s.
Caught live on prov #80 (omantel.biz, 2026-05-14) after the Gateway hostNetwork
fix (#1480) finally activated host-bind on :30443. Verified by:
- envoy debug log: cilium.l7policy DENY for endpoint 10.42.0.201 port 30443
- cilium-dbg endpoint get 3282 -o json: l4.ingress: [] and allowed-ingress-identities: [1]
- transiently applying the same CCNP via kubectl: console.omantel.biz → 200
Fix: ship a CCNP scoped to reserved:ingress that allows ingress from world,
cluster, host, remote-node (multi-region CP-to-CP), and kube-apiserver,
plus egress to all so envoy can forward to any backend service. This is
the canonical Cilium hostNetwork Gateway-API zero-trust pattern.
Chart bump: catalyst 1.4.142 → 1.4.143.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: e3mrah <catalyst@openova.io>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: e3mrah <1234567+e3mrah@users.noreply.github.com>
clusters/_template/sovereign-tls/cilium-gateway-cert.yaml hardcoded
letsencrypt-dns01-prod-powerdns regardless of qa_test_session_enabled.
On high-cadence QA reprov cycles this hits the LE PROD 5/168h rate
limit (caught on prov #76 at 13:45 UTC, retry-after 16:49 UTC) and
the wildcard Certificate sticks Ready=False — Cilium Gateway has no
valid TLS secret → envoy listener never binds → public TLS handshake
to console.<fqdn> dies with SSL_ERROR_SYSCALL.
Add tofu local.wildcard_cert_issuer = qa_test_session_enabled ?
staging : prod. Thread WILDCARD_CERT_ISSUER through the sovereign-
tls Kustomization postBuild.substitute. cilium-gateway-cert.yaml
references it as ${WILDCARD_CERT_ISSUER}.
Default behaviour unchanged for non-QA (production) Sovereigns —
they still resolve to letsencrypt-dns01-prod-powerdns.
Co-authored-by: e3mrah <catalyst@openova.io>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cilium gatewayAPI.hostNetwork.enabled=true was set in values.yaml,
but without nodes.matchLabels Cilium silently DISABLES hostNetwork
mode. The configmap key gateway-api-hostnetwork-nodelabelselector is
rendered EMPTY → eBPF redirect for the gateway NodePorts is never
programmed → envoy listener has empty bind address → incoming
30443/30080 traffic dead-ends at the Hetzner LB target.
Caught on prov #76 (omantel.biz, 2026-05-14): public TLS handshake
to console.omantel.biz returns SSL_ERROR_SYSCALL because envoy
isn't listening on the NodePort. cilium service list shows zero
30443/30080 entries. cilium proxy status shows 0 redirects active.
Set nodes.matchLabels: kubernetes.io/os: linux (every k3s node carries
this label) so the gateway listener is exposed on every CP.
Chart: 1.3.4 → 1.3.5. bootstrap-kit slot 01 version pin bumped to match.
Co-authored-by: e3mrah <catalyst@openova.io>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Empty handover/apps phase groups (no Jobs emitted yet for those
lifecycle phases) were hardcoded to 'running' which propagated up
to the root phase groups. With the rollup fix preserving stored
status when no children, the correct stored default is 'pending'.
After this, fresh-prov handover + apps groups show 'pending'
(accurate — those phases haven't started) and the rollup correctly
classifies bootstrap-kit + cutover region groups based on their
real install-* children.
Co-authored-by: e3mrah <catalyst@openova.io>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bug found on prov #76 rollup: cluster-bootstrap (a leaf with
family='bootstrap') was being treated as an empty group and reset
from succeeded → pending. That status then cascaded up through
provisioner (whose 5 children include cluster-bootstrap) making
provisioner show pending despite all 5 phase jobs being succeeded.
Fix: when a node in groupNodeIdx has zero children in contains rels,
keep its STORED status instead of forcing pending. This preserves
leaf-with-group-family nodes (cluster-bootstrap) AND empty phase
groups (handover/apps before their Jobs exist).
Co-authored-by: e3mrah <catalyst@openova.io>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bug found on prov #76: rolled-up group status fix wasn't visible
because catalyst-api Pod restart (image roll) killed the emit
goroutine. startFlowEmitLoop is only invoked from phase1_watch start
— for a deployment already at status=ready, the new Pod has no emit
loop until someone fires phase1 again.
Add idempotent startFlowEmitLoop call inside HandleFlowSnapshot so
any UI page load (which polls snapshot) reactivates the emit loop.
Combined with the existing phase1-start invocation, this covers both
fresh provisioning and post-restart UI access patterns.
Co-authored-by: e3mrah <catalyst@openova.io>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Founder reported on prov #76: 'there are pending and running jobs
still I dont think they are true'. Examination showed all 135
install-* leaf statuses are succeeded but the synthetic group nodes
(cutover, handover, apps + per-region sub-groups) carried hardcoded
placeholder statuses ('running' / 'pending') from emit time.
Add bottom-up roll-up after all nodes/rels are emitted:
- all descendants succeeded → succeeded
- any descendant failed → failed
- any descendant running → running
- else → pending (no descendants or all pending)
Now cutover phase bubble shows succeeded when its install-self-
sovereign-cutover child has finished, etc. handover/apps stay pending
until real Jobs are emitted for them (jobs.Store integration is the
follow-up that materialises those phases).
Co-authored-by: e3mrah <catalyst@openova.io>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
CI build failed with missing go.sum entry for pgx after the
in-memory→CNPG rewrite (now has real deps). The previous Dockerfile
only COPYed go.mod — fine when the codebase had zero external deps,
broken once pgx + pgxpool + x/text + x/sync landed in go.sum.
Co-authored-by: e3mrah <catalyst@openova.io>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Founder feedback on prov #75: "uncappetabel stupid design… if our pods
are restarting entire flow information are exec logs are being wiped".
Root cause: openova-flow-server had ZERO persistence (in-memory
map+RingBuffer per flowId) so pod restart wiped all canvas state.
catalyst-api's flow_snapshot_local.go composer was added as a "fallback"
precisely because openova-flow-server couldn't be trusted — but that
created TWO half-broken paths instead of one durable backend.
## Waterfall delivery — single PR, end-to-end
### openova-flow-server: in-memory → CNPG (Postgres) backed
- New schema: `flow_instances`, `flow_nodes`, `flow_relationships`,
`flow_events`, `flow_log_lines`, `flow_executions` with CASCADE FK,
indexes on (flow_id, status/region/family), and a bounded-retention
trigger on `flow_events` (keeps last 4096 per flow_id — matches the
prior RingBuffer capacity).
- `pgstore.go` rewires Append/Snapshot/Subscribe/Drop with pgxpool
transactional writes + LISTEN/NOTIFY pub/sub via per-flow channel
hash. Migrations applied at startup via embedded `embed.FS`.
- Backend abstraction (`store.Backend`) lets api/ swap between
PGStore (production) and the legacy MemBackend (tests/dev).
`FLOW_SERVER_BACKEND=pg|memory` env selects.
- New endpoints: POST/GET `/v1/flows/{id}/log-lines` for exec log
ingest+replay against the `flow_log_lines` table.
### Helm chart: CNPG Cluster CR + DSN wire-in
- New `templates/cnpg-cluster.yaml` provisions `openova-flow-pg` via
bp-cnpg's `postgresql.cnpg.io/v1.Cluster`. CASCADE-FK-aware schema
+ Reflector annotations for cross-NS secret access.
- Deployment env wires `FLOW_SERVER_PG_DSN` from CNPG's auto-generated
`<cluster>-app` Secret (`uri` key — full libpq URI with auth).
- `chart 0.1.1 → 0.2.0` (breaking schema change).
- bootstrap-kit slot 56: `dependsOn: bp-cnpg` so cold install order
is correct.
### catalyst-api: emit loop + remove local fallback first
- New `internal/flowemit/` HTTP client posts FlowMessage envelopes
(snapshot, upsert-nodes, upsert-rels, delete-*) to
`OPENOVA_FLOW_SERVER_URL/v1/flows/{id}/events`. Bounded retry,
fire-and-forget.
- New `flow_emitter.go` runs a per-deployment 5s ticker goroutine
that composes the current snapshot via `flowSnapshotFromJobs` and
emits it. State changes via Bridge call `triggerFlowEmit(depID)`
for sub-second propagation.
- `HandleFlowSnapshot` order INVERTED: proxy to openova-flow-server
FIRST, fall back to local composer ONLY in degraded mode (proxy
unreachable). Production traffic now durably reads from CNPG.
- Emit loop starts when phase 1 watch begins; idempotent; survives
catalyst-api restart because state is in CNPG.
## What this delivers
- ✅ Canvas data is DURABLE — survives any pod restart (catalyst-api,
openova-flow-server, or both).
- ✅ openova-flow-server is now stateless — every read hits CNPG.
- ✅ Wire contract (FlowMessage envelopes) unchanged. UI unchanged.
- ✅ catalyst-api can be horizontally scaled — no in-memory state
needed for the graph path (deployments map + jobs.Store retire
in follow-up).
## What's NOT in this PR (clear follow-up)
- jobs.Store + PVC retirement: exec logs still on PVC. Moving them
to `flow_log_lines` requires updating ~30 callers across the
catalyst-api handler/ package — out of scope for this single PR's
blast radius. The new `POST /v1/flows/{id}/log-lines` endpoint is
already in place; only the call sites need to migrate.
- flow_snapshot_local.go: kept as the degraded-mode fallback (proxy
unreachable). Will be deleted once jobs.Store retirement removes
the underlying read path.
Co-authored-by: e3mrah <catalyst@openova.io>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Founder caught on prov #75 (b7ae422089d4fde9) after PR #1469 deploy:
all 3 regions' 45-children dep wiring vanished after the catalyst-api
pod restart. Root cause: the deps were never in Job.DependsOn — they
were only in the Pod's in-memory hrDeps cache built from
liveWatcher.SnapshotComponents() Layer-2 in flow_snapshot_local.go.
Pod restart killed the cache.
## Two fixes
### Fix A — Bridge.OnHelmReleaseEvent preserves existing DependsOn
`OnHelmReleaseEvent` previously hardcoded `DependsOn: []string{}` on
every HR state-transition event, relying on `mergeJob` to keep the
prior list. That works when SeedJobsFromInformerList wrote the deps
FIRST. But the seed fires once at OnInitialListSynced; if the seed
ran during a window when HR.spec.dependsOn was being applied/rolling,
or if the seed didn't run at all (silent informer failure post-Pod
restart), Job.DependsOn stays `[]` forever and every subsequent event
re-confirms it.
Fix: load the existing Job from store first, carry its DependsOn
through on the upsert. Same pattern as OnRawComponentLog at line
~939. Combined with mergeJob's preserve-prev behaviour, deps are
durable across event waves.
### Fix B — /refresh-watch respawns secondary watchers
`POST /refresh-watch` rebuilt the PRIMARY helmwatch.Watcher and
re-ran SeedJobsFromInformerList for the primary. But it did NOT
respawn secondary watchers — so after a Pod restart, secondaries'
90 install Jobs stayed flat indefinitely. Fix: call
`spawnSecondaryRegionWatchers(dep)` from RefreshWatch (idempotent —
already running watchers short-circuit on `stopWatchers[region]`).
With this, /refresh-watch restores deps for ALL regions, not just
primary.
## Validation
Caught the bug via per-region edge audit on prov #75 (NOT aggregate
counts — per `feedback_validate_full_dod_before_declaring_pass.md`).
Pre-fix: fsn1=0 / hel1-2=0 / nbg1-1=0 intra-region edges. Post-fix
target: fsn1=71 / hel1-2=71 / nbg1-1=71.
Co-authored-by: e3mrah <1234567+e3mrah@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Founder rule on prov #75 review: "make sure the bubbles are no more
transparent and wires are always below the bubbles".
Two fixes:
1. **Opaque bubbles always**. Previously `groupOpacity = isDimmed ? 0.35 : 1`
dropped the entire group's opacity to 35% when another job was open
and this node wasn't on the focused path — making the bubble fill
see-through and the edges behind visible THROUGH the bubble. Replaced
with a CSS `filter: grayscale + brightness` treatment that desaturates
the dimmed node without making it transparent.
2. **Explicit edges-then-nodes paint layers**. Wrapped the edges loop in
`<g className="flow-edges-layer" data-layer="edges">` and the nodes
loop in `<g className="flow-nodes-layer" data-layer="nodes">`. SVG
paint order already produced the correct ordering via JSX source
order, but a future code change inserting another element between the
two could quietly break it; the explicit wrappers make the contract
load-bearing and inspectable.
Co-authored-by: e3mrah <1234567+e3mrah@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Regression caught immediately after PR #1467 by founder on prov #74
(be70efe343e58b5a). My validation declared "✅ all 5 issues passed"
based on aggregate 292 edges + 5 sampled hel1-2 deps, missing that
PRIMARY fsn1 had 0 intra-region edges + 71 phantom cross-region edges.
## Root cause
PR #1467 wired primary install jobs into a primary region sub-group
(jobRegion = dep.Request.Region) for symmetric multi-region rendering.
`regionalise()` triggered on `jobRegion != ""` — over-applying the
`fsn1:` prefix to PRIMARY's bare-named DependsOn entries:
install-cilium → install-fsn1:cilium (PHANTOM — no such node exists)
PRIMARY install Jobs have BARE JobNames in the store
("install-cilium"); only SECONDARY install Jobs have region-prefixed
JobNames ("install-hel1-2:cilium"). Region-prefixing primary deps
produces a JobID that matches no node, so the edge is dropped or
points at nothing.
A second related bug: Layer-1 heuristic
`!strings.Contains(dep, ":")` was used to detect bare-jobName form,
but with the new `:` separator a region-prefixed JobName
("install-hel1-2:cilium") now contains a colon — so the heuristic
mis-classified it as "already a full JobID" and emitted FromID
without the deploymentID prefix. Phantom edge.
## Fixes
1. `isSecondaryRegionJob := strings.IndexByte(j.AppID, ':') > 0`
replaces `jobRegion != ""` as the regionalise() gate. Primary
jobs have no `:` in AppID → no prefix injection.
2. `fullJobIDPrefix := deploymentID + ":"` replaces the
`strings.Contains(dep, ":")` heuristic. Only deps that ALREADY
carry the deploymentID prefix are passed through verbatim; bare
JobNames (with or without region prefix) get the JobID() wrap.
## Lesson learned
Saved `feedback_validate_full_dod_before_declaring_pass.md` —
aggregate metrics and sample checks are NOT validation. Every DoD
bullet must run an explicit per-tier pass/fail check before
declaring resolved.
Co-authored-by: e3mrah <1234567+e3mrah@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Founder caught 5 canvas defects on prov #73 (8cd1ff1a80430dc5):
1. ✅ depth=1 shows 2 bubbles (provisioner + bootstrap-kit) — confirmed
correct architecture per composer.
2. ✅ Expanding bootstrap-kit shows 3 region sub-groups — confirmed.
3. 🐛 All 135 install-* nodes had ZERO inter-HR dep edges. Snapshot
showed only 5 finish-to-start rels (tofu chain + bootstrap-kit
sequence). install-cert-manager → install-cilium etc. all missing.
4. 🐛 Canvas only emitted 2 phase groups (provisioner + bootstrap-kit).
Missing cutover/handover/apps despite being part of the canonical
5-phase lifecycle.
5. 🐛 /jobs/install-hel1-2/newapi returned 404 because TanStack Router
splits "/" in the $jobId param.
## Fixes
### Fix 3a: mergeJob preserves prev.DependsOn when next is empty
store.go:283 — `if len(next.DependsOn)==0 && len(prev.DependsOn)>0`
keeps prior list. Without this, every OnHelmReleaseEvent (which
hardcodes `DependsOn: []string{}` at line 508 because it doesn't
re-look up HR.spec.dependsOn per event) CLOBBERED the seeded deps.
Confirmed in store: 135/135 install Jobs had `dependsOn: []`
despite SeedJobsFromInformerList running with proper deps. Founder
reported this same flat-leaves bug 4 sessions in a row.
### Fix 3b: secondary watchers get region-aware seeder hook
New `attachSecondaryBridgeSeederHook` + `snapshotsToSeedsForRegion`
wire the seed path for secondary helmwatch.Watchers. Without this,
secondary install-* Jobs were only ever created by per-event
OnHelmReleaseEvent (DependsOn=[]) so the canvas dep graph was
permanently flat under secondary region groups regardless of fix
3a.
### Fix 3c: composer Layer-2 reads secondary watchers' HR.spec.dependsOn
flow_snapshot_local.go now also walks dep.secondaryWatchers and
populates hrDeps with region-prefixed keys + region-prefixed values.
After fix 3a+3b the stored Job.DependsOn is the authoritative source
(Layer 1) — this Layer-2 enrichment is the safety net for hot-
shipped charts that bypass the seed path.
### Fix 4: cutover/handover/apps phase groups
types.go — add GroupCutover/Handover/Apps constants + Display.
flow_snapshot_local.go — add phaseForChart() classifier (currently
maps self-sovereign-cutover → cutover), reparent install jobs to
the correct phase sub-group, synthesise per-region sub-groups for
each phase, emit top-level phase groups, and chain them with
finish-to-start: provisioner → bootstrap-kit → cutover → handover
→ apps.
### Fix 5: JobName separator `/` → `:` (canonical per memory rule)
phase1_watch.go:457 emits ev.Component = region + ":" + chart.
jobs_backfill.go + flow_snapshot_local.go updated to detect ":"
instead of "/". useJobLinkBuilder's encodeURIComponent already
handles ":". /jobs/install-hel1-2:newapi now matches the TanStack
Router $jobId route.
Co-authored-by: e3mrah <1234567+e3mrah@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR #1465 added `catalyst` + `newapi` to default-deny allowlist and
shipped `allow-kube-apiserver` CNP for qa-omantel, but the chart
version wasn't bumped so HRs across active provisions kept resolving
the OLD 1.4.141 artifact (with the broken allowlist). Bumping to
1.4.142 forces Flux on every Sovereign to upgrade and pick up the fix.
Co-authored-by: e3mrah <1234567+e3mrah@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(flow_snapshot): region-scope dep edges (no cross-region wiring)
Founder caught on prov #66 (3dc9249ea73a6840, 2026-05-13): hel1-2's
install-* nodes all rendered dep arrows pointing at PRIMARY's install
nodes — cross-region edges where NAMING-CONVENTION §1.3 demands
independent fault domains (no cross-region wiring).
Root cause: helmwatch.Bridge persists secondary-region Jobs with bare
dep names ("install-cilium") because HR.spec.dependsOn carries chart
names without region context. The snapshot composer's normaliser
turned `install-cilium` → `<depID>:install-cilium` which IS the
primary's cilium JobID, not hel1-2's `<depID>:install-hel1-2/cilium`.
Every secondary install therefore drew a phantom cross-region edge.
Fix: in flow_snapshot_local.go, region-scope dep names when the source
Job is regional:
jobRegion=="hel1-2" + dep="install-cilium"
→ "install-hel1-2/cilium" → "<depID>:install-hel1-2/cilium"
Same fix applied to the Layer-2 hrDeps derivation path (per-AppID
lookup also gets bare chart names from the primary watcher). hrDeps
lookup is now done with the unprefixed AppID so it actually hits.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(cloud-init): wait for private NIC before k3s install (prov #71)
Hetzner Cloud hot-attaches the private-network NIC ~10-20s AFTER server
create. cloud-init init-local fetches /hetzner/v1/metadata/private-networks
BEFORE the NIC is ready, renders netplan with only eth0, and the
private NIC (kernel-renamed eth1 → enp7s0 by udev) stays DOWN.
Effect on secondary CPs: k3s server starts with
--node-ip=10.0.<10+idx>.2 --advertise-address=10.0.<10+idx>.2
and fatals on
"listen tcp 10.0.11.2:2380: bind: cannot assign requested address"
then crashloops. Caught on prov #71/omantel.biz/nbg1-1-cp1: k3s.service
restart counter reached 5394, kubeconfig never PUT back to mothership,
canvas showed secondary region as a permanent black hole. Diagnosed via
Hetzner rescue mode SSH 2026-05-14. Primary CP works by luck of faster
fsn1 zone NIC attach.
Fix: in cloud-init runcmd, BEFORE the k3s install, poll up to 120s for
the expected private IP (control plane) or a route to it (worker). If
the NIC appears DOWN with no netplan stanza, generate one with dhcp4:true
and `netplan apply`. Bail loudly if the IP/route never appears — failures
surface in cloud-init.log instead of disguising as a slow boot.
Symmetric fix in worker template covers autoscaler-spawned secondary
workers when worker_count > 0.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(qa-fixtures): allow catalyst+newapi NS + kube-apiserver egress (prov #72)
The qa-fixtures chart's `default-deny` CiliumClusterwideNetworkPolicy
excluded `catalyst-system` from its NotIn list but FORGOT `catalyst`
(where bp-self-sovereign-cutover's Jobs live: auto-trigger,
gitea-mirror, harbor-projects, registry-pivot) and `newapi` (where
bp-newapi's Application pods live).
Effect on prov #72:
- bp-self-sovereign-cutover-auto-trigger Job stuck 20m+ on HTTP 000000
curling http://catalyst-api.catalyst-system.svc → DNS resolution + TCP
egress both denied by default-deny. Cutover never fires → handover
blocked → bp-catalyst-platform's --wait never completes.
- newapi-bp-newapi pod gets `secret newapi-oidc not found` but its
inability to resolve apiserver compounds the issue.
- qa-omantel cnpg cluster-primary/replica stuck "Setting up primary"
for 18m because initdb's `dial tcp 10.43.0.1:443 i/o timeout` — the
ClusterIP-rewritten kube-apiserver address has no allow-egress.
Fixes:
1. Add `catalyst` + `newapi` to $excludedNamespaces — they're first-party
blueprint namespaces analogous to catalyst-system.
2. Add `allow-kube-apiserver` CNP in qa-omantel using Cilium's canonical
`toEntities: [kube-apiserver]` directive so cnpg initdb can reach the
apiserver regardless of whether traffic resolves to ClusterIP, node
IP, or Service VIP.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: e3mrah <1234567+e3mrah@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(flow_snapshot): region-scope dep edges (no cross-region wiring)
Founder caught on prov #66 (3dc9249ea73a6840, 2026-05-13): hel1-2's
install-* nodes all rendered dep arrows pointing at PRIMARY's install
nodes — cross-region edges where NAMING-CONVENTION §1.3 demands
independent fault domains (no cross-region wiring).
Root cause: helmwatch.Bridge persists secondary-region Jobs with bare
dep names ("install-cilium") because HR.spec.dependsOn carries chart
names without region context. The snapshot composer's normaliser
turned `install-cilium` → `<depID>:install-cilium` which IS the
primary's cilium JobID, not hel1-2's `<depID>:install-hel1-2/cilium`.
Every secondary install therefore drew a phantom cross-region edge.
Fix: in flow_snapshot_local.go, region-scope dep names when the source
Job is regional:
jobRegion=="hel1-2" + dep="install-cilium"
→ "install-hel1-2/cilium" → "<depID>:install-hel1-2/cilium"
Same fix applied to the Layer-2 hrDeps derivation path (per-AppID
lookup also gets bare chart names from the primary watcher). hrDeps
lookup is now done with the unprefixed AppID so it actually hits.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(cloud-init): wait for private NIC before k3s install (prov #71)
Hetzner Cloud hot-attaches the private-network NIC ~10-20s AFTER server
create. cloud-init init-local fetches /hetzner/v1/metadata/private-networks
BEFORE the NIC is ready, renders netplan with only eth0, and the
private NIC (kernel-renamed eth1 → enp7s0 by udev) stays DOWN.
Effect on secondary CPs: k3s server starts with
--node-ip=10.0.<10+idx>.2 --advertise-address=10.0.<10+idx>.2
and fatals on
"listen tcp 10.0.11.2:2380: bind: cannot assign requested address"
then crashloops. Caught on prov #71/omantel.biz/nbg1-1-cp1: k3s.service
restart counter reached 5394, kubeconfig never PUT back to mothership,
canvas showed secondary region as a permanent black hole. Diagnosed via
Hetzner rescue mode SSH 2026-05-14. Primary CP works by luck of faster
fsn1 zone NIC attach.
Fix: in cloud-init runcmd, BEFORE the k3s install, poll up to 120s for
the expected private IP (control plane) or a route to it (worker). If
the NIC appears DOWN with no netplan stanza, generate one with dhcp4:true
and `netplan apply`. Bail loudly if the IP/route never appears — failures
surface in cloud-init.log instead of disguising as a slow boot.
Symmetric fix in worker template covers autoscaler-spawned secondary
workers when worker_count > 0.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: e3mrah <1234567+e3mrah@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Founder caught on prov #66 (3dc9249ea73a6840, 2026-05-13): hel1-2's
install-* nodes all rendered dep arrows pointing at PRIMARY's install
nodes — cross-region edges where NAMING-CONVENTION §1.3 demands
independent fault domains (no cross-region wiring).
Root cause: helmwatch.Bridge persists secondary-region Jobs with bare
dep names ("install-cilium") because HR.spec.dependsOn carries chart
names without region context. The snapshot composer's normaliser
turned `install-cilium` → `<depID>:install-cilium` which IS the
primary's cilium JobID, not hel1-2's `<depID>:install-hel1-2/cilium`.
Every secondary install therefore drew a phantom cross-region edge.
Fix: in flow_snapshot_local.go, region-scope dep names when the source
Job is regional:
jobRegion=="hel1-2" + dep="install-cilium"
→ "install-hel1-2/cilium" → "<depID>:install-hel1-2/cilium"
Same fix applied to the Layer-2 hrDeps derivation path (per-AppID
lookup also gets bare chart names from the primary watcher). hrDeps
lookup is now done with the unprefixed AppID so it actually hits.
Co-authored-by: e3mrah <1234567+e3mrah@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Founder caught on prov #65 (6e2fd14bb8b6ed4d, 2026-05-13): canvas shows
ASYMMETRIC structure — primary's 45 install jobs render as BARE LEAVES
directly under bootstrap-kit, while secondary regions get a proper
region sub-group. Result: M×N fan-out from provision-hetzner cascades
onto every primary leaf because there's no primary region group to
absorb the elided-group edge.
PR #1454 introduced region derivation from JobName's `/` separator
(secondary watchers emit `install-<region>/<chart>`). Primary's bridge
emits bare `install-<chart>` names — no `/`, no region derived, no
group synthesized.
Fix: derive primary region from `dep.Request.Region` and apply it to
every install job with no `/` in AppID. The synth-region-group loop
below already creates one group per discovered region, so primary
automatically gets its own `<deploymentId>:<primaryRegion>:bootstrap-kit`
bubble containing all 45 primary installs.
End state: 3 symmetric region sub-groups under bootstrap-kit
(fsn1 + nbg1-1 + hel1-2 for 3-region prov), each with exactly 45
install-* children, region-bounded temporal-endpoint cascade prevents
M×N fan-out at depth=all.
Co-authored-by: e3mrah <1234567+e3mrah@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two related fixes for multi-region + qa-fixtures DoD on prov #64:
1. **k3s TLS cert needs the public IPv4 in SAN.**
Mothership helmwatch.Bridge connects to secondary CPs via PUBLIC IP
(cloud-init rewrites kubeconfig 127.0.0.1 → CP_PUBLIC_IPV4). k3s
auto-generates the server cert with SANs from --tls-san flags. We
only had [sovereign_fqdn, cp_private_ip] → cert valid for 10.0.10.2
+ cluster-ip + 127.0.0.1 only. Bridge connection from contabo
rejected with:
"x509: certificate is valid for 10.0.10.2, 10.43.0.1, 127.0.0.1,
::1, not 204.168.212.113"
→ silent watcher failure → 0 secondary HRs observed → canvas missing
region sub-groups.
Fix: pre-fetch the CP's public IPv4 from Hetzner metadata before
k3s install, add it as --tls-san=$CP_PUBLIC_IPV4.
2. **openova.io/region=hz-fsn-rtz-prod node label.**
qa-fixtures Pods (CNPGPair primary/replica, status seeder Jobs,
qa-wp Application) carry hard nodeAffinity for
`openova.io/region in [hz-fsn-rtz-prod]` (per qaFixtures.primaryRegion
default in products/catalyst/chart/templates/qa-fixtures/*.yaml).
Without the label every fixture pod FailedScheduling → bp-catalyst-
platform post-install hook waits forever → bootstrap-kit chain hangs
at 44/45 with bp-catalyst-platform Running.
Fix: --node-label openova.io/region=hz-fsn-rtz-prod on primary CP
(qa-fixtures pin to primary by design).
Both shipped in same commit since both are inside the same k3s server
install line.
Co-authored-by: e3mrah <1234567+e3mrah@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
prov #63 (cpx52 × 3, all PRs live): bp-catalyst-platform install hook
timed out because the catalyst-api Helm-released pod stayed Pending
with "Too many pods. 0/1 nodes are available".
k3s kubelet default max-pods is 110. Full bootstrap-kit (~45 HR-managed
deployments, each with 1-3 pods) + qa-fixtures stack (qa-omantel ns
Application + Continuum + CNPGPair + PDM CRs + seeder Jobs) + Cilium/
flux/cnpg sidecars saturate the slot cleanly. With workers NotReady on
prov #63 the CP carried everything alone and dropped scheduling at 110.
Bump to 220 on both CP and worker so the saturation point doesn't gate
the bootstrap chain. Safe ceiling: each Hetzner cpx52 node has 16 vCPU
+ 32GB RAM, plenty of headroom for 220 pods of typical bootstrap-kit
weight.
Co-authored-by: e3mrah <1234567+e3mrah@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
prov #62 (cpx52, kernel 6.8.0-111): primary CP cilium init CrashLoop
with "dial tcp 10.0.1.2:6443: i/o timeout". k3s server auto-detects
its node IP from the primary interface, which on Hetzner cpx52 binds
to the public IPv4 (49.x.x.x) instead of the private network IP
(10.0.1.2). kube-apiserver advertises 49.x.x.x and binds there;
nothing answers on 10.0.1.2:6443. Cilium agent's k8s-client wants the
private IP from cilium-config k8sServiceHost — times out, CrashLoop.
Worked by luck on cpx42 (earlier kernel + Hetzner network attach
timing). cpx52 reproduces 100%.
Fix: pass --node-ip=${cp_private_ip} + --advertise-address=${cp_private_ip}
in INSTALL_K3S_EXEC. k3s then binds kube-apiserver on the private IP
AND advertises it as the node's INTERNAL-IP. Pods reaching ${cp_private_ip}:6443
(cilium-config substitute) find the API server every time.
Co-authored-by: e3mrah <1234567+e3mrah@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
prov #61 (2e197a934a0e0461, 2026-05-13): catalyst-api OOMKilled 6× during
phase-1 watch on a 3-region Sovereign. The in-memory state has grown
substantially since the 1Gi limit was set:
- 1 primary helmwatch.Watcher (45 HRs + informer cache)
- N secondary helmwatch.Watchers (45 HRs × 2 secondary regions, each
with its own informer cache)
- jobs.Store backed by on-disk + in-memory tree
- per-/snapshot poll: composes per-region region groups across all
Job rows + cross-references hrDeps from the live primary watcher
Combined steady-state exceeds 1Gi on cpx32-equivalent loads. Bumped
limits to 4Gi (request 512Mi up from 128Mi). The mothership node has
8GB+ resident, no other tight constraint. Future fix: persist region
in Job rows so secondary watchers don't need to be retained post
phase-1 (orthogonal cleanup).
Co-authored-by: e3mrah <1234567+e3mrah@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(JobsTable): strip <deploymentId>: prefix from row link (404 fix)
Founder caught on prov #59 (a43364f11c10cde3, 2026-05-13): clicking a
running secondary-region install-* row on /sovereign/provision/<id>/jobs
landed on /provision/<id>/jobs/<id>:install-nbg1-1/self-sovereign-cutover
and returned "404 page not found".
Root cause: useJobLinkBuilder was passing the FULL canvas JobID form
through encodeURIComponent.replace(/%3A/g, ':') WITHOUT first stripping
the "<deploymentId>:" prefix. The canvas emits ids like
"<deploymentId>:install-X" (single-region) or
"<deploymentId>:<region>:install-X" (multi-region, see
flow_snapshot_local.go:410). jobs.Store.GetJob keys by the BARE jobName —
exact-match URL lookup of the prefix-bearing form misses every time.
FlowPage.handleNodeDoubleClick (FlowPage.tsx:355) already strips the
first `:` prefix for canvas drill-down; JobsTable now matches so a /jobs
row click and a canvas drill-down resolve to the SAME backend endpoint.
The existing JobsTable row-link test uses a job.id with no `:` prefix,
so the strip is a no-op for that fixture and the `/jobs/job-install-cilium`
assertion still holds.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(flow_snapshot_local): derive region from persisted JobName, synth region groups
Founder caught on prov #59 (a43364f11c10cde3, 2026-05-13): the multi-region
canvas at /sovereign/provision/<id>/jobs/tofu-output renders 135 install-*
leaves as direct children of bootstrap-kit (no region sub-groups visible),
and the provision-hetzner→bootstrap-kit edge fans M×N across all 135.
Root cause: spawnSecondaryRegionWatchers (phase1_watch.go:429) emits
events with `ev.Component = region + "/" + componentName`. The jobs
bridge persists them with `JobName=install-<region>/<chart>` and
`AppID=<region>/<chart>`, BUT ParentID=bootstrap-kit (the bridge has no
region awareness). After phase 1 terminates the deferred stopSecondaries()
clears `dep.secondaryWatchers`, so the multi-region snapshot block
(line 408-460, gated on `len(secondaryWatchers) > 0`) becomes a no-op.
flowSnapshotFromJobs then emits all 135 install Jobs flat under
bootstrap-kit, no Region field set, no region group bubbles, and
flowLayoutOrganic.ts's temporal-endpoint cascade fans the
provisioner→bootstrap-kit edge onto all 135 because there's no
intermediate region group to absorb it.
Fix: in the per-Job loop, detect `/` in `j.AppID` (the canonical
multi-region prefix marker), derive the region key, set
FlowNode.Region, and re-parent to a synthesised
"<deploymentId>:<region>:bootstrap-kit" group. After the loop,
synthesise one bootstrap-kit sub-group node per discovered region
with a `contains` edge to the parent bootstrap-kit. The resulting
shape:
bootstrap-kit
├── 45 primary install-* (legacy parent, no region)
├── <region-A>:bootstrap-kit ── 45 install-* (region tagged)
└── <region-B>:bootstrap-kit ── 45 install-* (region tagged)
This persists ACROSS phase-1 termination because the source of truth
is jobs.Store (durable), not dep.secondaryWatchers (transient).
The multi-region block (line 408+) still runs WHEN secondary watchers
are alive (during phase 1) — it emits ADDITIONAL FlowNodes with
"<deploymentId>:<region>:install-X" IDs distinct from the persisted
"<deploymentId>:install-<region>/<chart>" IDs, so the two paths don't
collide. Post-phase-1 the watchers clear and only the persisted-Job
path remains, but now WITH region structure preserved.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(flow_snapshot): remove duplicate live-watcher multi-region block
PR #1454 added region-group synthesis from persisted Job rows. The old
secondaryWatchers-based block at line 442+ emitted nodes with the SAME
region-group IDs AND child nodes, so during phase 1 (when both paths
are live) the snapshot rendered with 90 children per region group
instead of 45 — visible on prov #61 (2e197a934a0e0461):
bootstrap-kit: 49 children
hel1-2:bootstrap-kit: 90 children (should be 45)
nbg1-1:bootstrap-kit: 90 children (should be 45)
Plus the region groups appeared twice in the node list.
Root cause: the per-Job loop (PR #1454) and the legacy block both write
to the same region-group IDs without deduping. The per-Job path covers
the persisted-Job state (durable across phase-1 termination), so the
live-watcher path is redundant.
Fix: delete the legacy block. The earlier
secondaryWatchers-snapshot-into-map work (lines 182-205) is kept
because that path also reads dep.liveWatcher (primary) for the hrDeps
lookup the per-Job loop uses for primary-region dep edges.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: e3mrah <1234567+e3mrah@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(JobsTable): strip <deploymentId>: prefix from row link (404 fix)
Founder caught on prov #59 (a43364f11c10cde3, 2026-05-13): clicking a
running secondary-region install-* row on /sovereign/provision/<id>/jobs
landed on /provision/<id>/jobs/<id>:install-nbg1-1/self-sovereign-cutover
and returned "404 page not found".
Root cause: useJobLinkBuilder was passing the FULL canvas JobID form
through encodeURIComponent.replace(/%3A/g, ':') WITHOUT first stripping
the "<deploymentId>:" prefix. The canvas emits ids like
"<deploymentId>:install-X" (single-region) or
"<deploymentId>:<region>:install-X" (multi-region, see
flow_snapshot_local.go:410). jobs.Store.GetJob keys by the BARE jobName —
exact-match URL lookup of the prefix-bearing form misses every time.
FlowPage.handleNodeDoubleClick (FlowPage.tsx:355) already strips the
first `:` prefix for canvas drill-down; JobsTable now matches so a /jobs
row click and a canvas drill-down resolve to the SAME backend endpoint.
The existing JobsTable row-link test uses a job.id with no `:` prefix,
so the strip is a no-op for that fixture and the `/jobs/job-install-cilium`
assertion still holds.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(flow_snapshot_local): derive region from persisted JobName, synth region groups
Founder caught on prov #59 (a43364f11c10cde3, 2026-05-13): the multi-region
canvas at /sovereign/provision/<id>/jobs/tofu-output renders 135 install-*
leaves as direct children of bootstrap-kit (no region sub-groups visible),
and the provision-hetzner→bootstrap-kit edge fans M×N across all 135.
Root cause: spawnSecondaryRegionWatchers (phase1_watch.go:429) emits
events with `ev.Component = region + "/" + componentName`. The jobs
bridge persists them with `JobName=install-<region>/<chart>` and
`AppID=<region>/<chart>`, BUT ParentID=bootstrap-kit (the bridge has no
region awareness). After phase 1 terminates the deferred stopSecondaries()
clears `dep.secondaryWatchers`, so the multi-region snapshot block
(line 408-460, gated on `len(secondaryWatchers) > 0`) becomes a no-op.
flowSnapshotFromJobs then emits all 135 install Jobs flat under
bootstrap-kit, no Region field set, no region group bubbles, and
flowLayoutOrganic.ts's temporal-endpoint cascade fans the
provisioner→bootstrap-kit edge onto all 135 because there's no
intermediate region group to absorb it.
Fix: in the per-Job loop, detect `/` in `j.AppID` (the canonical
multi-region prefix marker), derive the region key, set
FlowNode.Region, and re-parent to a synthesised
"<deploymentId>:<region>:bootstrap-kit" group. After the loop,
synthesise one bootstrap-kit sub-group node per discovered region
with a `contains` edge to the parent bootstrap-kit. The resulting
shape:
bootstrap-kit
├── 45 primary install-* (legacy parent, no region)
├── <region-A>:bootstrap-kit ── 45 install-* (region tagged)
└── <region-B>:bootstrap-kit ── 45 install-* (region tagged)
This persists ACROSS phase-1 termination because the source of truth
is jobs.Store (durable), not dep.secondaryWatchers (transient).
The multi-region block (line 408+) still runs WHEN secondary watchers
are alive (during phase 1) — it emits ADDITIONAL FlowNodes with
"<deploymentId>:<region>:install-X" IDs distinct from the persisted
"<deploymentId>:install-<region>/<chart>" IDs, so the two paths don't
collide. Post-phase-1 the watchers clear and only the persisted-Job
path remains, but now WITH region structure preserved.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: e3mrah <1234567+e3mrah@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Founder caught on prov #59 (a43364f11c10cde3, 2026-05-13): clicking a
running secondary-region install-* row on /sovereign/provision/<id>/jobs
landed on /provision/<id>/jobs/<id>:install-nbg1-1/self-sovereign-cutover
and returned "404 page not found".
Root cause: useJobLinkBuilder was passing the FULL canvas JobID form
through encodeURIComponent.replace(/%3A/g, ':') WITHOUT first stripping
the "<deploymentId>:" prefix. The canvas emits ids like
"<deploymentId>:install-X" (single-region) or
"<deploymentId>:<region>:install-X" (multi-region, see
flow_snapshot_local.go:410). jobs.Store.GetJob keys by the BARE jobName —
exact-match URL lookup of the prefix-bearing form misses every time.
FlowPage.handleNodeDoubleClick (FlowPage.tsx:355) already strips the
first `:` prefix for canvas drill-down; JobsTable now matches so a /jobs
row click and a canvas drill-down resolve to the SAME backend endpoint.
The existing JobsTable row-link test uses a job.id with no `:` prefix,
so the strip is a no-op for that fixture and the `/jobs/job-install-cilium`
assertion still holds.
Co-authored-by: e3mrah <1234567+e3mrah@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
