Labels, Not Addresses: Workload Identity in Network Policy

Part 3 of Microsegmentation from First Principles, a series that works through the design of Innerwall, an open-source microsegmentation platform, one problem at a time.

Part 2 closed on an assumption it had carried unexamined: that a policy's rules stay true as the estate changes underneath them. This post examines that assumption and replaces it.

An address is a claim about an assignment

Part 1 ended its "generalize" step with a rule in two forms. The specific form is what the flow map records:

allow 10.0.1.23, 10.0.1.24 → 10.0.2.9 port 5432/tcp

Each address in it claims that a particular workload holds that address today. Assignments drift. Instances are replaced, groups scale out, leases expire and hosts return somewhere else. No rule changes when this happens, so the policy goes wrong silently, in both directions. In Part 1's notation, with A the allowed flows and R the required ones:

• A new instance is missing from a peer set. The third checkout instance, added at scale-out, is refused by the database. R ∖ A ≠ ∅: an outage, arriving under exactly the load that triggered the scale-out.
• A departed instance's address stays in one. The address is free for reassignment, and whoever receives it next inherits database access. A ∖ R ≠ ∅: exposure, handed to a host nobody reviewed.

The firewall is not malfunctioning in either case. It matches exactly the addresses it was given; nothing in the kernel knows that an address changed hands. The first failure surfaces eventually; the second never announces itself. Both come from the same mistake. The operator never meant 10.0.1.23. They meant the checkout service, and the address was a proxy for it, valid only while someone kept it current by hand. Policy should name the workload: app=checkout may reach app=payments-db on 5432/tcp. That moves the problem rather than dissolving it, because a label is only as trustworthy as whatever decides which workload carries it.

An identity has to be assigned, not claimed

A rule that admits app=checkout is an authorization decision about every workload carrying that label. If a workload could state its own labels, the rule would admit anything willing to say app=checkout. The identity policy is written against has to be granted by something other than its subject.

Innerwall splits that identity into two layers.

The credential answers "which workload is this?" At enrollment the agent generates its key pair locally and sends a certificate signing request with a provisioning token. The control plane assigns the workload a UUID and issues a certificate carrying exactly one URI subject alternative name, innerwall://workload/<uuid>, and no DNS, IP, or email names. The signing request contributes its public key and nothing else: the subject, alternative names, and extensions the enrollee asked for are all discarded, and a test that submits a hostile request naming another workload stays in the repository as the executable statement of that rule. This layer is authentication.

The registry answers "what is it allowed to do?" Hostname, labels, and enforcement mode live in the control plane's registry, keyed by the workload id and never written into the credential. The design record puts the split in four words: cert = authentication; registry = authorization. Every agent request takes its identity from the connection's certificate and never from its payload, which is why Part 1's flow records could trust the identity of the workload reporting them.

Labels enter the registry from the token. A provisioning token is minted for a label set, may enroll any number of workloads while valid, and assigns its labels to each of them. The workload never chooses, and the wire contract's design record gives the reason: agents never self-assign labels "because an agent-supplied label would let a compromised host place itself into a more permissive policy scope." After enrollment, labels change only by an operator's edit.

That makes the token the trust anchor for labels, and it bounds what stealing one buys. A leaked token enrolls a new workload with exactly the token's labels: a new identity, visible in the registry and tied to the token that enrolled it, and revoking the token stops any more. It cannot relabel an existing workload, widen its own scope, or impersonate an enrolled workload, whose identity rests on a private key that never left its host. Tokens are stored only as hashes, expire after thirty days unless the operator sets otherwise, and begin with a fixed prefix so a secret scanner can recognize one that leaked.

The split matters because labels and credentials change at different rates, for different reasons. Relabeling is what labels are for: a workload moves from staging to production, a service is renamed, an owner redraws its scopes. If labels lived in the certificate, every relabel would be a reissue, and every certificate issued before it would carry stale authorization until it expired. With the split, a relabel is a registry write, conditioned on the version of the label set the operator read so that two concurrent edits cannot silently overwrite each other. It takes effect at the next render, and the credential, which only ever says who, does not change. Credentials have their own lifecycle, short-lived and renewed, and Part 5 covers it.

Selectors stay in the control plane

Rules are authored against selectors. In Innerwall a ruleset has a scope selector naming the workloads it protects, and each inbound rule names its peers (by selector, address group, or literal CIDR) and the services it permits. Hosts never see any of that. The control plane resolves each workload's policy into a fully concrete artifact: resolved rules, each with peer CIDRs, exactly one protocol, and port ranges. A managed peer appears as host routes of its current addresses.

The artifact is small by construction, because a workload receives only its own inbound rules: payments-db-1 gets one rule with two addresses, and the checkout hosts get nothing from it. Here is everything payments-db-1 receives, rendered as JSON (on the wire it is protobuf, and the rule id is abbreviated):

{
  "version": 7,
  "mode": "ENFORCEMENT_MODE_SIMULATION",
  "inbound_rules": [
    {
      "rule_id": "r1/tcp",
      "peer_cidrs": ["10.0.1.23/32", "10.0.1.24/32"],
      "protocol": "PROTOCOL_TCP",
      "ports": [{ "start": 5432, "end": 5432 }]
    }
  ]
}

No label appears in it, and no selector. Three reasons keep selectors on the control-plane side of the line.

The host is the subject of the decision, not its judge. Deciding which workloads count as app=checkout is authorization, and authorization lives in the registry. A host that evaluated selectors would need the registry's contents to do it, every workload's labels and addresses, shipped to every host including the compromised one, and each host's copy of the registry would become an input to authorization that the control plane cannot see. A rendered artifact discloses only the addresses allowed to reach that host.

Behavior is deterministic from the artifact on disk. An agent's kernel state is a function of one thing: the rendered policy it last applied, which it persists and re-applies on start (Part 2's fail-static). Debugging a host means reading that artifact, not reconstructing what the agent believed about the estate when it last evaluated a selector. Two hosts holding the same artifact install the same rules, and the question "why did this address reach the database?" is answered by the artifact and the version that installed it, not by a selector evaluation nobody recorded.

Selector semantics can evolve without touching the fleet. How selectors match, what a rule may name as a peer, how services expand: all of it lives in the control plane, and the agent contract does not move. The wire contract sits behind a breaking-change gate, so a richer selector is a control-plane release, not a fleet upgrade.

Addresses become derived state

"Labels, not addresses" does not make addresses disappear. The kernel still matches them: the chain in Part 2 accepts ip saddr @r1, and @r1 is a set of addresses. What changes is who owns them. The operator speaks labels, the kernel speaks addresses, and the renderer is the single owner of the translation.

That ownership holds only if the translation is never stale, so every input the renderer reads is a trigger. An authored change (a ruleset, service, or address group created, updated, or deleted), an enrollment, a label change, an enforcement-mode change, or an inventory report that changes a workload's addresses triggers a render. In this version a render recomputes every workload's policy from the complete authored model and the current registry, in one transaction. There is no affected-set computation to get wrong. Correctness comes from re-deriving everything from the same inputs, and narrowing the set with an index from labels to the selectors that mention them is a later optimization that must not change the result.

Re-deriving everything would be expensive to ship, so versions follow output, not events. For each workload the render diffs the new policy against the persisted one. An empty diff leaves the version where it was and sends nothing. A non-empty diff advances that workload's version by one, persists the policy, and announces it. Because the output is a pure function of its inputs, the persisted state is stable: a control-plane restart keeps every version, and the render after it changes nothing. Diffs use a small vocabulary: upsert or remove a rule, add or remove peers in a rule, set the mode. Peer membership is the quantity that churns, and on the host a peer-only change is applied as set element additions and deletions rather than a rebuilt ruleset.

Deltas need stable names for what they change. An authored rule that permits several protocols renders one resolved rule per protocol, and each resolved rule's id is the authored rule's id, a slash, and the protocol: r1/tcp, r1/udp. Deltas address rule removals and peer changes by that id, and anything that needs provenance, such as a diagnostic pointing a host's rule back to the intent behind it, recovers the authored rule by splitting the id.

Not every peer can carry a label. A database without an agent, a monitoring range, a partner network: these are address groups, and their addresses remain claims an operator maintains, like Part 1's unmanaged peers. Naming them once still pays. An address group is an authored object, so editing it renders, and every rule that names it changes together instead of one hand-kept copy at a time.

The render runs synchronously in the process that made the edit, so by the time the edit is acknowledged, every affected workload's new version is persisted. The relabeled workload itself stays at its version: its own inbound rules don't mention env, so nothing it enforces changed. A new version always means different rendered content, never merely that something happened.

All of this rests on one contract: for every pair of consecutive versions, applying the delta between them to the earlier snapshot yields the later snapshot exactly, under a canonical serialization. Like Part 2's one-line chain difference, it is a tested invariant. TestSnapshotDeltaProperty, in the diff-and-apply package that the control plane and the agent both link, walks every delta variant alone and in combination, then randomized pairs, and fails unless each result matches the later snapshot byte for byte, a second diff comes back empty, and the delta is empty exactly when the two policies are equal.

Back to the two drift cases. Scale-out is an enrollment: the new instance arrives with its token's labels, and once the registry holds its addresses, the render puts them in every peer set those labels match, which closes the outage half with no rule edited. Re-addressing is an inventory report, and the render moves the address. Derivation keeps every peer set true to the registry, but the registry cannot know a workload is gone. Retiring one, taking its identity and addresses out of the render, is an operator lifecycle act that has no surface yet.

Labels across time

A label is current truth, and current truth changes. So everywhere the system reads a label, it is explicit about when:

• Flow history reads labels at ingest. Part 1's records store the peer's identity and a snapshot of its labels as they stood when the flow arrived. Records written before the relabel above still say env=staging, and they answer "which scope was this flow in then?"
• A review scope reads labels at query time. Part 2's rollup resolves the scope to the workloads in it now and reads their history. It answers "what have the workloads in this scope now seen?" After the relabel, a review of env=prod workloads includes checkout-3's history from its staging days, which is the right answer to that question and the reason the stored snapshot exists for the other one.
• Promotion pins the answer. A mode change names its set by selector or by workload id, always with an expected count. In one transaction the control plane resolves the selection, records the set it resolved to, and flips exactly that set by id, so a label change committed meanwhile cannot move it. A selection that resolves to a different number of workloads than expected is refused with 409 match-count-mismatch: the estate drifted between looking and acting.

"Which scope was it in then" and "which scope is it in now" are different questions with different correct answers, and nothing in the system answers one when asked the other.

Next: getting the policy to the hosts

A render ends in the database: a persisted policy and a version for each workload. That policy still has to reach every host, and stay correct across agent restarts, reconnects, and partitions, without a delta ever being applied to the wrong base. That is Part 4, Streaming Desired State: Snapshots, Deltas, and Per-Workload Versions.

Summary

• A rule that names addresses is a claim about assignments, and drift breaks it silently: an outage at scale-out, exposure when a stale address is reassigned.
• Naming workloads by label is safe only if identity is assigned: one control-plane-issued workload id in the credential, and labels from the provisioning token's scope, never from the workload.
• The credential authenticates and the registry authorizes, so a relabel is a registry write, never a reissue.
• Selectors are evaluated only in the control plane. Hosts receive concrete per-workload rules and behave deterministically from that artifact.
• Addresses are derived: every input change renders, a version advances only on a non-empty diff, and peer churn ships as set updates. TestSnapshotDeltaProperty holds the delta contract.
• Flow history keeps labels as they were, review scopes resolve labels as they are, and promotion pins its set by id.

Next in the series: Streaming Desired State: Snapshots, Deltas, and Per-Workload Versions.