Streaming Desired State: Snapshots, Deltas, and Per-Workload Versions

Part 4 of Microsegmentation from First Principles, a series that works through the design of Innerwall, an open-source microsegmentation platform, one problem at a time.

Part 3 ended in the database: every render leaves a persisted policy and a version for each workload. This post gets that policy onto the hosts and keeps it there.

The problem a render leaves

A row in a database protects nothing. The policy has to reach hundreds of hosts and hold there across agent restarts, control-plane restarts, dropped connections, and partitions. It also has to keep the promise Part 2 depends on: the ruleset a host runs is exactly the one the evidence was gathered about. A clean simulation of version 13 says nothing about a host that is running something else.

Most failures on that path are loud. One is silent, and it is the one to design against: a delta applied to the wrong base. A delta is a list of changes (add these peers, remove that rule), and it is correct only relative to the state it was computed from. Applied to any other state, it produces a policy no render ever produced, with rules nobody authored and peers nobody reviewed, under a version number that claims otherwise. Nothing errors. The host reports version 14, the control plane believes version 14, and the kernel holds something else.

Desired state, not commands

A command protocol sends operations, such as "add 10.0.1.25 to set p1", and its question is did every host run my change? Answering it means tracking every command on every host: delivered, applied, retried. A host that misses one stays wrong until someone notices, and a host that comes back after an hour has a backlog to replay in order. Missed messages are a correctness problem.

A desired-state protocol sends the state each host should hold, and its question is does every host converge on the persisted state? The control plane keeps one target per workload, its latest rendered policy, and the agent's job is to reach it from wherever it is. A missed message costs time, never correctness, because the next exchange carries the target rather than a step toward it. An agent offline for an hour reconnects and receives the current state once, not the hour's history.

Deltas still have a place, as compression, but only where both sides provably know the base. The next section says where that is.

Convergence is observable per workload. Each agent reports the version it has applied, and the control plane persists that beside the latest version it rendered, with a sync state: PENDING while an update is in flight, SYNCED once the agent acknowledges the latest version sent, DEGRADED after a failed apply, OFFLINE with no stream open. That one read, applied against latest, is what Part 2's verdict leaned on when it refused any workload "behind on its version, still converging, or offline". A workload whose applied version equals its latest runs exactly the rendered artifact. Any other workload does not, and the verdict says so.

Snapshot on connect; deltas never cross a session

The rule in Innerwall's wire contract is that every new sync stream begins with a full snapshot, whatever version the agent claims. The agent opens the stream with Hello, reporting the version it has applied. The server answers with HelloAck and then the persisted policy as a PolicySnapshot. Deltas follow only after that, only on that stream, each computed from what that stream has already carried.

Session 1 · one stream

1. agent → cp

   Hello(v12)

   A claim about the past. Recorded; never used as a base.

2. cp → agent

   Snapshot(v12)

   Full persisted state. The stream's base is now known to both sides.

3. cp → agent

   Delta v12 → v13

   From the tip of what this stream carried.

4. agent → cp

   ACK APPLIED v13

   Applied atomically, in order.

5. cp → agent

   Delta v13 → v14

   Pipelined; the server does not wait for acks.

6. agent → cp

   ACK FAILED v14

   Applied as a unit or not at all: v13 stays installed.

7. cp → agent

   Snapshot(v14)

   Recovery. A snapshot has no base, so it is correct whatever the agent holds.

disconnect · backoff · reconnect

A delta never crosses this line.

Session 2 · a new stream

1. agent → cp

   Hello(v14)

   Reports the version it holds, from disk if it restarted.

2. cp → agent

   Snapshot(v14)

   Again, regardless of the claim.

The version in Hello is recorded, but it does not choose the base, and that is the point. A claimed version is a statement from a previous session about state the server can no longer check. The agent may have applied an update whose acknowledgement was lost with the connection, failed one and stayed behind, or restarted from a file older than its last acknowledgement. Each case leaves the two sides disagreeing about the base, and a delta chosen from a claimed version is exactly the wrong-base failure. The snapshot replaces whatever the agent holds with a state the server knows, so within a stream the base is always the last thing that stream carried. The contract's design record states the payoff: "there is no state in which the server and agent disagree about the base a delta applies to." The failure class is not handled. It cannot be constructed.

The agent enforces the same rule from its side. A delta carries its from_version and to_version, and the contract makes it valid only on the stream that established its base. A delta whose from_version is not the version the agent holds is refused as a unit and acknowledged as failed, with the installed policy untouched.

The cost is one snapshot per reconnect, and Part 3 already showed why it is small: a workload receives only its own inbound rules, rendered concretely. A snapshot is a few kilobytes, and reconnects are rare. Connecting is not a render trigger either: a new stream is served the persisted artifact.

Versions that mean something

The snapshot rule needs a version to name the state, and the version needs three properties.

Per workload. The wire contract knows one version per workload, opaque except for ordering. There is no estate-wide generation in it. If the agent contract carried one, every render that changed anything would advance it everywhere, and every agent would be told its policy was out of date when nothing in it had changed. One workload's recompile would stampede the fleet. An estate-wide view, if an audit trail or a console wants one, is a mapping the control plane may keep for itself; it never reaches an agent.

Monotonic, and advancing only on real change. Part 3 covered the diff gate: a render diffs each workload's new policy against its persisted one, and the version advances by one only when the diff is not empty. A new version therefore always means different rendered content, and the delta between consecutive versions is exactly the diff that justified the bump.

Describing complete state. A version means something only if it describes everything installed, and that fixes the acknowledgement semantics. The server pipelines updates without waiting for acknowledgements. The agent applies them strictly in stream order, each atomically, and acknowledges each one APPLIED or FAILED. There is no third status. Partial application is not representable, because a version that described half an installed delta would describe a state no render produced.

Two tests in the agent's sync package hold these semantics. TestAppliesInOrderAndAcksEach sends a snapshot at version 1 and two deltas behind it without waiting, and passes only if the agent acknowledges 1, 2, and 3 as APPLIED, in that order, and ends holding exactly version 3. TestFailedApplyKeepsLastGoodAndAcksFailed has the host's firewall refuse version 2, and passes only if the agent acknowledges FAILED with the refusal as its detail while version 1 stays installed, unchanged. It then sends a delta against a base the agent does not hold, and requires the same: failed, untouched.

What follows a failure is the server's move, and it is never a retry. A failed delta can fail again for the same reason, and retrying it assumes the server knows the agent's state, which the failure has just put in doubt. So the server records the workload as DEGRADED, with the agent's detail, and sends a fresh snapshot of the current persisted version. A snapshot has no base, so it is the one update that is correct whatever the agent holds. Deltas already in flight behind the failure may fail too. The server recognizes those failures as older than the recovery snapshot and does not answer each with another one, which would loop.

On the convergence read, a workload that failed version 14 shows applied 13, latest 14, DEGRADED, and the reason; Part 2's verdict refuses it until it is SYNCED again.

Push, not poll

A render runs in whichever process made the edit: the command line today, an API process later. Those processes hold no streams. Streams live in control-plane replicas, each holding some share of the fleet's connections, so something has to carry "this workload changed" from one process to another. Innerwall's sync design rules out polling in steady state. A poll interval trades convergence latency against load, and it would set every replica querying the database on a timer for changes that are rare.

The database carries the announcement instead.

Authoring process

Command line or API

Makes the edit and runs the render. Holds no streams.

✕No path to a gateway. It does not know where any stream is, and nothing tells it.

Render transaction

1. take the render lock
2. persist payments-db-1 · v13
3. notify "payments-db-1 13"
4. commit

The notification is delivered on commit: it exists only if the policy it names is durable.

Every replica listens

• Replica A

  streams of web-1, web-2

  No stream for payments-db-1: drops it.

• Replica B

  stream of payments-db-1 · tip v12

  Persisted v13 is newer than the tip: Delta v12 → v13.

• Replica C

  streams of checkout-1, checkout-2

  No stream for payments-db-1: drops it.

Host

payments-db-1

Receives Delta v12 → v13 on its stream, applies it atomically, and acknowledges.

If a listener was down

Announcements made meanwhile are gone. When the listener reconnects, the replica reconciles every live stream it holds against persisted state, pushing wherever the persisted version is newer than the tip. A lost announcement costs one reconciliation, never a missed version.

1. The render transaction persists and announces. Every render takes an advisory lock that serializes renders across processes. Inside the transaction it writes each changed workload's policy and version and, in the same transaction, publishes a notification naming the workload and its new version on a policy channel. The database delivers notifications only on commit, so an announcement exists only if the policy it names is durable, and a rolled-back render announces nothing.
2. Every replica listens. Each holds one dedicated listening connection, outside its pool, and hears every announcement.
3. The replica holding the stream pushes. A replica without a stream for the named workload drops the announcement. The replica that has one reads the persisted policy and compares it with the tip of what that stream has already carried. If the persisted version is newer, it computes the delta from that tip to the persisted policy and sends it. Deltas are computed per stream, at push time.

The authoring process never needs a path to a gateway. It does not know where any stream is, and nothing has to tell it, so there is no presence table: each replica pushes only to the streams it holds. The same path serves every trigger, so an enrollment or an inventory change made by the control plane itself is announced exactly as a command-line edit is.

An announcement is a hint, not state, and that makes losing one cheap. Notifications are not durable: one published while a replica's listener is down is simply gone. So whenever a listener connects, the first time or after a failure, the replica reconciles every live stream it holds against persisted state with the same push-if-newer step. A lost announcement costs one reconciliation, never a missed version. The step compares versions rather than replaying events, so reconciling a stream that is already current sends nothing.

The same bridge serves "send it again." An operator sometimes wants one workload's stream restarted from a snapshot. POST /workloads/{id}/resend-snapshot publishes a directive on the same channel, keyed by the workload. The replica holding that stream sends the agent a Reconnect directive, and the new stream begins with a snapshot by the rule above. Nothing new was built for it. Delivery is best-effort by design: no replica can know for certain that a stream exists, and a directive published while a listener is down is not reconciled, because it is a request rather than state. The outcome is observed instead of confirmed. The sync path records the instant it last sent each workload a snapshot, whatever the cause, and the request has worked when a later read shows last_snapshot_sent_at advanced past the value the request returned. A workload whose stored sync state says offline is refused with 409 agent-offline and the instant it was last seen. That refusal mirrors the record; it is not a liveness probe.

When the link is down

Reconnect is a fresh Hello, under jittered backoff. The agent never resumes a session; every reconnect starts again from a snapshot. The wait before attempt n is random(0, min(cap, base × 2^n)), with a base of one second and a cap of five minutes. The jitter is the point: a control-plane restart drops every stream at once, and agents retrying in step would arrive together, each with a mutual-TLS handshake to pay for. Full jitter spreads them across the window. TestReconnectIsFreshHelloWithBackoff fails three dials, then drops the stream, and requires the agent to return with a new Hello reporting what it applied and to take a snapshot again. TestBackoffIsBoundedWithFullJitter requires every wait to fall inside its window and waits for the same attempt to differ.

Telemetry rides its own connection. Part 1's flow windows travel on a separate RPC, over a connection of their own, with their own jittered backoff. When a partition heals and the agent's buffered windows drain, that burst cannot queue a policy update behind it, and a failing telemetry path cannot take the policy stream down with it. Policy and evidence share neither a queue nor a fate.

The floor is the host's last good policy. None of this is safety-critical, because a host does not need the control plane to keep enforcing. The kernel rules outlive the agent's process, the agent persists every policy it applies successfully, and on start it re-applies that policy from disk before dialing anything: Part 2's fail-static. Its Hello then reports the persisted version, and the server answers with a snapshot all the same, so a restarted agent converges by the same path as a reconnected one. The control plane's side matches. Rendered policy, versions, and each workload's applied version and sync state live in the database, a restart preserves all of them, and the render the control plane runs at startup changes nothing. TestSyncStreamEndToEnd restarts the control plane against the same database and requires the render after restart to change no workload, every version to be exactly what it was, and a reconnecting agent to receive that version as a snapshot.

So a distribution failure is a management concern, never a safety concern. A partition delays convergence and shows on the applied-against-latest read as a workload behind or offline, which Part 2's verdict refuses to count as evidence. It never leaves a host with less than the policy it last acknowledged, and never with a policy no render produced.

Next: who is on the other end

Every stream in this post is mutually authenticated, and the workload on it is whatever its certificate says, never what its messages claim. That certificate lives for a day by default, so renewal is on the critical path from the first version: an agent that cannot renew cannot hold a stream. That is Part 5, Short-Lived Certificates for an Agent Fleet: Enrollment and Renewal.

Summary

• Policy moves as desired state: each workload has one persisted target, and a missed message costs time, never correctness. Convergence is one read per workload, applied version against latest, and Part 2's verdict reads it.
• Every stream opens with a full snapshot whatever version the agent claims, and a delta never crosses a session, so a delta applied to the wrong base cannot be constructed.
• Versions are per workload, monotonic, and advance only on real change. No global generation reaches the agent, so one workload's recompile disturbs no other.
• Updates are pipelined and applied in order, each atomically: APPLIED or FAILED, never partial. A failure keeps the last good version, and the server answers with a snapshot, never a retried delta.
• A render announces changed workloads on a database channel delivered on commit, and the replica holding each stream computes that stream's delta and pushes it. Nothing polls, a lost announcement costs one reconciliation, and a directed resend rides the same bridge, best-effort and observed.
• Reconnects are fresh Hellos under full-jitter backoff, telemetry has its own connection, and the host enforces its last good policy from disk throughout.

Next in the series: Short-Lived Certificates for an Agent Fleet: Enrollment and Renewal.