Simulation as a Deployment Stage: Running Policy Without Dropping a Packet

Part 2 of Microsegmentation from First Principles, a series that works through the design of Innerwall, an open-source microsegmentation platform, one problem at a time.

Part 1 ended with a candidate policy and an unsolved problem. The policy was authored from a flow map, reviewed for traffic that shouldn't exist, and generalized from address pairs to identities. It is still a hypothesis about R, the set of flows the application requires, and the evidence behind it has a hole that observation cannot fill. The flows in R ∖ O, needed but not seen during the window, are by definition absent from the map. A policy built from the map denies them.

Testing the hypothesis by enforcing it turns every member of R ∖ O into an outage, so between authoring and enforcement there has to be a stage that runs the policy against real traffic at zero blast radius: evaluate every packet, record what would have been refused, refuse nothing. This post argues that the stage is not a preview feature bolted onto a policy editor but a deployment stage, in the sense that staging is one for code: the artifact under test is the artifact that ships, it runs where it will run, and promotion out of it is a decision made on recorded evidence. Each of those properties is a design constraint, and each is easy to get subtly wrong.

Test the artifact, not the source

A simulation is evidence about a policy only to the extent that what it runs is what enforcement will run.

The natural way to build it is a dry run in the control plane: take the stored flows and the candidate policy, and ask of each flow whether some rule admits it. It is cheap, needs nothing on the hosts, and runs over history instantly. It is also static analysis. It evaluates the policy source with an evaluator written for the purpose, while the thing that will make decisions is a different evaluator: the host's packet filter, running the compiled ruleset. Every difference between the two is a place they can disagree. An interval set built wrong, a port range off by one, an address family missing its twin, a rule order other than the one assumed: each passes the dry run and fails in production, because the dry run never ran the compiled rules. And it has a second limit. It can only judge flows that were stored, which are exactly O, and the point of this stage is to reach past O.

The alternative is to make simulation a property of the compiled artifact. Innerwall compiles a policy in simulation mode to the same host chain as enforcement: the same named sets of peer addresses, the same per-rule accepts in the same order, the same connection marks written by each accept. Exactly one line differs, the terminal rule that catches whatever no rule admitted.

Here are the two terminal rules as the agent renders them for one policy:

# enforcement
log prefix "innerwall " group 200 drop comment "innerwall terminal enforced"

# simulation
log prefix "innerwall " group 200 counter ct mark set (ct mark & 0x0000ffff) | 0xffff0000 accept comment "innerwall terminal simulation"

The upper half of the kernel's connection mark belongs to the agent. Each rule's accept writes that rule's value there, which is how an accepted connection is attributed to its rule from the connection tracker alone. The value 0xffff is never a rule's: it means "accepted only because this is a simulation".

The one-line difference is not a convention a future change could quietly erode. It is a tested invariant. A unit test in the agent's nftables backend, TestSimulationDiffersOnlyInTerminalRule, renders one multi-rule policy in both modes and fails unless the outputs have the same number of lines and differ in exactly one: the enforced drop on one side, the marked simulation accept on the other, both logging to the same group. The two modes share one rendering function, so a change to the chain structure changes both, and the test makes that a guarantee rather than a hope.

What a would-block event is

The terminal rule logs a packet. The agent's collector subscribes to that log group and turns each logged packet into one attempt. Under enforcement the packet was dropped, and the attempt is recorded as BLOCKED. Under simulation it was accepted, and the attempt is recorded as WOULD_BLOCK. The counting is identical; only the decision differs.

The decision is part of the key the collector aggregates on. Part 1 gave that key as source, destination, destination port, and protocol; the full key also carries the decision and the matched rule. Observed, allowed, would-block, and blocked records for one address pair are different records, even in the same window, so a workload that changes mode mid-window produces one record per decision.

Under simulation the would-be-refused connection still completes, so the connection tracker reports it like any other, and the would-block mark keeps it from counting twice. When the tracker announces a new connection carrying 0xffff, the collector ignores the announcement, because the log path already counted the attempt. When the connection closes, its bytes go into the same would-block record. One caution follows. A simulated TCP connection is accepted at its first packet and is usually logged once, while under enforcement the same client retransmits into the drop and logs several attempts. Read would-block counts as connections that would have failed, not as a forecast of blocked counts.

Now the payoff. A month-end reconciliation job connects to a ledger database once a month, and observation ran for two weeks. By Part 1's periodicity model, a process with period T is seen in a window W with probability min(W / T, 1): here 14 / 30.4 ≈ 46%, roughly an even chance the job is in R ∖ O with no rule in the candidate policy. Simulate for thirty days and month-end falls inside the window with probability 30 / 30.4 ≈ 0.99; for thirty-one days, 1. When the job runs, its connection reaches the terminal rule and is logged, marked, and accepted. The job succeeds, and the flow map gains a would-block record naming the job's host, the ledger workload, and port 5432: R ∖ O caught live, a log line where enforcement would have produced an outage.

Reviewing the evidence

A simulation review answers one question: what would this policy have blocked, and for whom? "What" is a peer and a service. "For whom" is how many workloads under the policy saw it; a would-block pair that reached one host of forty is a different finding from one that reached all forty.

Innerwall's flow storage is shaped for that question. Its rollup reads the stored windows of a label scope's workloads over a time range, optionally filtered to one decision, and groups them by resolved peer and service, with summed counters, first and last instants seen, and the distinct number of workloads in scope that saw each group. It is bounded: the top groups by connections or by recency, with the totals and group count of the whole, a flag when truncated, and the span of windows the range actually covered, so a range with no data cannot pass for a range with no findings. The peer-and-service grouping with per-group workload counts is the newest part of the storage design (ADR-0019, decision 4, as amended 2026-09-28). The existing groupings could have produced it only by assembling rows on the client, and the design rules that out: the store groups, the client displays. Peers are the ones resolved at ingest, as Part 1 argued, so a would-block from three weeks ago names the workload and labels true three weeks ago.

Each would-block group gets one of the two outcomes Part 1 applied to the flow map:

• An intended dependency. The month-end job is supposed to reach the ledger; the policy was incomplete, and the fix is a rule. Adding it renders a new policy version, and would-blocks recorded before the workloads applied it are evidence about the previous hypothesis. The range that matters restarts from there.
• A finding. The connection should not exist: a decommissioned client, a misconfigured scanner, something worse. It stays unadmitted, and enforcement will block it deliberately. Simulation has done its other job, surfacing O ∖ R that was rare as well.

Turning a dependency into a rule written against labels was Part 1's "generalize" step and is Part 3's subject.

"Safe to enforce" has to be composed, not stored

At the end of a review someone wants one answer: is this scope safe to enforce? The tempting implementation is a flag, set when the review looks clean and read at promotion time. That is the wrong shape. A flag records a judgment about evidence at one moment, and the evidence keeps moving: new windows arrive, a workload falls behind on its policy, an agent goes offline, a host joins the scope.

"Safe to enforce" has to be a verdict composed at read time from the evidence as it stands, and refused whenever any one of these holds:

1. Any would-block pair in the range. Enforcing would drop it. The scope's rollup, filtered to would-block and grouped by peer and service, must report zero groups.
2. An empty scope. A selector matching no workloads, or a range covering no windows, yields zero would-blocks trivially. Nothing was tested. The scope's workload list must be non-empty, and the rollup's covered span non-null.
3. A workload whose evidence is not about the candidate policy: behind on its version, still converging, or offline. Each workload read carries the version it last applied against the latest rendered for it, its sync state, and its last-seen instant. A workload behind or converging is running some other ruleset, so its would-blocks, or their absence, test a different hypothesis; an offline one reports nothing, and silence is not cleanliness.
4. A workload still in visibility. It runs no policy and produced no verdict at all. Its mode is on its read.
5. A scope already fully enforced. Nothing is being simulated; the question has no subject.
6. Known evidence loss. Here Part 1's last loose end comes due. When the agent's bounded buffer overflows it drops the oldest windows and counts every dropped record; the count rides each heartbeat and appears on the workload read. On a flow map, absence becomes a deny. In a simulation review, absence becomes a false "safe": zero would-blocks over incomplete evidence is absence of evidence. So any workload in scope reporting dropped records blocks "safe", full stop.

The counter reports records dropped since the agent process started, and a count since process start cannot clear a time range: it cannot say whether the loss fell inside the window under review or a week before it, and it restarts at zero with the agent. Before a verdict can ever scope loss to a range, known loss must carry its interval. A count says evidence was lost; only an interval says which evidence. Until then, the only sound reading of a nonzero count is that the evidence is incomplete.

Every condition is computable from reads that exist: the rollup over the scope, and each workload's mode, sync state, applied and latest versions, last-seen instant, and dropped-record count. None is a stored conclusion. And a verdict must carry its reasons, because the remedy for each failed condition differs: write a rule, wait for convergence, bring an agent back, finish moving the scope into simulation, or wait out a clean range.

Promotion is a recorded intent

Evidence belongs to the workloads that produced it. The review covered a set of workloads, resolved from a label scope at query time, and promotion should name that set by workload id rather than re-send the selector. A selector resolved again at enforce time resolves against the estate as it stands then: a host labeled into the scope after the review, still in visibility and never simulated, would be swept into enforcement with no evidence behind it. Innerwall's mode-change request takes exactly one of a selector or a list of workload_ids; naming the ids says "enforce the workloads I reviewed, and nothing else".

The size of the set is stated, not assumed. Every mode change carries a required expected_match_count. If the selection resolves to a different number, the request is refused with both:

{
  "type": "urn:innerwall:problem:match-count-mismatch",
  "title": "Match count mismatch",
  "status": 409,
  "expected": 2,
  "matched": 3
}

That catches an estate that drifted between looking and acting. The selector preview resolves with the same resolver a mode change uses, so a preview and the change it precedes cannot disagree about what a selector means; they can differ only because the estate changed in between, which is exactly what the 409 reports. The answer to a 409 is to preview again and look at what changed, not to resubmit with the new number.

The response acknowledges an intent, not an outcome. The set is recorded at submission, by id, and convergence is read per workload from the same sync state the verdict uses.

Overriding "not safe" should exist, and must cost something. A mode change consults no verdict; it is a recorded intent over a set of ids. So the gate belongs to whatever composes the verdict and submits the change, and the requirement on it is precise: an override must name what will drop, the peer and service pairs and the workloads they reach, and require an explicit acknowledgment of that list. "Are you sure?" is not an acknowledgment. "These five pairs will be refused on these four workloads" is.

Enforcement stays opt-in and per scope. Mode is a property of each workload. Promotion moves a named set, never the estate, and the same mode change moves any workload back to simulation or visibility. In visibility the owned table is installed with no verdict: nothing in it can decide; observation is the connection tracker's job, and with the table in place a later mode change is a change of content, not an install.

What simulation promoted then stays enforced, because the agent fails static. The kernel rules outlive the agent's process, the persisted policy is re-applied from disk at start before the control plane is dialed, and losing the control plane changes nothing.

How long to simulate

Part 1's stopping rule for observation had three parts: the rate of new dependencies has decayed, the window spans the longest business cycle you can afford to wait for, and rarer paths are enumerated rather than awaited. The same shape applies, with different economics. Simulation is cheap to run long: nothing drops, and a simulated scope costs the kernel the same evaluation enforcement would, plus a log line per would-be-refused attempt. There is no operational pressure to cut it short, so the window is bounded by the business calendar, not by nerve.

1. The would-block rate has decayed to zero, or to findings, and the verdict is clean over the whole window, not just today. Every remaining group has been reviewed and decided; a new one restarts the clock.
2. The window spans the longest business cycle you can afford to wait for, measured from when every workload in scope applied the current version. For most applications that means a month-end, and a thirty-one-day window is certain to contain one.
3. Rarer paths are exercised, not awaited. Trigger failover, quarterly reporting, and certificate renewal deliberately while the scope simulates. It is the safest moment there will ever be to run them: a missing rule shows up as a would-block, not a failed failover.

One assumption has ridden through this post unexamined: that the policy's rules stay true as the estate changes. A rule that names addresses is only as correct as the assignments it was written against, and those drift between simulation and enforcement as surely as label scopes do. Part 3, Labels, Not Addresses: Workload Identity in Network Policy, covers why rules should be written against identity, and what makes an identity safe to write policy against.

Summary

• A candidate policy is a hypothesis about R, and observation cannot see R ∖ O. Simulation tests it as a deployment stage, not a preview: it runs the artifact enforcement will run, where it will run it, which a control-plane dry run over stored flows cannot.
• Innerwall compiles simulation to the enforced chain with exactly one line changed: the terminal rule logs, writes the reserved would-block mark, and accepts instead of dropping. A test asserts the one-line difference.
• A would-block is one logged packet, counted as enforcement counts a drop and kept apart by the aggregation key, so a rare path that fires during simulation is recorded rather than broken.
• Review asks what the policy would have blocked and for whom, by peer and service with the workloads each reached. Each pair becomes a rule or a finding.
• "Safe to enforce" is composed at read time and refused by any would-block, an empty scope, a workload not running the candidate or in visibility, a scope with nothing left to test, or any reported loss. A count says evidence was lost; only an interval says which evidence.
• Promotion names the workloads that produced the evidence, states the expected count, and records an intent; an override must name what will drop. Enforcement stays opt-in, per workload, reversible, and fail-static, after a simulation window that spans a business cycle.

Next in the series: Labels, Not Addresses: Workload Identity in Network Policy.