Native Firewalls as the Enforcement Plane: Compiling Policy Atomically

Part 7, the last, of Microsegmentation from First Principles, a series that works through the design of Innerwall, an open-source microsegmentation platform, one problem at a time.

Part 6 ended by naming what six parts had taken on trust. Simulation's evidence, desired-state versions, and the stream that can't be severed all assumed the host's firewall enforces exactly the chain it is handed. This post opens the box. What makes the executor trustworthy: why the kernel's own firewall, and what must the agent guarantee when it programs one?

Why the kernel's own firewall

Enforcement can be built two ways. The agent can ship a datapath, as a kernel module, a packet-processing proxy, or an overlay, or it can program the packet filter the host already runs. Innerwall's record (ADR-0003) rejects the first in one sentence: a custom datapath "adds performance risk, kernel-compatibility burden, and an opaque failure mode to every host." The last three words carry the argument. A firewall that fails opaquely fails at the worst moment, in a way nobody on the host can read.

So the agent programs nftables, which every mainstream Linux host already ships. The kernel evaluates every packet, and the agent writes rules and reads what the kernel reports. That is another way of saying that nothing of the agent's stands in the packet path: a slow or crashed agent changes no packet's fate. When something goes wrong, the operator reads it with the host's own tools, and the record makes that a design goal: "nft list ruleset is a complete debugging surface".

The price, accepted for L3/L4 segmentation, is that enforcement expresses only what the native filter can. The real hazard is different, and ADR-0003 names it too: changing firewall state other software also manages is "a classic source of rule-clobbering bugs and partial-application outages." Ownership and atomicity answer it.

One table, one transaction

Every rule the agent installs lives in one inet table named innerwall; no other table is read, written, or flushed. A full apply is a single script handed to nft -f:

table inet innerwall {}
delete table inet innerwall
table inet innerwall {
	# sets, then the one base chain
}

The first line declares the table (a no-op when it exists, so the delete never fails on a fresh host), the second deletes it, and the third recreates it with the new content. nft -f commits the whole script as one kernel transaction, so the host runs the previous ruleset or the new one, never a mixture. If the kernel refuses any line, it refuses the script, and the previous ruleset stays exactly as it was.

That refusal is the kernel half of Part 4's acknowledgement. Part 4 said an update is APPLIED or FAILED, never partial, and that a failure keeps the last good version. A refused transaction changes nothing, so the agent can report FAILED and stay, provably, on the version it last acknowledged.

The table's one base chain hooks input at priority filter + 10, ten after the conventional filter priority, so the host's own filter chains run first. In nftables, an accept ends a packet's journey through one base chain only. Every base chain on the hook must accept the packet, and a drop in any of them is final. The record draws the conclusion: "the agent adds constraints and removes none." Nothing the agent installs can open what the host has closed.

Every line in that chain came from an earlier part: Part 2 drew the order and the terminal rule, and Part 6 showed why the established-and-related accept keeps the agent's own stream out of every policy's reach. This post adds how it is installed, changed, and numbered.

Sets carry the churn

Each resolved rule's peers live in two interval sets, one per address family. The set's name is the resolved rule id with every character outside letters, digits, and underscore replaced by an underscore, so r1/tcp becomes r_r1_tcp_v4 and r_r1_tcp_v6 (rule ids abbreviated, as in Part 3). The authored rule reads back from the kernel's own listing.

Part 3 established peer membership as the quantity that churns, and said a peer-only change would land on the host as set updates rather than a rebuilt ruleset; Part 4 carried such changes as small deltas. Here is the last leg. When every change between the installed policy and the next one adds or removes peers, the agent renders only element changes, in one transaction:

add element inet innerwall r_r1_tcp_v4 { 10.0.1.25/32 }
delete element inet innerwall r_r1_tcp_v4 { 10.0.1.23/32 }

Every rule and every other set stays untouched. Any other change (a port, a protocol, a rule added or removed, the mode) is a full rendering. If the kernel refuses a set update, the agent falls back to one full rendering before it reports the apply as failed, so a refusal costs one replacement of the whole table and never leaves the host wedged on half an update. A delta is only compression; the full rendering is always there.

Marks attribute, and never lie

Part 2 introduced the connection mark: each rule's accept writes that rule's value into the agent's half, and the collector reads it back to say which rule admitted a connection. Precisely, the agent claims bits 16 through 31 of the 32-bit mark, mask 0xffff0000, and nothing outside them. Every write keeps the low sixteen bits exactly as they were:

ct mark set (ct mark & 0x0000ffff) | 0x00010000

A host tool that uses the low sixteen bits coexists without configuration; one that uses the high bits conflicts, and the record states that as its boundary. The network-namespace suite tests that directly: a foreign table hooked before the agent's chain marks every new connection 0x2a. The suite requires an accepted connection to carry the rule's value beside it, 0x0001002a, with attribution resolving and the foreign bits surviving every write.

Time is the subtler problem. The kernel keeps an established connection's mark across policy changes, and the collector often learns a connection's rule when it closes, many versions later. Until 2026-10-05, values were assigned in canonical rule order and recomputed for each policy. Adding a rule that sorted earlier renumbered every rule after it, so a connection admitted under version 12 could be credited to whichever rule held its number in version 13. A conformance audit reproduced the misattribution, and the record was amended to replace it.

Values now come from a persistent allocator, stored beside the policy file:

• A rule keeps its value for life. Every version that carries the rule gives it the same value, and nothing renumbers.
• A new rule takes the lowest value never handed out.
• A freed value is recycled last. When a rule leaves the policy, its value is freed but not forgotten. No other rule gets it until all 65,534 values have been handed out once, and then "the value freed longest ago goes first", never one the installed policy holds.

Classification resolves a mark through that allocation, not the installed policy's order, so a connection admitted under version 12 is still credited, three versions later, to the rule that admitted it. If that rule has since been removed, the connection is attributed to the removed rule by its own id, in the record's words "never to another rule." The namespace suite holds connections open while an earlier-sorting rule is added and their own is removed, and requires both attributions to stay true.

The value 0xffff is never a rule's. It is Part 2's would-block mark, the one line simulation changes, written with the same masked write. That leaves 65,534 values, and a policy with more rules than that is refused before the kernel sees it. Reuse is the documented edge: once 65,534 values have been handed out over an agent's life, a connection older than the reuse of its value is credited to the value's new holder.

Visibility had a bug, and the fix is a lesson

Part 2 described the visibility mode this way: "In visibility the owned table is installed with no verdict: nothing in it can decide; observation is the connection tracker's job". The rendering behind it was then an empty table, which assumed something else on the host had switched connection tracking on.

Part 1 had stated the condition without noticing what it implied: the tracker holds state "once tracking is active on the host, which any stateful firewall rule requires". The kernel registers connection tracking in a network namespace only once a loaded ruleset references it. On a host where no other ruleset did, such as a minimal image with no firewall of its own, the amendment of 2026-10-08 records the outcome plainly: "visibility observed no flows at all, and nothing reported an error."

That is the failure this series fears most: an empty map that reads as no dependencies at all. The fix is one line. The visibility table now holds one base chain at the same priority, with a policy of accept and a single rule:

ct state new counter accept comment "innerwall observation visibility"

It references connection state, so the kernel tracks connections, and since an accept in one base chain never skips another, the chain still decides nothing. Part 2's sentence is as true as it was: no verdict, and nothing in the table can decide. What changed is that observation turned out to need a line in the kernel too. TestVisibilityObservationInNamespace holds the fix. It runs where nothing but the owned table references connection tracking, and requires connections to arrive in closed flow windows as OBSERVED, nothing dropped. It fails on the old rendering.

Disk before kernel

Part 6 promised to say what the host does when everything else is gone: the control plane, the stream, the daemon. Fail-static is the answer, and the record states it twice. ADR-0011 makes it a safety property: the last-acknowledged ruleset stays enforced across restarts and reboots. ADR-0020's sixth decision, "Rules outlive the daemon", makes it a mechanism. The mechanism is an order of operations, amended on 2026-10-05 after the same audit found the agent applying first and persisting second. That order could acknowledge a version that had no copy on disk.

Every apply now runs this sequence:

1. The mark allocation is written durably, first: it only ever grows, so the policy beside it always finds a value for every rule.
2. The canonical policy and its version are written the same way. A durable write goes to a temporary file at mode 0600, because policy discloses network structure, under a magic line and with a SHA-256 checksum, so a foreign or torn file is refused rather than loaded. The temporary file is synced, renamed into place so a reader sees the old file or the new one, never a mixture, and then the directory is synced, because a rename is not durable until the directory holding it is.
3. Only then does the kernel transaction run.

The agent acknowledges APPLIED only when both succeeded. A persistence failure is acknowledged FAILED with the kernel untouched, so the installed policy, the persisted one, and the last acknowledgement still agree. A kernel failure after persistence is also FAILED, and it leaves the file one version ahead of the kernel. Part 4's desired-state model makes that benign. The file holds the target, re-applying it is idempotent, the control plane answers a failure with a snapshot, and a reboot inside the window restores the desired version, not an older one.

When the daemon exits, the table stays. On start, the daemon loads the file and re-applies it as a full rendering before it dials anything: Parts 2 and 4 asserted that, and this is what it is made of. The namespace suite stops the daemon and requires denied traffic still to drop, then deletes the table as a reboot would, restarts the agent, and requires the drop back before any control plane exists.

There is one way out, and it is local. innerwall-agent down deletes the owned table and nothing else. The agent's usage text puts the trust model in four words: "root outranks the platform". ADR-0011 says the same in a sentence: "Host operators outrank the platform on their own machine". Root can't be resisted, so the exit is documented, exact, and honest. The command prints ENFORCEMENT REMOVED FROM THIS HOST., confirms that every other table is untouched, and says the persisted policy stays on disk for the next start to re-apply: "To keep enforcement off, stop the daemon too."

Fail-static is now fully mechanized: a transaction that can't half-succeed, a file that can't lag the acknowledgement, a restart that re-applies before connecting, and an exit only root can take.

The loop the series opened, closed

Part 1 counted loss: every record the agent's buffer dropped was counted, and the count rode the heartbeat. Part 2 refused to let a count since process start clear a time range, and stated the requirement in one sentence: "A count says evidence was lost; only an interval says which evidence."

On 2026-10-04, five days after Part 2 was published, the repository shipped that sentence. An evidence gap is an interval, from one instant to another, with a kind, a source, and a count where one is known. As the code puts it, what was lost inside it "cannot be recovered, only bounded in time." Four kinds arrived that day, each a place Part 1's pipeline could lose evidence. source_overrun is the kernel's full socket buffer, Part 1's ENOBUFS. source_restart covers a source that was down until it subscribed again. buffer_overflow is closed windows dropped from the bounded buffer. dump_truncated is the table dump at subscribe, Part 1's fix for connections that predate the collector, cut at its bound. The next day's audit added a fifth, window_overflow, for an open window that has reached its limit of distinct keys (ADR-0019, amended 2026-10-05). Every loss the agent knows about becomes a gap.

Gaps ride with the next window and are put back when its stream fails. They are stored in flow_gaps beside the windows, and retention deletes a gap only after it has ended before the horizon, "so a gap is kept at least as long as any window it could describe." The storage record's amendment of 2026-10-04 states the rule the readers enforce: "A range whose evidence has a gap in it is not complete evidence". The simulation review's verdict gained the failing condition Part 2 argued for, evidence-gaps. Any gap that intersects the judged range, on any workload in scope, refuses "safe". The verdict line names the workloads and the interval, and the bare counter is demoted to a note. Part 2 said that until loss carried its interval, a nonzero count could only mean incomplete evidence. That "until" has passed: now the interval decides.

That is the mechanism. The meaning is that the series has been one argument from its first paragraph: evidence before enforcement, and evidence honest about its own gaps. A map that hides loss writes outages into policy; a simulation that hides loss certifies them. The system now applies that argument to itself. Its own verdict refuses its own incomplete evidence, and names exactly when and where the evidence ran out.

Closing the series

Observe before you enforce, because an allowlist is a claim about every flow and only observation can support it. Test the artifact, not the source: simulate the exact ruleset enforcement will install, with one line changed. Name workloads, not addresses, with identities granted by the control plane and never claimed by the host. Ship state, not commands, so a missed message costs time and never correctness. Make credentials routine, short-lived and renewed early, so losing one needs no action. Enforce the half that pays, at every destination, where the evidence was gathered. Then compile it all into the kernel atomically: one owned table, one transaction, marks that never misattribute, and a file on disk before the kernel sees anything.

The series is complete. The work continues in the repository, where every claim in these seven posts can be checked against code, tests, and design records, and on the project page.

Summary

• The agent programs the kernel's own firewall instead of shipping a datapath, which ADR-0003 rejects for adding "an opaque failure mode to every host"; nothing of the agent's stands in the packet path.
• One owned table is replaced in one nft -f transaction, previous or new and never a mixture, which is the kernel half of FAILED. At filter + 10, the agent adds constraints and removes none.
• Per-rule named sets carry rule ids and absorb peer churn as element updates, with one full rendering as the fallback.
• The agent writes only the upper half of the connection mark, and a persistent allocator keeps each rule's value for life, so a connection is credited to the rule that admitted it, even after that rule is removed.
• Policy reaches disk durably before the kernel, APPLIED means both, the daemon re-applies before dialing, and only innerwall-agent down removes enforcement.
• Known loss is now an interval: five kinds, stored beside the windows and outliving them, and any gap in a review's range fails "safe to enforce".