Short-Lived Certificates for an Agent Fleet: Enrollment and Renewal

Part 5 of Microsegmentation from First Principles, a series that works through the design of Innerwall, an open-source microsegmentation platform, one problem at a time.

Part 4 ended on the other end of the stream. Every stream is mutually authenticated, and the workload on it is whatever its certificate says. This post covers where that certificate comes from and how it stays valid.

Certificates as routine

For people, a certificate is an event. Someone generates a request, someone approves it, and the result lives for a year in a file that someone has to remember to renew. That doesn't scale to a fleet of agents. Hundreds of hosts arrive from images and pipelines with nobody watching, each needing a credential before it can receive a single rule. For an agent fleet, certificates have to be routine: issued without a human, renewed without a human, and safe to lose.

Two questions structure the rest:

1. How does the first credential arrive without trusting whoever asks first?
2. How does it stay valid without an operator in the loop?

The design record states the stakes of the first in one line: "The only hard identity problem is the first certificate." After that, everything is rotation.

Enrollment without trust on first use

Trust-on-first-use accepts whoever connects first and alerts if the key changes later. For a system whose policy discloses network structure, the open window is the problem: anyone reachable can enroll as anything, receive policy, and learn the topology. Innerwall closes the window by requiring a secret the operator minted before the host existed.

The exchange has three parties, and each is trusted for exactly one thing.

The agent contributes a public key. It generates its key pair locally, and the private key never leaves the host. What it sends is a PKCS#10 signing request, which proves possession of the key and carries nothing the control plane will use beyond it. The subject, alternative names, and extensions in the request are discarded. Part 3 covered the hostile-request test that holds that rule.

The token is the entire authorization to enroll. The agent presents it with the request, and the control plane checks it before parsing the request at all. The enrollment code gives the reason: "an unauthenticated caller learns nothing about what the signing boundary accepts."

The identity comes from the control plane. Once the token is valid, the control plane assigns a fresh UUID, signs a certificate binding it as innerwall://workload/<uuid> with a 24-hour lifetime, and returns the certificate with the chain to verify it. It assigns labels from the token's scope and records the token's use. The agent checks that the certificate verifies against the returned chain and names the workload the response claims, then stores both.

The channel is server-authenticated TLS. The agent has no credential yet, so it can't prove itself at the transport layer, but it still must not hand its token to an impostor. The operator distributes the authority's certificate together with the token, and the agent uses it as the trust anchor for that first connection. The design record puts the symmetry in two clauses: "The token proves the agent to the control plane; the anchor proves the control plane to the agent."

Because the token carries the whole authorization, its mechanics matter. A token is iw_ followed by 32 random bytes, and the fixed prefix lets a secret scanner recognize a leaked one without knowing which deployment minted it. The control plane stores only its SHA-256 digest, plus a short listing hint so tokens can be told apart, and shows the plaintext once, at mint. A token carries a label set, an expiry of 30 days unless the operator sets one, and a revocation timestamp. It may enroll any number of workloads while valid, so it can be baked into an image. Each use is recorded, never consumed. A rejected token reports precisely why: malformed, unknown, expired, or revoked. That looks like an oracle, and the design record answers the worry directly: the operator who minted the token needs to know, "and an attacker guessing values learns nothing from 'unknown' they did not already know." TestEnrollRejectsBadTokens drives all four refusals.

Revoking a token stops future enrollments and nothing else. Workloads it already enrolled keep their identities, because after enrollment the token is no longer what authenticates them.

Two doors, split on authentication

Before enrollment a workload holds a token. After, it holds a credential. The wire contract makes those two states two services:

• EnrollmentService is reachable with a token and nothing else. It has one method, Enroll, and it is the only surface a workload can reach before it has an identity.
• AgentService is reachable only with the issued credential, over mutual TLS. Everything else lives there: the sync stream, flow reports, and renewal.

Both share one TLS listener. The listener asks for a client certificate and verifies it when one is presented, but doesn't require one at the transport layer, since an unenrolled workload has none. A server interceptor then enforces the boundary by service name. Exactly one service is exempt. Every other method requires a verified chain whose leaf carries a parseable workload identity, which the interceptor places in the request context, and handlers read identity from there and nowhere else.

The point of drawing the line per service is what it does to the next change. With per-method rules, every new method needs its own decision, and one forgotten decision exposes a method to anyone holding nothing. With per-service rules and a default of deny, a new method on AgentService inherits mutual TLS without anyone deciding anything, and a new unauthenticated service needs a deliberate edit to the interceptor's one exception. The design record names that edit as the review point. TestAgentServiceRequiresClientCertificate holds the default.

Expiry instead of revocation

The second question is the one this post exists for: how should a credential stop working? The conventional answer is revocation, and revocation is a distribution problem. A revocation list or an OCSP response has to reach every verifier, and be consulted by it, faster than an attacker can use the stolen credential. A verifier that is stale, unreachable, or configured to proceed on a missing answer accepts a revoked credential and says nothing. Part 4 was a whole post about how hard correct distribution is, and that was for policy, where a missed message costs time. A missed revocation costs the thing revocation was for. The design declines to solve distribution twice.

It uses two facts instead. A workload credential lives for a day. Every stream ends at the control plane. Workload credentials are client certificates, presented only to the control plane. No agent verifies another agent's credential, so there is no fleet of verifiers to keep current.

Here is what that buys, stated precisely. A credential that is copied and not renewed is dead within a day. That covers a leaked backup, an image snapshot, a certificate file read after the fact. Nobody has to be told, and no list has to arrive anywhere. A credential that is renewed stays alive only by renewing, and every renewal goes through the control plane. That is the honest shape of the trade. Revocation by distribution has been exchanged for a single chokepoint, and the chokepoint is where a renewing thief has to keep showing up. Each renewal stamps the workload's renewal instant, which is readable on the workload. That leaves a trace. It does not raise a flag: nothing yet tells an operator that two parties are renewing one identity. Turning the chokepoint into a lock is the remaining piece: the design record pairs short lifetimes with a control-plane deny-list of agent ids.

Missed renewal fails closed. The daemon never enrolls itself. A credential that expires ends the workload's standing, and getting back in costs what getting in cost: a fresh, scoped token presented by an operator. Everything else in this section runs the same way. The automatic direction is the safe one. Losing a credential needs no action, and regaining one needs a person. The host keeps enforcing its last good policy throughout, which is Part 2's fail-static, so expiry costs updates and evidence, never enforcement.

Clock skew gets five minutes, not a disabled check. A host whose clock runs slightly slow would reject a certificate issued a moment ago as not yet valid. The fix is bounded. Every issued certificate's validity starts BackdateNotBefore = 5 * time.Minute in the past. Validation stays on, with a fixed tolerance in one direction. TestSignVerifiesAgainstBundleAndHonorsTTL signs a 24-hour credential and requires it to verify against the bundle, end exactly 24 hours after issue, and start exactly five minutes before.

Renewal is the critical path

Short lifetimes move the risk from compromise to availability. An agent that cannot renew cannot hold a stream, and a day is not long. So renewal is automatic, early, and observable.

When. The daemon schedules its first attempt at the two-thirds point of the credential's lifetime, pulled earlier by a random amount of up to a twelfth of the lifetime. Jitter only ever moves the attempt earlier, so the design's actual guarantee is stronger than a schedule. The first attempt always fires with more than a third of the lifetime remaining, and the entire last third is retry margin, not renewal window. For a 24-hour credential the first attempt lands roughly fourteen to sixteen hours after issue, and eight hours remain for retries. TestRenewAtWindow draws 500 renewal points and fails if any falls outside that band or leaves less than a third. The jitter has the same purpose as Part 4's reconnect backoff. A fleet enrolled from one pipeline in one afternoon would otherwise renew in the same minute every day.

How. Renewal is an authenticated re-CSR on AgentService. The agent dials with its current credential, which is still valid, and sends a fresh signing request. The control plane binds whatever identity the connection's certificate carries. The request can't choose the workload any more than an enrollment request could. The new certificate gets a fresh serial and a fresh day. The agent writes it to disk first and memory second, so a crash between the two leaves the newer credential on disk. Its TLS configuration then presents the new certificate at the next handshake. The current stream isn't dropped, because the old credential is still valid and the two overlap. TestRenewerFiresAtThreshold runs the schedule on a fake clock. The first wait lands in the band. The first attempt fails and retries under bounded backoff. The success then shows up in the holder, on disk, and in what the TLS configuration presents.

When it fails. A failed attempt retries under jittered exponential backoff, on a one-minute base capped at an hour, and never waits past expiry. The resilience isn't the point, though. The point is visibility. The renewer keeps the reason the latest attempt failed, and every heartbeat carries it as credential_renewal_error, clearing it when an attempt succeeds. The control plane stores it, and the workload read reports the credential's state as one of renews, renewal-failed, or expired, with the expiry, the last renewal instant, and the reason. An operator sees a workload heading for expiry while it still has hours left, not after its stream has gone. This is Part 1's dropped-records counter again. A condition that will invalidate what the system tells you is surfaced before anyone trusts it. expired is derived from the stored expiry, so it shows even after the agent has gone quiet.

Two edge tests close the loop. TestRenewerRenewsImmediatelyWhenLittleRemains hands the renewer a credential that is already mostly spent and requires it to renew at once rather than schedule. TestRenewerReportsExpiry hands it an expired one and requires the opposite: no attempt with a credential that can no longer authenticate, just the expiry error, which ends the daemon and leaves the host to an operator with a token.

Custody of the signing key

Everything above rests on one private key. In version 1 the signing authority is embedded in the control plane, behind an interface with two operations: Sign(csr, identity, ttl) and Bundle(). Enrollment and renewal are written against the interface and know nothing about where the key lives. The default backend keeps a key and a self-signed root on the control-plane host's disk, created by innerwall ca init with the key at mode 0600 in a 0700 directory.

The embedding buys operational simplicity. There is one binary to deploy, and the control plane, already the one place every stream ends, is also the one place every credential begins. The root lives ten years and each workload credential a day, for a reason the design record gives in one sentence: rotating a root means re-enrolling every workload, while rotating a leaf costs one round trip.

The custody record handles the one awkward fact. The control plane keeps all durable state in its database so replicas are interchangeable, and the signing key is the exception. A key that everything with database access can read is not a signing boundary. So the authority directory is operator-provisioned configuration, copied identically to every replica like the database connection string. Backup and rotation of that directory are the operator's job.

What the interface defers is the backend. A signer backed by a secrets manager, a KMS, or an HSM replaces the file authority without touching enrollment, and removes the key from the replica's disk entirely. Two rules survive the swap. Every backend must read requests through the one function that returns only the public key. And enrollment checks that whatever comes back names the identity it asked for, "whatever the backend."

Next: what to enforce first

Identity, policy, distribution, and credentials are now settled. A workload is who its certificate says. What it may reach is rendered from labels it cannot choose, delivered exactly, and enforced fail-static. The open question is scope. A first release could enforce in both directions, and Innerwall's enforces only inbound. That is Part 6, Inbound First: Scoping Enforcement for a First Release.

Summary

• Enrollment has no trust-on-first-use window. The agent's key stays on the host, the request contributes only a public key, the token is the whole authorization, and the control plane assigns the identity over TLS anchored by a certificate shipped with the token.
• Tokens are prefixed for scanners, hashed at rest, scoped to labels, expiring, revocable, and reusable. Each use is recorded, and each refusal is precise.
• EnrollmentService takes a token and AgentService takes mutual TLS. Authentication is decided per service with a default of deny, so a new method cannot land on the unauthenticated side by omission.
• Day-long credentials replace revocation. A copied credential that is not renewed dies within a day, and a renewing one must keep returning to the single verifier. Expiry fails closed into operator-led re-enrollment, and clock skew gets five minutes of backdating.
• Renewal is automatic and early. The first attempt always comes with more than a third of the lifetime left, so the last third is retry margin, and every heartbeat carries the latest failure.
• The signing authority is embedded behind a two-method interface. Its key is the one durable thing outside the database, and an external signer can replace it without touching enrollment.

Next in the series: Inbound First: Scoping Enforcement for a First Release.