Skip to content

0067 — Configuration is immutable per instance; credentials and State are read live

Status: accepted; refines 0063 and 0051 for configuration Date: 2026-10-05

Decided together with truvity/sluis ADR 0036, which carries the same rule for that service; the two are written to be read side by side.

Context

An audit installation is one main configuration plus the documents it points at: deployment, grants, workloads and the catalogues. Until now nothing said how those reach a running instance or when an instance notices a change. On Kubernetes they are ConfigMaps; on AWS Lambda the question is sharper, because the function's code is the released artefact and the writer's ingest is an SQS queue whose failures drain to a dead-letter queue. A configuration that is fetched at start, or changes under a running instance, is a new way for the record path to fail or for two instances to disagree about a catalogue.

The truvity/policy contract (service.md section 1, config.md section 2) says configuration is a file, its path from an argument or one variable, and that schemas are authored with tests binding the Go structs.

Decision

  1. Configuration is immutable per instance; a change is a new set of instances.
  2. Kubernetes: checksum/* annotations on the pod template roll the Deployment when a document changes. A file a process reads once at start may be mounted with subPath: the checksum/* annotation replaces the pod on any change, so an in-place update would never be read. Mount a directory only where a file is meant to change in place (secret files read per call, signing keys).
  3. AWS Lambda: the documents ship as an immutable Lambda layer at /opt/audit/. The function code is the released zip, byte-identical and sha256-verified. A change is a new layer version, then a function update; AWS replaces all instances at once. No aliases and no canary ($LATEST).
  4. Server: a restart.
  5. Credentials and State are read live: signing and encryption key files, client secrets, projected tokens. These are not part of the configuration and rotating one is not a deployment.
  6. Configuration is a file everywhere, its path from an argument or one variable, AUDIT_CONFIG. The documents are the main configuration and its deployment, grants, workloads and catalogue sub-documents, each read whole; layering happens only when a document is rendered, never in the running binary.
  7. Versioning. Each document carries apiVersion; absent means v1. A binary accepts N and N-1 and converts N-1 on load. Schemas stay authored by hand, with tests that bind them to the structs.
  8. Amendment, 2026-10-05 (owner decision D3). The group is <product>.truvity.github.io/<kind>/vN: audit moves from truvity.github.io/<kind>/v1 to audit.truvity.github.io/<kind>/v2, and the schemas' $ids move with it (.../schemas/v2/config/). The loader reads both: a version-1 document under the old group, or with no apiVersion, is validated against the frozen version-1 schema, converted, validated against version 2's and read, with a deprecation warning. Version 1 is read for one minor and then removed (D1).
  9. Secrets. A declared secret name is delivered by an environment variable, a mounted file, or a declared secret source (the amendment to contract section 5), by the same rule as in sluis.
  10. Amendment, 2026-10-05 (owner decision D4). A field that holds a secret is named ...Secret and holds the secret's name; the file's one secrets: {source: env|file|ssm, root} says how to find it. ...Env fields (passwordEnv, credentialsEnv, tokenEnv) are deprecated and exist in version 1 only. On AWS Lambda a secret is never taken from the function's environment: it is a SecureString in SSM under a root (/audit/<instance>/private/config/<name>), read with the function's own role, which the Pulumi library grants for that root only. On Kubernetes the chart projects each name as a file (source: file).

Consequences

  • A pre-deploy catalogue/version guard is added. A new instance set must not start against a catalogue or record version that the existing records and decoders do not cover (0051: decoders forever); the guard runs before the layer is applied and fails the deploy, not the ingest.
  • A bad configuration is caught at the start of an instance, in front of the queue, instead of surfacing as messages draining to the DLQ.
  • On Lambda a configuration change remains a manual Pulumi apply, as in sluis.
  • The previous layer version is the rollback.
  • The N-1 converters are a cost carried for as long as the window exists.

Alternatives considered

Configuration baked into the zip. Rejected: build, release and run are no longer separate, and the deployed code is not the release.

S3 documents pinned by versionId. Rejected: a fetch on cold start, extra IAM, and a new init failure. For audit that failure sends the SQS ingest to the DLQ, which is the worst place to find a configuration fault.

AWS AppConfig. Not rejected on merit: staged rollout with alarm rollback is real value. It is an extension layer, about six resources per estate, and AWS-only. Deferred; it can be added later.

Environment-only configuration via caarlos0/env. Rejected: Lambda's 4 KB environment cap, lists turned into _0_ variables, and a conflict with contract section 1.

Policy and catalogues in SSM. Rejected: 4 KB / 8 KB per parameter, and a tree of parameters is not read atomically.

Lambda aliases with CodeDeploy canary. Deferred; the owner chose all at once.

Runtime reload of configuration. Rejected: a window in which instances disagree, and more code. Replacing instances is cheap because State is external.