# tentaflake documentation (complete)

> Every documentation page of tentaflake v0.4.37 in one file. Pinned source: https://github.com/timfewi/tentaflake/tree/e21be475a3afc6e95c5087003edd0b7f780e4df4

---

# Overview

> How tentaflake turns a NixOS host into a reproducible platform for isolated AI agents.

Canonical page: https://docs.tentaflake.dev/getting-started/overview/

tentaflake is a generic NixOS flake template for running isolated AI agents on one machine. It provides reusable modules, agent builders, an installer, and operational checks for Hermes and ZeroClaw.

## Agent adapters

The [adapter API](https://docs.tentaflake.dev/configuration/agent-adapters/) exposes `mkAgent` alongside the compatible Hermes and ZeroClaw builders. The registered `openclaw` adapter is a stopped refusal scaffold: it creates no OCI container, and an explicit start fails. A reviewed image and actual model, Research, execution, and lifecycle acceptance are still pending.

The same API accepts generic digest-pinned OCI definitions with explicit commands, ownership, workspace, and resources. These workloads remain stopped pending capability admission; generic model and Research hooks are not accepted, and the generic contract requires `balanced`.

Use `tentaflake runtimes --json` to discover presets. [Installed-host onboarding](https://docs.tentaflake.dev/configuration/agents/#add-an-agent-on-an-installed-host) offers `agent template` and `agent validate|plan|import` for stopped additions. Import does not track files in Git, rebuild, activate, or start services; review and track the definitions before an explicit host update.

The [LLM broker](https://docs.tentaflake.dev/configuration/brokered-egress/#streaming-model-responses) offers bounded Chat Completions and Responses streaming when explicitly enabled per agent; streaming remains disabled by default and does not establish vendor compatibility.

## Infrastructure and evidence

The system is designed around three properties:

1. **Isolation:** each agent runs in a hardened container with explicit resources, workspace ownership, and networking.
2. **Controlled access:** a per-agent model broker, the pinned web research relay, and disposable workers narrow the paths from an agent to external services and host tools.
3. **Recovery:** declarative configuration, quotas, backups, and testable NixOS generations make failure modes observable and reversible.

The `dev` security profile preserves compatibility and is not a security boundary for untrusted, continuously running agents. Review [security profiles](https://docs.tentaflake.dev/security/profiles/) and the [threat model](https://docs.tentaflake.dev/security/threat-model/) before production use.

## Next steps

- [Install a dedicated host](https://docs.tentaflake.dev/getting-started/install/)
- [Configure your first agent](https://docs.tentaflake.dev/configuration/agents/)
- [Understand the architecture](https://docs.tentaflake.dev/concepts/architecture/)

---

# Tentaflake install guide

> Build the installer and provision a dedicated tentaflake host.

Canonical page: https://docs.tentaflake.dev/getting-started/install/

> Source: [`docs/00-install.md`](https://github.com/timfewi/tentaflake/blob/e21be475a3afc6e95c5087003edd0b7f780e4df4/docs/00-install.md) at `e21be475a3af` · docs version **0.4.37**

Tentaflake ships one bootable image: the installer ISO. The former live-agent
ISO and firstboot agent environment were removed from the core.

## Build

From the repository root:

```bash
nix build .#installer-iso
```

Or use the wrapper:

```bash
./scripts/build-iso.sh installer
```

The image is written below `result/iso/`.

After disk confirmation, the wizard imports `installer/disk.sh` for partitioning,
formatting and mounts. Its cleanup stops swap, LVM and encrypted mappings only
in the selected disk's stack. An LVM group spanning another disk is refused
before cleanup; migrate that group explicitly first. Other disks' swap and
volume groups remain active. The host VM regression imports this same file and
uses disposable disks to check filesystem types, preservation and refusal.
The wizard runs in the dedicated installer ISO, where `/mnt` is reserved for
its installation staging tree.

## Test in a disposable VM

From the contributor shell, the repository can build the ISO, resolve its
pinned QEMU and OVMF tools, create one 32 GiB sparse QCOW2 disk, and start the
interactive UEFI installer:

```bash
just e2e-installer
```

The recipe loads the pinned contributor tools automatically; entering a separate
`nix develop` shell is optional.

The VM receives no host block device. The installer can erase only
`/var/tmp/tentaflake-e2e-<user>/tentaflake.qcow2`, and still asks for
confirmation in its TUI. The first boot uses the ISO once and then prefers the
installed disk. To boot the installed VM again without the ISO:

```bash
just e2e-run-vm
```

Set `TENTAFLAKE_E2E_DIR` to an absolute path to keep this disposable VM state
elsewhere. The scripts deliberately provide no automatic reset or deletion.

## Resolve the USB device

List block devices and identify the USB drive by size and transport:

```bash
lsblk -o NAME,SIZE,TYPE,TRAN,MOUNTPOINTS
```

Do not infer the target from an example device name. The next step destroys
all data on the selected device.

Unmount its mounted partitions, substituting the exact partition:

```bash
sudo umount /dev/sdX1
```

## Write the image

Resolve the exact ISO filename first:

```bash
find result/iso -maxdepth 1 -name '*.iso'
```

Then write it to the verified whole device, not a partition:

```bash
sudo dd \
  if=result/iso/tentaflake.iso \
  of=/dev/sdX \
  bs=4M \
  status=progress \
  oflag=sync
```

Replace `/dev/sdX` only after checking it again with `lsblk`.

## Boot and install

Boot the target machine from the USB device. The installer auto-logs into its
local console and starts the `dialog`-based installer. It asks before
partitioning and runs `nixos-install` only after the selected disk is shown.

Disk formatting and installation are runtime mutations. A successful ISO build
does not authorize either operation.

New installations use Btrfs for the root partition and FAT32 for the UEFI
boot partition. The generated hardware configuration records both filesystem
types for subsequent boots. The template does not convert an existing ext4
host during an update; preserve a verified backup and plan a separate migration
or fresh installation. Agent workspace images also use Btrfs; see the
[workspace migration guide](https://docs.tentaflake.dev/configuration/workspace-quota/) before updating existing
quota-enabled deployments.

After installation and reboot, continue with
[the quick start](https://docs.tentaflake.dev/getting-started/quickstart/).

The installer environment itself uses the dev profile because it runs no
agents. The generated installed host uses the default `balanced` profile. Old
direct-secret/port agent definitions therefore require the explicit migration
described in [security profiles](https://docs.tentaflake.dev/security/profiles/); installation never
silently downgrades the target to dev.

---

# Secure installed-host quickstart

> Reach a secure first agent deployment on an installed host.

Canonical page: https://docs.tentaflake.dev/getting-started/quickstart/

> Source: [`docs/01-quickstart.md`](https://github.com/timfewi/tentaflake/blob/e21be475a3afc6e95c5087003edd0b7f780e4df4/docs/01-quickstart.md) at `e21be475a3af` · docs version **0.4.37**

Installed hosts default to the fail-closed `balanced` profile. This guide
verifies that baseline. An agent remains offline unless its exact container
name has an explicit broker policy. Model access uses the LLM broker; web
access uses the separate [research relay](https://docs.tentaflake.dev/configuration/research/), never general
internet access.

## 1. Install the external management policy

Replace the placeholder identity in `docs/tailscale-policy.example.json`,
validate it in the Tailscale admin console, and install it before relying on
Tailscale as the only management path. The policy must restrict both network
grants and Tailscale SSH to the operator and `tag:agent-host`.

The default tailnet allow-all policy is not acceptable. Do not enable Funnel
or publish an agent API with Serve. Protect the identity provider with
hardware-backed MFA; plan Tailnet Lock with at least two signing nodes.

## 2. Declare a stopped capsule

Copy the current example:

```bash
sudo cp \
  /etc/nixos/my-agents.nix.example \
  /etc/nixos/my-agents.nix
```

The example contains stopped Hermes and ZeroClaw capsules, a stopped OpenClaw
scaffold, and matching workers and Btrfs workspace quotas for Hermes/ZeroClaw.
Remove the builder, worker, and quota entries for any runtime you do not need.
OpenClaw creates no OCI container and refuses an explicit start; see
[adapter support limits](https://docs.tentaflake.dev/configuration/agent-adapters/).

`autoStart = false` stops automatic controller startup, but activation still
creates enabled workspace filesystems and worker services. On an existing host,
follow [workspace migration](https://docs.tentaflake.dev/configuration/workspace-quota/) before enabling quotas.
Do not add `envFile`,
`agenixFile`, `hostPort`, `servePort`, host networking, or literal secret
values under balanced; evaluation rejects them.

## 3. Evaluate and build without activation

From `/etc/nixos`, resolve the installed hostname and build its toplevel:

```bash
hostname
nix build \
  .#nixosConfigurations.<hostname>.config.\
system.build.toplevel \
  --no-link
```

Review any assertion and the source diff. A successful build proves the
candidate closure, not the live host.

## 4. Activate only in an approved window

Activation is an explicit host mutation. Once the exact host and build result
are reviewed, the operator may run:

```bash
sudo nixos-rebuild switch \
  --flake /etc/nixos#<hostname>
```

Tentaflake never performs this step merely because evaluation succeeded.

## 5. Verify the activated posture

```text
tentaflake status
tentaflake doctor --security
tentaflake doctor --security --json
sudo systemctl cat \
  docker-hermes-coding.service
```

For Podman, the unit prefix is `podman-`. A balanced capsule should show
`runsc`, non-root user, cap-drop-all, read-only root, no-new-privileges,
hardened tmpfs, and resource flags. Without a broker it also shows
`network=none`, and the security doctor reports `TFSEC-013`. With a broker it
shows one internal network, runtime virtual credentials, and no direct route.
It should not report `TFSEC-020`; that finding means the corresponding
disposable worker boundary is absent. `TFSEC-021` means its persistent
workspace has no managed hard size ceiling.

`autoStart = true` is intentionally rejected under `balanced` until the same
container key has an enabled LLM broker, worker, workspace quota, and research
relay. Keep the controller stopped while those policies are incomplete or while
migrating an existing workspace.

## 6. Keep the agent stopped

Do not place real OpenAI, Anthropic, OpenRouter, GitHub, Firecrawl, or
infrastructure credentials in the capsule. Without a scoped broker
declaration, a balanced agent cannot perform external model calls. Web calls
additionally require its research relay and the operator-configured Research
service and VPN. To grant narrow access, follow
[brokered egress](https://docs.tentaflake.dev/configuration/brokered-egress/) and [web research](https://docs.tentaflake.dev/configuration/research/),
build, review the generated units and firewall rules, then activate deliberately.
The [secure adapter example](https://github.com/timfewi/tentaflake/blob/e21be475a3afc6e95c5087003edd0b7f780e4df4/examples/adapter-secure.nix) combines the required
declarations; actual vendor tool discovery and execution still need runtime evidence.

For a trusted development-only compatibility host, the intentional escape is:

```nix
tentaflake.security.profile = "dev";
```

That permits legacy direct credentials and networking but is explicitly not a
security boundary for untrusted 24/7 agents. There is no silent fallback from
balanced or strict.

## Next reading

- [Security profiles and migration](https://docs.tentaflake.dev/security/profiles/)
- [Tailscale management policy](https://docs.tentaflake.dev/operations/tailscale/)
- [Operations and recovery](https://docs.tentaflake.dev/operations/recovery/)
- [Agenix boundaries](https://docs.tentaflake.dev/configuration/secrets/)
- [Brokered egress](https://docs.tentaflake.dev/configuration/brokered-egress/)
- [Disposable worker and approval](https://docs.tentaflake.dev/configuration/disposable-worker/)
- [Persistent workspace quota](https://docs.tentaflake.dev/configuration/workspace-quota/)
- [Threat model](https://docs.tentaflake.dev/security/threat-model/)
- [Agent adapters and support limits](https://docs.tentaflake.dev/configuration/agent-adapters/)

---

# Use with AI agents

> Read, explain, and operate tentaflake with Claude, ChatGPT, Gemini, and coding agents through Markdown pages, llms.txt, and installable skills.

Canonical page: https://docs.tentaflake.dev/getting-started/ai-agents/

This documentation is published for two kinds of readers: people, and the assistants and coding agents that work for them. Everything an agent needs is plain text at a predictable URL.

:::caution
An assistant can misread or invent details. Treat its answer as a guide to the docs, not a replacement: confirm commands on the linked page, and review the [threat model](https://docs.tentaflake.dev/security/threat-model/) yourself before you activate anything. Never paste provider keys, tailnet credentials, or other secrets into a chat.
:::

## Explain a page or a chapter

Every page has an **Explain this page** menu under its title, and every chapter heading has a smaller **Explain** menu. Both open a new chat in the assistant you pick, with a prompt that points at the Markdown version of that page and asks the assistant to stay within it.

| Assistant | What the link does |
| --- | --- |
| Claude, ChatGPT, Perplexity, Le Chat, Grok | Opens a new chat with the prompt already filled in |
| Gemini | Copies the prompt, then opens Gemini; paste it to start |

**Copy prompt** gives you the same text for any other assistant. **Copy page as Markdown** puts the whole page on your clipboard for tools that cannot browse.

## Machine-readable formats

| Resource | URL | Use it for |
| --- | --- | --- |
| Documentation index | [`/llms.txt`](https://docs.tentaflake.dev/llms.txt) | A short map of every page with a one-line description |
| Complete documentation | [`/llms-full.txt`](https://docs.tentaflake.dev/llms-full.txt) | One file with every page, for a single fetch |
| Markdown twin | Any page URL with `.md` instead of the trailing slash | The exact text of one page, for example [`/getting-started/quickstart.md`](https://docs.tentaflake.dev/getting-started/quickstart.md) |
| Skill index | [`/.well-known/agent-skills/index.json`](https://docs.tentaflake.dev/.well-known/agent-skills/index.json) | Discovering the skills this site publishes |

Each HTML page also announces its Markdown twin with a `<link rel="alternate" type="text/markdown">` element.

## Give your coding agent the tentaflake skill

A skill is a short instruction file an agent loads when a task matches it. The `tentaflake` skill teaches an agent where the documentation lives, how to read it, and which operations it must leave to you.

Install it with the open `skills` CLI, which supports Claude Code, Codex, Cursor, Gemini CLI, and many other agents:

```bash
npx skills add \
  https://docs.tentaflake.dev/.well-known/agent-skills/tentaflake/SKILL.md
```

Or save the file yourself. Claude Code reads project skills from `.claude/skills/`; Codex, Cursor, and Gemini CLI read `.agents/skills/`:

```bash
mkdir -p .claude/skills/tentaflake
curl -fsSL -o .claude/skills/tentaflake/SKILL.md \
  https://docs.tentaflake.dev/.well-known/agent-skills/tentaflake/SKILL.md
```

Read the file before you install it, as you would any instruction you hand to an agent. The skill index publishes a SHA-256 digest for it.

The tentaflake repository also bundles operator and Hermes skills in `.agents/skills/`. They are listed in the [skill index](https://docs.tentaflake.dev/reference/skills/).

## Starter prompts

Paste one of these into a coding agent that can fetch URLs.

Plan an installation:

```text
Read https://docs.tentaflake.dev/llms.txt, then the install guide and the
secure quickstart it links to. Ask me about my hardware and whether this is a
new machine or an existing NixOS host. Then write a step-by-step plan.
Do not run any command that formats a disk or activates a configuration.
```

Review a configuration:

```text
Read https://docs.tentaflake.dev/security/profiles.md and
https://docs.tentaflake.dev/security/threat-model.md. Review the Nix files in
this directory against the balanced profile. List each finding with the
documentation section it comes from, and mark anything you could not verify.
```

## What an agent should not do for you

The [quickstart](https://docs.tentaflake.dev/getting-started/quickstart/) separates building a configuration from activating it, and that separation applies to agents as well. Reading, planning, evaluation, and a non-activating `nix build` are reasonable to delegate. Writing an installer image to a disk, `nixos-rebuild switch`, and anything that touches credentials are decisions for the operator.

---

# Architecture and scope

> Architecture, scope, modules, and consumption model for tentaflake.

Canonical page: https://docs.tentaflake.dev/concepts/architecture/

> Source: [`README.md`](https://github.com/timfewi/tentaflake/blob/e21be475a3afc6e95c5087003edd0b7f780e4df4/README.md) at `e21be475a3af` · docs version **0.4.37**

:::caution
**Pre-1.0 project:** Tentaflake is under active development and has not yet
reached version 1.0. APIs, NixOS options, defaults, installation flows,
security boundaries, and documented workflows may still change
substantially between releases, including breaking changes. Pin a release
or commit for deployments, and review the [changelog](https://docs.tentaflake.dev/reference/changelog/) and
migration notes before updating.
:::

Tentaflake is a generic NixOS flake template for running isolated AI agents
on one machine. Hermes and ZeroClaw agents are declared as OCI
containers and supervised by systemd.
Secure controllers and policy brokers recover from crashes with capped
systemd restart backoff, including after prolonged transient failures.
Transient broker crashes preserve the controller process. Explicit broker stops
still stop dependent controllers; model requests may fail while a broker recovers.

The core is intentionally small. It contains the host modules, agent builders,
an installer ISO, and a Rust operator CLI. Observability and runtime detection
are separate opt-in profiles.

The [roadmap](https://github.com/timfewi/tentaflake/blob/e21be475a3afc6e95c5087003edd0b7f780e4df4/docs/roadmap.md) prioritizes a common isolated runtime and simple
agent onboarding on installed hosts, before employee-profile and company
features. This is planned work; the current support matrix below still applies.

Discover presets with `tentaflake runtimes --json`. A digest-pinned image and
command can use the [common isolated workload definition](https://docs.tentaflake.dev/configuration/agent-adapters/#generic-isolated-workload)
without a new adapter; generic startup remains gated pending capability admission.

## Current scope

| Component | Status |
|---|---|
| Installed NixOS host | Core |
| Adapter registry and Hermes/ZeroClaw builders | Core |
| Rust `tentaflake` CLI | Core |
| Rust LLM/fetch policy broker | Core, opt-in per agent |
| `secure-research-tool` web research | Pinned public service; required for balanced auto-start |
| Disposable no-egress tool worker | Core; required for balanced auto-start |
| Fixed-size persistent workspace and optional private state | Core; workspace required for balanced auto-start |
| Cosign image start gate | Core, opt-in per agent |
| Encrypted Restic backup including selected state/workspace quota mounts | Core, opt-in |
| Installer ISO | Core |
| Prometheus, Grafana, Loki, Alloy | Optional profile |
| Falco runtime detection | Optional profile |

There is no live-agent ISO, bundled audit daemon, SQLite event store, custom
web console, or Go workspace.

## Quick start

Run the contributor gate, which loads the pinned development tools itself:

```bash
just fast
```

Enter `nix develop` before invoking Cargo or lint tools directly. The fast gate
includes formatting, lint, Rust checks, policy and adapter evaluation, and the
installer-generated flake. For changes that need runtime evidence, run the
affected VM suite explicitly; the full flake check includes both VM suites:

```bash
nix flake check
```

Alternatively, open the checkout in a Dev Container-compatible editor. The
committed `.devcontainer` installs the repository's Nix version, then warms the
same lock-file-backed `nix develop` environment used above. It does not mount a
container runtime socket or inject credentials.

Build the installer ISO:

```bash
nix build .#installer-iso
```

New installations use Btrfs for the host root and fixed-size agent workspaces.
Existing ext4 hosts and workspace images require an explicit migration; see
[installation](https://docs.tentaflake.dev/getting-started/install/) and [workspace migration](https://docs.tentaflake.dev/configuration/workspace-quota/).
Installer cleanup preserves other disks' swap, LVM and encrypted mappings;
LVM groups spanning the selected and another disk require migration first.

The result is written below `result/iso/`. Writing it to a block device is
destructive; follow [the install guide](https://docs.tentaflake.dev/getting-started/install/) and resolve the
target device explicitly.

Contributor end-to-end entry points keep the pinned tool and VM setup behind
short recipes. They load the Nix development tools automatically, so `just lint`,
`just rust` and `just security` also work from an ordinary shell. `just list`
lists the available recipes:

```bash
just fast                # contributor checks without VM or ISO builds
just e2e                 # complete automated local gate
just e2e-devcontainer    # rebuild and smoke-test the locked Dev Container
just security            # Semgrep source scan + OSV dependency scan
just e2e-installer       # install into one isolated UEFI/QCOW2 VM
just e2e-run-vm          # boot that installed VM again
```

The installer VM never receives a host block device. Its persistent test disk
and UEFI variables live below `/var/tmp/tentaflake-e2e-<user>/`.
See [build boundaries](https://docs.tentaflake.dev/reference/builds/) for fast checks, GitHub check selection
and the Nix/Bazel assessment.

## Define agents

Copy the example and keep it generic in this repository:

```bash
cp my-agents.nix.example my-agents.nix
```

`my-agents.nix` may use `mkAgent` or the compatible runtime-specific builders:

```nix
{ mkHermesAgent, mkZeroClawAgent, ... }:
[
  (mkHermesAgent {
    name = "assistant";
    autoStart = false;
  })
  (mkZeroClawAgent {
    name = "assistant";
    autoStart = false;
  })
]
```

Alternatively, use the non-secret data shape in `agents.json.example`.
The versioned generic `agents` array accepts the same adapter arguments.
See [agent adapters](https://docs.tentaflake.dev/configuration/agent-adapters/) for the contract, complete secure
consumer example and evidence limits. OpenClaw v2026.9.7 is a stopped scaffold;
it creates no runtime container and is not operationally supported yet.
`agents.json` is declarative input; the removed interactive wizard no longer
edits it.

Installed systems default to `tentaflake.security.profile = "balanced"`.
Balanced agents are fail-closed capsules: no host network, published ports,
direct egress, real provider credentials, or mutable images. Without a
per-agent broker declaration they remain at `network=none`. With one, they
join exactly one internal network and can reach only their configured host
broker endpoints. Research agents use only the LLM broker on that network;
web access goes through the separate Unix-socket relay. `autoStart = false`
keeps the controller stopped at boot; activating the host can still create
declared workers and workspace filesystems.
Setting `autoStart = true` under `balanced` is accepted only after the exact
container also has an enabled LLM broker, disposable worker, fixed-size
workspace quota, and research relay. Web access uses the pinned public
[tentaflake-research](https://github.com/timfewi/tentaflake-research) service;
native web tools and the legacy fetch broker are disabled for research agents.
See [configuration and verification](https://docs.tentaflake.dev/configuration/research/). This prevents an
incomplete 24/7 declaration from silently starting with missing policy boundaries.

Research VPN readiness uses the pinned module's existing
`services.secureResearch.vpnObserver` options. The observer stays disabled by
default; operators can select WireGuard or tun link/path evidence, with optional
WireGuard handshake/public-peer checks and planned draining. Management enrollment
does not supply an Internet exit. See the [VPN selection and operator drain
procedure](https://docs.tentaflake.dev/configuration/research/#vpn-readiness-selection); configuration evaluations
do not prove live tunnel, exit-node or production readiness.

Existing configurations that require direct provider credentials or host
networking must deliberately select `dev`; this is a breaking change and is
not suitable for untrusted 24/7 agents. Never put secret values in Nix
expressions, JSON, Git, or the Nix store. See
[security profiles and migration](https://docs.tentaflake.dev/security/profiles/) and the
[threat model](https://docs.tentaflake.dev/security/threat-model/).

## Operator CLI

The CLI is built from `crates/tentaflake-cli` and installed by
`modules/shell.nix`.

```text
tentaflake help
tentaflake status [--json] [--hide]
tentaflake health [--json] [--hide]
tentaflake doctor [--json] [--hide]
tentaflake doctor --security [--json] [--hide]
tentaflake logs <agent>
tentaflake restart <agent>
tentaflake shell <agent>
tentaflake exec <agent> -- <command>
tentaflake stats
tentaflake ps
tentaflake backup <agent>
```

`tentaflake-status` is a status alias. The deprecated `hermes` executable is
retained as a compatibility shim. `tentaflake top`, `console`, and the agent
wizard were removed with the old audit stack.

`help`, `--help`, and `-h` work before the host configuration is installed.
Create stopped definitions with `tentaflake agent template <preset> <name>`.
On installed hosts, `agent validate|plan|import <file>` uses the shared Nix
parser and adds entries without activation. See
[onboarding](https://docs.tentaflake.dev/configuration/agents/#add-an-agent-on-an-installed-host).

`health` and `doctor` return non-zero for failed units, unknown agent states,
or root-disk usage of at least 90%; unavailable host queries are errors.
Stopped agents remain valid. `--hide` redacts host and agent names in both
text and JSON reports.

The `rebuild` and `update` subcommands are explicit runtime operations. Source
evaluation or a successful build does not authorize activation.

See [the CLI guide](https://docs.tentaflake.dev/operations/cli-shell/) and
[agent configuration guide](https://docs.tentaflake.dev/configuration/agents/).
Management-plane policy is covered by the
[private management and Tailscale policy guide](https://docs.tentaflake.dev/operations/tailscale/).

## Security profiles

| Profile | Meaning |
|---|---|
| `dev` | Compatibility path; broad authority may be configured |
| `balanced` | Default; gVisor capsule, non-root, read-only, no direct egress or real credentials |
| `strict` | Reserved; evaluation fails until a tested separate-kernel boundary exists |

The shared policy in `lib/containerSecurity.nix` is applied after caller
overrides and asserts the secure invariants. The secure path drops all
capabilities, uses `runsc`, applies CPU/RAM/swap/PID/ulimit/tmpfs limits, and
rejects ports, devices, caller networks, real credential files, sensitive
mounts, secret-like environment keys, and attempts to override OCI security
flags. The only secure network exception is the module-generated internal
broker network and its runtime-generated virtual credential file.
The administrative user is not placed in the root-equivalent Docker group.
Balanced also requires configured private management and SSH authorization through
`tentaflake.management`. Defaults retain the currently reviewed `tailscale`
transport, `ssh.policy = "tailnet-policy"`, and `tag:agent-host`; public OpenSSH
remains forbidden. `tentaflake.tailscale.enable` aliases `management.enable` for
existing consumers. Version-2 security manifests distinguish this configured
contract from unknown legacy evidence. Enrollment, effective tailnet grants and
operator access still require separate evidence; remote policy remains an
operator-owned control installed separately. See the [contract and
migration](https://docs.tentaflake.dev/operations/tailscale/#host-contract-and-migration).

Phase B adds per-agent LLM credential and SSRF-safe fetch brokers, model/host
allowlists, request and daily budgets, prompt-free JSONL audit, DNS pinning,
redirect revalidation, quarantine, and host/FORWARD firewall rules. Agents
without that explicit declaration stay at `network=none`. Phase C adds an
opt-in disposable worker with bounded FD-safe snapshots, gVisor, no network or
secrets, runtime/resource/tmpfs limits, host-side action approval, cleanup, and
a read-only result path. An opt-in fixed-size Btrfs volume places a hard ceiling
on each persistent controller workspace and, when configured, its private state.
Other host state and aggregate sparse-image allocation still need capacity
monitoring. The broker marks web material as untrusted;
it does not claim that prompt injection is solved. Digest pinning is mandatory
in secure profiles. Optional per-agent Cosign policies add a fail-closed
publisher-signature gate before controller start; they do not prove that signed
software is harmless.

## Architecture

```text
flake.nix
├── modules/             core NixOS modules, brokers, worker
├── modules/profiles/    observability, Falco
├── adapters/            runtime implementations and schema-v1 contracts
├── lib/                 common builder, compatible wrappers, helpers
├── crates/              Rust workspace
├── pkgs/                Nix package wrappers
├── installer/           installer ISO
└── tests/               evaluation and VM tests
```

The default module imports only:

- host options, boot, hardening, locale, networking, broker/worker and image
  provenance policy, agent-instance inventory, research integration, Nix settings;
- base packages, users, the private management contract, SSH, Tailscale,
  and operator shell.

Optional profiles must be imported explicitly from the flake output.

## Brokered egress

Configure brokers by exact OCI container name. Provider credentials remain
runtime-only host files and are loaded only into the LLM broker. The agent
receives a random per-boot virtual key; it never receives the provider key.

Chat Completions and Responses support bounded SSE when the exact broker's
`llm.streaming.enable = true`. It defaults to false and preserves conservative
budget accounting. See [streaming limits and failure behavior](https://docs.tentaflake.dev/configuration/brokered-egress/#streaming-model-responses).
Both routes preserve optional, bounded `x-opencode-session` conversation metadata
for JSON and SSE. The client supplies the value; see the
[header contract](https://docs.tentaflake.dev/configuration/brokered-egress/#declaration).

```nix
tentaflake.broker.agents.hermes-coding = {
  enable = true;
  subnet = "10.203.20.0/30";
  gateway = "10.203.20.1";

  llm = {
    enable = true;
    upstreamBaseUrl =
      "https://api.openai.com/v1/";
    providerCredentialFile =
      "/run/agenix/openai-key";
    allowedModels = [
      {
        name = "gpt-5-mini";
        inputMicrousdPerMillion = 250000;
        outputMicrousdPerMillion = 2000000;
      }
    ];
  };

  fetch.enable = false;
};
```

Every enabled agent needs a unique `/30`. This is the LLM boundary only; declare
the [research relay](https://docs.tentaflake.dev/configuration/research/) separately. Legacy fetch cannot be
enabled for a research agent. See
[brokered egress](https://docs.tentaflake.dev/configuration/brokered-egress/) for the full trust boundary,
failure behavior, budgets, and verification steps.

## Disposable tool worker

Enable the worker by exact OCI container name and match its workspace and
numeric user to the corresponding builder:

```nix
tentaflake.worker.agents.hermes-coding = {
  enable = true;
  workspace =
    "/var/lib/hermes-coding/workspace";
  containerUid = 10000;
  containerGid = 10000;
};
```

Jobs enter through `.tentaflake-worker/inbox/<id>.json`. Only
`local-reversible` runs automatically. External, financial, productive,
communicative, irreversible, or privileged classes remain pending until a
host operator approves the exact privately captured job; `forbidden` never
runs. Approval never grants network, secrets, host mounts, capabilities, or a
runtime socket. Results appear read-only below
`/run/tentaflake-worker/results/<id>/` in the controller. The queue shares one
operator/worker lock, bounds aggregate entries and bytes, and retires interrupted
claims without automatic dispatch replay. See
[the disposable-worker guide](https://docs.tentaflake.dev/configuration/disposable-worker/).
Tentaflake creates the matching host group automatically; deployments that
already own a custom container GID can select its existing group with
`hostGroup`.

The secure example also declares `tentaflake.workspaceQuota.agents` for each
controller. First activation creates and formats a sparse fixed-size Btrfs image
for an empty workspace. Existing data is never hidden or migrated implicitly.
Read [the workspace quota guide](https://docs.tentaflake.dev/configuration/workspace-quota/) before enabling it
on an existing host.
The same declaration accepts `state = { path = "/var/lib/hermes-coding"; sizeMiB = 1024; };`
to bound private state separately. Its path/owner must match adapter metadata;
existing state requires an explicit offline migration.

## Optional observability

Import `nixosModules.observability`, enable the profile, and provide runtime
credential files for Grafana:

```nix
imports = [
  inputs.tentaflake.nixosModules.observability
];

tentaflake.profiles.observability = {
  enable = true;
  grafanaSecretKeyFile =
    "/run/agenix/grafana-secret-key";
  grafanaAdminPasswordFile =
    "/run/agenix/grafana-admin-password";
};
```

Prometheus, Grafana, Loki, Alloy, and the node exporter bind to loopback. The
profile does not open firewall ports or publish a dashboard.

## Optional runtime detection

Falco is separate because it needs host eBPF visibility and powerful kernel
capabilities. The pinned nixpkgs revision does not package Falco, so consumers
must supply a reviewed, pinned package:

```nix
imports = [
  inputs.tentaflake.nixosModules.falco
];

tentaflake.profiles.falco = {
  enable = true;
  package = myPinnedFalcoPackage;
};
```

The service uses Falco's modern eBPF engine. See
[observability and detection](https://docs.tentaflake.dev/operations/observability/) for prerequisites and
trust boundaries.

## Optional profiles

These modules are exported but not imported by the core:

| Output | Source |
|---|---|
| `nixosModules.observability` | `modules/profiles/observability.nix` |
| `nixosModules.falco` | `modules/profiles/falco.nix` |

Editor, Hive Research, and Piper TTS modules and voice assets have been removed.
Remove their imports, options, and package references when updating a consumer;
deployment-specific integrations belong in the consumer flake.

See the [archive comparison](https://github.com/timfewi/tentaflake/blob/e21be475a3afc6e95c5087003edd0b7f780e4df4/docs/archive-comparison.md) for selected backports,
preserved current fixes, and the next useful archive changes.

## Build and test

```bash
just fast
```

See [contributor checks](https://github.com/timfewi/tentaflake/blob/e21be475a3afc6e95c5087003edd0b7f780e4df4/CONTRIBUTING.md) for focused recipes. `just e2e`
includes the full flake check, both VM suites, and the installer ISO. Run these
larger gates deliberately. Keep evaluation, build, activation, and live-runtime
proof separate.

## Documentation

The public repository is the source of truth. The [website](https://tentaflake.dev/)
and [documentation site](https://docs.tentaflake.dev/) present a pinned release;
their source lines identify the exact commit. Unreleased checkout changes can
therefore be newer than the website.

| Topic | Guide |
|---|---|
| Product direction and planned milestones | [Roadmap](https://github.com/timfewi/tentaflake/blob/e21be475a3afc6e95c5087003edd0b7f780e4df4/docs/roadmap.md) (planned work, not current support) |
| Installation and first host | [Install](https://docs.tentaflake.dev/getting-started/install/), [quickstart](https://docs.tentaflake.dev/getting-started/quickstart/) |
| Agent definitions and runtimes | [Declarative configuration](https://docs.tentaflake.dev/configuration/agents/), [adapters](https://docs.tentaflake.dev/configuration/agent-adapters/) |
| Day-to-day management | [Agent management](https://docs.tentaflake.dev/operations/agent-management/), [operator CLI](https://docs.tentaflake.dev/operations/cli-shell/), [recovery](https://docs.tentaflake.dev/operations/recovery/) |
| Skills and secrets | [Skill index](https://docs.tentaflake.dev/reference/skills/), [Agenix](https://docs.tentaflake.dev/configuration/secrets/) |
| Forks and builds | [Fork checklist](https://docs.tentaflake.dev/reference/fork-checklist/), [build boundaries](https://docs.tentaflake.dev/reference/builds/) |
| Security and management access | [Profiles](https://docs.tentaflake.dev/security/profiles/), [Tailscale](https://docs.tentaflake.dev/operations/tailscale/), [threat model](https://docs.tentaflake.dev/security/threat-model/) |
| Model, research and execution | [Broker](https://docs.tentaflake.dev/configuration/brokered-egress/), [research](https://docs.tentaflake.dev/configuration/research/), [worker](https://docs.tentaflake.dev/configuration/disposable-worker/) |
| Storage and monitoring | [Workspace quota](https://docs.tentaflake.dev/configuration/workspace-quota/), [observability](https://docs.tentaflake.dev/operations/observability/) |
| Documentation maintenance | [Source ownership and release synchronization](https://docs.tentaflake.dev/reference/documentation/) |

[Archive comparison](https://github.com/timfewi/tentaflake/blob/e21be475a3afc6e95c5087003edd0b7f780e4df4/docs/archive-comparison.md) records a historical review;
use the current guides for implementation and support status.

## Consume as a flake input

```nix
{
  inputs.tentaflake.url =
    "github:timfewi/tentaflake";

  outputs = { nixpkgs, tentaflake, ... }: {
    nixosConfigurations.host =
      nixpkgs.lib.nixosSystem {
        system = "x86_64-linux";
        modules = [
          tentaflake.nixosModules.default
          ./configuration.nix
        ];
      };
  };
}
```

The helper functions are exported under `lib.x86_64-linux`.

## Security boundaries

- The repository is a generic template. Deployment identities, real hosts,
  private agent context, and secrets belong in forks.
- Docker group membership is root-equivalent. Podman avoids that group but
  has different root/rootless store semantics.
- Optional dashboards stay loopback-only. Publishing them is a deployment
  decision and should add authentication.
- Falco detects suspicious runtime behavior; it is not container isolation.
- No source change, check, or build implicitly activates NixOS or mutates a VM.

Read [SECURITY.md](https://docs.tentaflake.dev/security/reporting/) and the
[operations guide](https://docs.tentaflake.dev/operations/recovery/) before deployment.

## Contributing

Use Conventional Commits and add the DCO sign-off with `git commit -s`. Keep
behavior changes, verification, and documentation synchronized. See
[CONTRIBUTING.md](https://github.com/timfewi/tentaflake/blob/e21be475a3afc6e95c5087003edd0b7f780e4df4/CONTRIBUTING.md).

Tentaflake is MIT-licensed. The name and logo are covered separately by
[TRADEMARK.md](https://github.com/timfewi/tentaflake/blob/e21be475a3afc6e95c5087003edd0b7f780e4df4/TRADEMARK.md).
The [artwork overview](https://github.com/timfewi/tentaflake/blob/e21be475a3afc6e95c5087003edd0b7f780e4df4/public/README.md) links the logo, illustrations, and
wallpaper downloads.

---

# Security boundaries

> A map of the trust boundaries that matter when running autonomous agents with tentaflake.

Canonical page: https://docs.tentaflake.dev/concepts/security-boundaries/

tentaflake separates configuration, credentials, network access, tool execution, and persistent state into explicit boundaries. The controls are composable, but no single control makes an untrusted agent safe.

## Boundary map

| Boundary | Mechanism | Review next |
| --- | --- | --- |
| Agent process → host | Hardened container and systemd policy | [Threat model](https://docs.tentaflake.dev/security/threat-model/) |
| Agent → model provider | Per-agent LLM broker, model allowlist, and virtual credential | [Brokered egress](https://docs.tentaflake.dev/configuration/brokered-egress/) |
| Agent → web | Pinned `secure-research-tool` relay; retrieved content stays untrusted | [Web research](https://docs.tentaflake.dev/configuration/research/) |
| Agent → tool execution | Disposable offline worker and approval gate; runtime tools must submit to its queue | [Disposable worker](https://docs.tentaflake.dev/configuration/disposable-worker/) |
| Agent → credentials | Agenix and runtime credential loading | [Secrets](https://docs.tentaflake.dev/configuration/secrets/) |
| Agent → persistent state | Fixed-size workspace and optional separate private-state quota | [Workspace and state quotas](https://docs.tentaflake.dev/configuration/workspace-quota/) |
| Operator → host | Configured private connectivity and SSH authorization; reviewed Tailscale backend | [Private management contract](https://docs.tentaflake.dev/operations/tailscale/) |

Treat the [threat model](https://docs.tentaflake.dev/security/threat-model/) and the configuration in your own fork as the authoritative security review inputs.

`tentaflake.management` declares operator-owned private connectivity and SSH
policy. Tailscale is the only reviewed backend; the old
`tentaflake.tailscale.enable` aliases `management.enable`. Balanced requires
read-only configured capabilities and still forbids public OpenSSH. The
version-2 security manifest describes configuration, not enrollment, live
connectivity, operator-tested access or effective remote grants. Legacy or
missing management records remain unknown. Management connectivity and SSH
authorization are separate from Research Internet exit readiness. See the
[contract and migration](https://docs.tentaflake.dev/operations/tailscale/#host-contract-and-migration).

Research Internet egress requires fresh trusted observations and controller
leases; missing or stale evidence withholds readiness. The optional
`services.secureResearch.vpnObserver` is disabled by default. Operators can
select link/path evidence and WireGuard handshake/public-peer checks. Evidence,
leases and markers must remain root-protected and unwritable by agents; VPN
credentials stay on the host. Tailscale or Headscale management membership does
not provide an Internet exit, and firewall markers and region remain operator
assertions.

For a planned exit change, keep the old exit and firewall protected until the
documented drain condition, then confirm a fresh ready generation. Read the
[VPN selection and drain procedure](https://docs.tentaflake.dev/configuration/research/#vpn-readiness-selection).
Option checks and upstream WireGuard fixtures are separate from Tentaflake
real-tunnel integration and operator acceptance, which remain outstanding.

Private-state quotas bound the adapter's exact state source separately from its workspace. Existing data requires explicit offline migration and quota-aware backup/restore; activation refuses unsafe sources or a mismatched backing image. Individual quotas do not reserve aggregate host capacity. See the [quota guide](https://docs.tentaflake.dev/configuration/workspace-quota/).

[Generic OCI definitions](https://docs.tentaflake.dev/configuration/agent-adapters/#generic-isolated-workload) use the same containment with no network, model credentials, or accepted model/Research hooks. Their private root-owned instance parent is not mounted into the capsule; startup refuses symlink mount sources. Generic workloads remain stopped pending capability admission. Native shell/files execute inside the capsule; declaring a worker does not redirect them automatically. [Onboarding imports](https://docs.tentaflake.dev/configuration/agents/#add-an-agent-on-an-installed-host) validate stopped definitions without activating the host.

Declaring a worker does not prove that every native controller tool uses it.
Actual vendor startup, MCP discovery and execution routing require evidence
from the exact image. The `dev` profile permits broader authority and is not a
security boundary for untrusted, continuously running agents.

The model broker can forward bounded Chat Completions and Responses SSE only
when enabled for the exact agent; streaming is disabled by default. Its event
and output limits, deadlines, cancellation, audit admission and conservative
budget reservation do not establish acceptance of a vendor runtime. See
[streaming model responses](https://docs.tentaflake.dev/configuration/brokered-egress/#streaming-model-responses)
and the [adapter support limits](https://docs.tentaflake.dev/configuration/agent-adapters/).

---

# Declarative agent configuration

> Define agents and validate or import stopped additions on installed hosts.

Canonical page: https://docs.tentaflake.dev/configuration/agents/

> Source: [`docs/08-agent-cli.md`](https://github.com/timfewi/tentaflake/blob/e21be475a3afc6e95c5087003edd0b7f780e4df4/docs/08-agent-cli.md) at `e21be475a3af` · docs version **0.4.37**

The interactive `tentaflake agent` wizard was removed. Agent definitions are
reviewable inputs: use `my-agents.nix` for the full builder API or
`agents.json` for versioned adapter arguments or the legacy data schema.

## Add an agent on an installed host

Discover presets with `tentaflake runtimes --json`, then create a stopped
definition, review it, and use the same Nix parser/builders as the installer:

```bash
tentaflake agent template hermes assistant > incoming.json
tentaflake agent validate incoming.json
tentaflake agent plan incoming.json --json
tentaflake agent import incoming.json
```

For an operator-reviewed OCI image, use the `generic` preset. Supply a real
digest reference and argument vector; this example deliberately needs your image:

```bash
tentaflake agent template generic coding \
  --image "$REVIEWED_AGENT_IMAGE" -- agent-command --workspace /workspace \
  > incoming.json
tentaflake agent plan incoming.json
tentaflake agent import incoming.json
```

Templates work without host configuration. Validation, plans and imports require
the generated CLI configuration/inventory and installed pinned Nix inputs.
Evaluation runs offline with a 60-second deadline and bounded output; it neither
downloads OCI images nor installs software. Definitions must be non-secret.
Incoming files are limited to 1 MiB and 128 additions. Import accepts only a
schema-v1 `agents` envelope and ordinary stopped workload fields; alternate host
mounts, operational services and credential paths remain administrator-owned.
Generic onboarding requires a balanced host profile.

Import adds entries to `agents.json`, preserves legacy arrays and operator notes,
and refuses duplicate identities in JSON, active inventory or the flake's current
declared OCI/adapter inventory. Inputs and destination must be regular files.
Validation failure leaves existing bytes intact. CLI imports share a file lock;
atomic publication checks that `agents.json` did not change during validation.
This detects observed manual edits; it cannot lock administrators out of changes
after the final check. A leftover `agents.json.pending` is refused rather than
overwritten: review it before removing it. The running host remains unchanged.

`plan`, `validate` and `import` accept `--json` and `--hide`. Hidden plans redact
names, paths, image references and failure details. Nix source diagnostics are
not printed because source excerpts may contain private operator data. Plans
report source validation and pending vendor evidence, not live readiness or
vendor acceptance. OpenClaw remains a non-runnable scaffold. Unsupported generic
model/Research capabilities are refused; current secure startup gates still apply.

Review and track `agents.json` before an explicit host update. Git-backed flakes
only evaluate tracked files, including source-only `my-agents.nix` instances:

```bash
git add agents.json
tentaflake rebuild
```

The import does not git-add, rebuild, activate or start services. A host update
still validates the complete configuration and its dependencies; a failed update
does not replace the last active system revision. Review any concurrent Nix
configuration edits before activation.

## Nix definitions

Start from `my-agents.nix.example`:

```bash
cp my-agents.nix.example my-agents.nix
```

Each builder returns a NixOS module. The repository auto-imports the resulting
list when `my-agents.nix` exists.

```nix
{ mkHermesAgent, mkZeroClawAgent, ... }:
[
  (mkHermesAgent {
    name = "assistant";
    autoStart = false;
  })
  (mkZeroClawAgent {
    name = "research";
    autoStart = false;
    settings.schema_version = 3;
  })
]
```

This evaluates under the default `balanced` profile. The capsules have no
direct network and receive no real credentials. Keep them stopped until a
reviewed broker path exists.

Use `mkAgent { adapter = "hermes"; ... }` or the compatible runtime-specific
builders according to the runtime contract. See [agent adapters](https://docs.tentaflake.dev/configuration/agent-adapters/)
for the lazy registry and OpenClaw's stopped scaffold and acceptance limits.
See the adapter implementations and example file for runtime-specific options.

## JSON definitions

Copy `agents.json.example`, keep only the agents you need, and validate it:

```bash
cp agents.json.example agents.json
jq -e . agents.json
```

`agentsFromData` accepts `schemaVersion: 1` and a generic `agents` array of
builder arguments including `adapter`, `name`, and `autoStart`. Stopped generic
entries can use balanced policy. The legacy `hermes` and `zeroclaw` arrays retain
their fields and defaults. Both forms are additive; unknown fields/adapters and
effective container collisions fail. The informational `_securityNote` is
accepted for compatibility. Input may name a runtime credential path, but must
never contain a credential value.

Legacy entries with direct environment files or ZeroClaw ports require the
host to deliberately select:

```nix
tentaflake.security.profile = "dev";
```

This is a migration bridge, not a secure 24/7 configuration. Prefer the Nix
builder API or generic `agents` entries for balanced capsules.

## Apply deliberately

First evaluate or build the selected host:

```bash
nix build \
  .#nixosConfigurations.tentaflake.config.\
system.build.toplevel \
  --no-link
```

Activation is a separate runtime operation. Review the build result and target
before using `nixos-rebuild switch` or `tentaflake rebuild`.

## Verify

After an explicitly authorized activation:

```bash
tentaflake status
tentaflake doctor
tentaflake doctor --security
systemctl status \
  docker-hermes-assistant.service
```

For Podman, unit names start with `podman-`.

## Remove an agent

Remove its declarative entry and matching broker, Research, worker, quota and
provenance declarations, then review backup paths before activation. See
[agent removal](https://docs.tentaflake.dev/operations/agent-management/#add-or-remove-an-agent). State directories,
backing images and secret files are not deleted automatically; decide separately
whether they must be archived, retained or removed.

---

# Agenix secrets and host-held credentials

> Encrypt and deliver agent credentials with Agenix.

Canonical page: https://docs.tentaflake.dev/configuration/secrets/

> Source: [`docs/04-agenix-secrets.md`](https://github.com/timfewi/tentaflake/blob/e21be475a3afc6e95c5087003edd0b7f780e4df4/docs/04-agenix-secrets.md) at `e21be475a3af` · docs version **0.4.37**

Agenix stores encrypted files in Git and decrypts them at activation/boot into
runtime files. It does not hide a credential from an agent that receives it.
Balanced/strict reject direct `envFile` and `agenixFile` inputs; secure model
credentials belong only in the host LLM broker.

## Enable the pinned input

In the consumer fork, enable the optional `agenix` input in `flake.nix`, review
and commit its lock revision, and import `inputs.agenix.nixosModules.age` in
the selected host's module list. Use the CLI from that reviewed revision, not
an unrelated moving package.

Two separate Nix files are needed:

| File | Reader | Purpose |
|---|---|---|
| `secrets/secrets.nix` | Agenix CLI, run from `secrets/` | Encryption recipients for each ciphertext |
| Root `secrets.nix` | NixOS module system | Runtime path, owner and permissions |

A NixOS `age.secrets` declaration does not supply the CLI's recipient rules.

## Declare encryption recipients

Create `secrets/secrets.nix` with authenticated public recipients:

```nix
{
  "llm-provider.age".publicKeys = [
    "<host-public-recipient>"
    "<editor-or-recovery-public-recipient>"
  ];
}
```

Replace both placeholders with actual age or supported SSH public keys. The
host must hold a matching private identity; retain a protected recovery identity
outside that host. Confirm a host key through an authenticated management path
and independently verify its fingerprint. `ssh-keyscan` alone does not
authenticate it. Never put private keys into these rules or the repository.

Using the pinned CLI, run from the rules directory:

```sh
cd secrets
agenix -e llm-provider.age
```

Enter the provider key in the editor as a single value, not as an
`OPENAI_API_KEY=...` environment assignment. Do not pass secret values through
command arguments, shell history or printed output. Commit only ciphertext
and non-secret declarations after review.

## Declare the runtime secret

Copy [secrets.nix.example](https://github.com/timfewi/tentaflake/blob/e21be475a3afc6e95c5087003edd0b7f780e4df4/secrets.nix.example) to root `secrets.nix` and
import it from `configuration.nix`. Its host-only declaration is:

```nix
{
  age.identityPaths = [ "/var/lib/agenix/identity" ];
  age.secrets.llm-provider = {
    file = ./secrets/llm-provider.age;
    owner = "root";
    group = "root";
    mode = "0400";
  };
}
```

Provision the matching private identity at that runtime path through the
operator's protected credential channel. Use a string, not a Nix path literal:
the private file must not become a store input. Confirm owner-only permissions
and persistence across reboot. Balanced disables OpenSSH, so do not assume an
SSH host key exists automatically.

Encrypted `.age` files can enter the store. Plaintext must never enter Nix
settings, JSON, `extraEnvironment`, seed directories or derivations; do not
use `builtins.readFile` to load a runtime secret.

## Connect only the host broker

In a module with `config` available:

```nix
tentaflake.broker.agents.hermes-coding = {
  enable = true;
  subnet = "10.203.20.0/30";
  gateway = "10.203.20.1";
  llm = {
    enable = true;
    upstreamBaseUrl = "https://api.openai.com/v1/";
    providerCredentialFile = config.age.secrets.llm-provider.path;
    allowedModels = [
      {
        name = "gpt-5-mini";
        inputMicrousdPerMillion = 250000;
        outputMicrousdPerMillion = 2000000;
      }
    ];
  };
};
```

Review the exact model and current prices for your deployment; these are example
policy values. systemd loads the provider value into the broker's private
credential directory. The agent gets a scoped virtual key. This fragment does
not configure the worker, quota or Research required for balanced automatic
startup; see [brokered egress](https://docs.tentaflake.dev/configuration/brokered-egress/) and
[the secure adapter example](https://github.com/timfewi/tentaflake/blob/e21be475a3afc6e95c5087003edd0b7f780e4df4/examples/adapter-secure.nix).

Build and review the selected host before an explicitly authorized activation.
A successful source check does not provision identities or decrypt runtime
secrets on the host.

## Verify without revealing values

On the activated host, check the exact path and intended reader:

```sh
sudo stat -Lc '%U %G %a %n' /run/agenix/llm-provider
sudo systemctl status tentaflake-broker-llm-hermes-coding.service --no-pager
tentaflake doctor --security
```

Expect root ownership and mode `400` for this declaration. Check broker
`/healthz` readiness and bounded logs. Local readiness does not prove a provider
call succeeds. Never dump `/run/agenix/*`, decrypted files or OCI environment
data for verification.

## Dev-only direct environment files

A trusted host may explicitly choose `tentaflake.security.profile = "dev"`
and use `agenixFile` on Hermes/ZeroClaw. Declare a separate encrypted environment
file containing the pinned runtime's supported variables; use the same
recipient/module separation and restrict its host reader.

The OCI runtime consumes this file at container creation; it is not a bind
mount and will not appear in `docker inspect .Mounts`. Verify its runtime path,
permissions and the declared environment-file input without exposing values.
Interactive provider setup and direct credentials are not a balanced workflow.

## Rotation and recovery

For a provider-value rotation, edit the ciphertext through the pinned CLI,
review/build, and activate the new secret within the authorized scope. Restart
only the exact LLM broker to refresh systemd's loaded credential. Its virtual key
survives broker restarts. Dev-only environment changes require recreating the
selected controller; use the coordinated Tentaflake lifecycle operation.

When changing recipients, edit the CLI rules and rekey from that directory:

```sh
cd secrets
agenix --rekey
```

Rekeying does not revoke credentials or rewrite Git history. Old ciphertext
remains decryptable with an old recipient key; if that key was compromised,
rotate every exposed secret value separately. Authenticate replacement public
keys before adding them. With no surviving recipient identity, encrypted values
cannot be recovered.

## Troubleshooting

| Symptom | Check |
|---|---|
| CLI finds no recipient rule | Run in `secrets/`; its `secrets.nix` must map the exact ciphertext filename |
| Decryption fails | Matching runtime identity, authenticated recipients and persistent identity path |
| Runtime secret is absent | Agenix module import, selected host and activation result |
| Broker refuses a credential | Exact runtime path, ownership/mode, single-value format and refreshed unit credentials |
| Secret enters the store | Remove Nix value interpolation/path literals; use runtime file paths |

For backup, Git and Grafana, use each host service's documented credential
format and permissions. See [operations](https://docs.tentaflake.dev/operations/recovery/),
[observability](https://docs.tentaflake.dev/operations/observability/) and the reviewed
[Agenix source](https://github.com/ryantm/agenix).

---

# Brokered egress

> Route provider traffic through an explicit broker boundary.

Canonical page: https://docs.tentaflake.dev/configuration/brokered-egress/

> Source: [`docs/12-brokered-egress.md`](https://github.com/timfewi/tentaflake/blob/e21be475a3afc6e95c5087003edd0b7f780e4df4/docs/12-brokered-egress.md) at `e21be475a3af` · docs version **0.4.37**

## Result

Balanced agents can use external models without receiving a provider credential
or general internet access. Web access uses [secure-research-tool](https://docs.tentaflake.dev/configuration/research/);
research-enabled agents must disable the legacy fetch broker described below. The
default remains fail-closed: an agent without a broker declaration uses
`network=none`.

Each enabled agent gets:

- one dedicated OCI `--internal` IPv4 `/30` network with a deterministic,
  policy-matched host bridge interface;
- no container DNS and no IPv6;
- host INPUT access only when both the exact bridge interface and agent source
  subnet match its own broker ports;
- a host FORWARD drop for the whole agent subnet;
- one random virtual key generated under `/run` on every boot;
- optional LLM and fetch broker processes with separate state and budgets.

The OCI container key must exactly match the generated container name, for
example `hermes-coding` or `zeroclaw-research`.

## Declaration

```nix
tentaflake.broker.agents.hermes-coding = {
  enable = true;
  networkName = "tf-hermes-coding";
  subnet = "10.203.20.0/30";
  gateway = "10.203.20.1";

  maxConcurrency = 4;
  maxRequestsPerMinute = 30;
  dailyRequestBudget = 1000;
  dailyTokenBudget = 1000000;
  dailyCostMicrousd = 10000000;

  llm = {
    enable = true;
    port = 7810;
    upstreamBaseUrl =
      "https://api.openai.com/v1/";
    providerCredentialFile =
      "/run/agenix/openai-key";
    maxCompletionTokens = 4096;
    allowedModels = [
      {
        name = "gpt-5-mini";
        inputMicrousdPerMillion = 250000;
        outputMicrousdPerMillion = 2000000;
      }
    ];
  };

  fetch = {
    enable = false;
    port = 7811;
    allowedHosts = [
      "platform.openai.com"
    ];
    allowedContentTypes = [
      "application/json"
      "text/html"
      "text/plain"
    ];
    maxRedirects = 3;
  };
};
```

Declare the research relay separately as described in [web research](https://docs.tentaflake.dev/configuration/research/).
The fetch options remain for stopped compatibility scaffolds; they cannot be
enabled alongside a research relay. Model allowlists must select inference-only
models. Local function/custom tool definitions are forwarded; provider-hosted
tools such as web search, file search and remote MCP, plus `web_search_options`,
`plugins`, `extensions` and `data_sources`, are rejected before budget reservation.

Both `/v1/chat/completions` and `/v1/responses` preserve the optional
`x-opencode-session` header for JSON and enabled SSE requests. Its value must be
1–128 ASCII characters from `A–Z`, `a–z`, `0–9`, `_`, `.`, `:`, and `-`;
malformed values return HTTP 400 before budget reservation or upstream dispatch.
Only this client header is forwarded alongside broker-owned transport headers;
arbitrary client headers are dropped and the virtual key is replaced by the
host-held provider credential. Session values are not written to audit logs.

The client owns this opaque conversation-affinity value. The broker neither
generates it nor verifies conversation ownership; it grants no authorization
and must not contain secrets or personal data. Older Hermes images or clients
configured against the local broker may need a client-side integration to emit
it, including on auxiliary calls. Broker forwarding alone does not establish
Hermes session propagation or provider acceptance.

Use a unique `/30` per agent. The network address must end on a four-address
boundary and `gateway` must be its first usable address. Evaluation rejects
duplicate networks/subnets, unrelated gateways, missing modes, plain HTTP
providers, provider files outside `/run`, duplicate models, wildcard hosts,
wildcard media types, and deterministic bridge-interface collisions.

## Host resource admission

Each LLM and fetch process has a systemd memory ceiling of 128 MiB, 64 tasks,
and 4096 open files. Override `tentaflake.broker.serviceMemoryMaxBytes`,
`serviceTasksMax`, or `serviceNoFileLimit` when a reviewed workload needs more.
The existing capped restart backoff still applies after failure or exhaustion.

Optional host admission ceilings live below `tentaflake.broker` and default to
`null` (no aggregate ceiling):

| Option | Counted declarations |
|---|---|
| `maxEnabledAgents` | Enabled broker agents |
| `maxTotalConcurrency` | `maxConcurrency` per enabled LLM/fetch service |
| `maxTotalRequestsPerMinute` | Request-rate budget per enabled LLM/fetch service |
| `maxTotalDailyTokenBudget` | Declared daily token budget per enabled agent |
| `maxTotalDailyCostMicrousd` | Declared daily cost budget per enabled agent |

An agent enabling both modes counts twice for concurrency and request rate.
Disabled agents do not count. Token/cost sums conservatively include every
enabled agent, even fetch-only declarations. Exceeding a ceiling rejects the
configuration during evaluation. These are admission checks on declared
budgets, not a shared runtime counter or a guarantee of total host memory use.

## Credential boundary

The real provider credential is supplied to the LLM unit through systemd
`LoadCredential`. It is read again for each request, so rotating the source
credential and restarting only that broker is sufficient. It is never added
to the agent environment, container mounts, Nix store, or audit log.

systemd exposes each loaded credential read-only and grants the dynamic service
user access with a named ACL. The broker accepts the resulting group-class ACL
mask bit only for an exact file directly below `$CREDENTIALS_DIRECTORY`;
ordinary credential files remain restricted to owner-only permissions, and
other-access bits are always rejected.

The credential setup unit creates:

```text
/run/tentaflake-broker/<container>/agent-token
/run/tentaflake-broker/<container>/agent.env
```

The credential unit owns only its agent-specific `RuntimeDirectory`, mode
`0700`, under `ProtectSystem=strict`; token and environment files use `0400`.
The shared parent is explicitly read-only and only the exact agent directory
is reopened writable. Credential and broker units explicitly clear their
capability bounding sets; an empty Nix list would omit that systemd directive.
The runtime directory survives unit stop/start and restart, so existing
brokers and controllers retain the same virtual key until the next boot.

The environment file contains the virtual key, `OPENAI_BASE_URL`, and the
Tentaflake broker URLs. It is the only environment file accepted by the
balanced container policy. The virtual key is scoped to one agent network and
is rejected by every other broker instance.

The broker writes one startup-readiness event after it has proved that its
private audit file can be opened and synced. `/healthz` rechecks credential and
audit readiness without appending an event, so health polling cannot rotate
useful audit history merely by probing. Audit files also reject symlink
targets. LLM input-token reservation uses request bytes as a conservative
upper bound rather than an optimistic characters-per-token estimate;
provider-reported usage remains separately recorded.

Host-side `tentaflake doctor` and VM probes reach `/healthz` through the exact
agent gateway address. VM readiness probes stop after 30 seconds and print the
raw health response, route, listener, firewall chains, broker unit status, and
journal instead of waiting for the test driver's global timeout.

## LLM policy

The LLM broker exposes only:

```text
POST /v1/chat/completions
POST /v1/responses
```

It accepts strict JSON, requires an exact model, clamps
the completion-token ceiling, reserves conservative token/cost budget before
the upstream call, disables environment proxies and redirects, resolves the
configured provider on the host, rejects non-public addresses, pins the
validated DNS answer into the TLS client, and rebuilds the Authorization
header from the host credential.

Audit events contain agent, route, model, outcome, status, reserved token/cost
figures, and timestamps. They never contain request bodies, prompts, virtual
keys, provider keys, or provider response bodies.

## Streaming model responses

Enable SSE explicitly for each exact agent broker; existing declarations keep
streaming disabled:

```nix
tentaflake.broker.agents.hermes-coding.llm.streaming = {
  enable = true;
  maxEventBytes = 64 * 1024;
  firstEventTimeoutSeconds = 30;
  idleTimeoutSeconds = 30;
  totalTimeoutSeconds = 120;
};
```

The same authenticated Chat Completions and Responses routes accept
`"stream": true`. `stream` must be a boolean. Streaming supports one completion
choice (`n` absent or `1`) and foreground requests; `background: true` is
rejected. Chat requests can use `max_tokens` or `max_completion_tokens`, but
cannot specify both. Responses use `max_output_tokens`. All retain the exact
model and local-function-tool allowlists, host-only credential substitution,
DNS address checks, runtime budget persistence and rate admission.

The host forwards complete events as HTTP/1.1 chunked `text/event-stream`
responses. It buffers at most one event, normalizes SSE line endings, handles
fragmented UTF-8 and multiline data, and retains text, reasoning and function
argument deltas. `maxEventBytes` must fit the existing `maxResponseBytes` total
upstream-body ceiling (8 MiB by default). Heartbeats do not satisfy the first
data-event deadline. Read/write inactivity and the whole exchange have separate
deadlines; the latter includes connection and downstream writes. All durations
must be positive and the first/idle limits must fit the total limit.

Client disconnects cancel pending sends, reads and writes, including a silent
provider. Clients must keep their connection open while receiving the stream;
a closed write half or additional request data terminates the exchange. The
parallelism slot stays occupied until forwarding finishes or aborts. Redirects,
environment proxies and request retries are disabled. Cancellation closes the
broker connection; it does not prove that a remote provider stopped generation
or will waive charges.

Before output, failures return bounded broker JSON errors when time remains;
an exhausted whole-exchange deadline closes the connection. After output starts,
a failure closes the incomplete chunked response. The broker never manufactures
`[DONE]` or a Responses completion. Valid `response.failed` and
`response.incomplete` events retain their actual outcomes. A terminal event is
released only after the audit record is durably synced and its parent directory
is synced to preserve log creation or rotation. A sync failure closes the stream
without releasing that event. Later client delivery failure
is recorded as an abort when the audit remains writable. Audit completion
admission is not proof that the client received every byte.

Budget reservation remains conservative for every request, including successful
streams and missing usage: reported usage does not refund tokens or cost.
Chat usage is optional; Responses usage comes from the terminal response. Only
consistent numeric input/output/total token counts enter audit. Malformed or
missing usage is `null`; prompts, tool arguments, output and provider error text
never enter audit. A stream ending without its genuine terminal event fails.

The implementation uses the pinned `reqwest 0.12.28` and `tokio 1.52.3`, with
explicitly disabled HTTP retry policy. Protocol references are the official
[Chat streaming events](https://developers.openai.com/api/reference/resources/chat/subresources/completions/streaming-events)
and [Responses streaming events](https://developers.openai.com/api/reference/resources/responses/streaming-events).
Local broker tests exercise forwarding, fragmentation, tools, usage, truncation,
deadlines, disconnects, audit failure and concurrency. They do not establish
OpenClaw runtime support or acceptance against a paid provider.

## Fetch policy

The fetch broker exposes only:

```text
POST /v1/fetch
```

Its body is `{"url":"https://exact-host/path"}`. It requires HTTPS and an
exact lowercase hostname. Every initial target and redirect is resolved and
checked again. The broker rejects loopback, private, link-local, metadata,
CGNAT/Tailnet, multicast, documentation, benchmark, and other non-public IP
ranges. All DNS answers must be public, and the validated addresses are pinned
for the request to block rebinding.

Responses are bounded by size/time and exact media type. Raw bytes are written
to a mode-0700 quarantine directory. Returned JSON strips unsafe control
characters and includes both `trust = "untrusted_external_content"` and an
instruction boundary. This narrows exposure; it cannot prove that content is
free of prompt injection.

## Failure and revocation

The brokers fail closed when authentication, policy, DNS, upstream TLS,
budget-state persistence, quarantine, or audit persistence fails. Budget
reservations are conservative and are not refunded after a failed upstream
call.

`GET /healthz` verifies that the virtual key, provider credential when
applicable, and prompt-free audit path are readable. Broker units restart on
failure with exponential backoff from ten seconds to one minute. Explicit
operator stops prevent further retries.
Broker units use `RestartMode=direct`: automatic retries keep dependent
controller processes running instead of stopping and starting them. The broker
endpoint remains unavailable during recovery, so in-flight calls may fail;
the existing health probes still report unavailable or non-ready service.
Explicitly stopping a broker stops its `Requires` controllers. Starting the
broker afterward does not restart those controllers automatically.

Direct retries skip systemd `OnFailure`/`OnSuccess` hooks. Monitor broker health
and restart counts rather than relying on those hooks for transient outages.
The generated security manifest records each exact host endpoint. The security
doctor treats connection failure as unknown (`TFSEC-035`) and an answering but
non-ready broker as high severity (`TFSEC-036`). Label/mode/endpoint drift is a
critical desired-state inconsistency (`TFSEC-037`).

To revoke one agent immediately, use the combined CLI kill switch:

```text
tentaflake stop <agent>
```

Runtime mutation still requires an operator decision. Do not delete or
recreate the network automatically; if a subnet declaration changes, the
network unit detects the drift and fails until the operator resolves the exact
old network. The same fail-closed migration applies to networks created before
the deterministic bridge-interface option existed: inspect the exact stopped
agents and old network, remove/recreate that one network in an approved runtime
window, then restart its broker/controller units.

## Verification boundaries

Repository checks prove formatting, static policy logic, mock credential
substitution, SSRF literals, prompt-free audit, redirect revalidation, and a
deterministic DNS-rebinding fixture where an allowed name changes from a public
to a blocked answer. Nix evaluation covers the declarative policy. The VM
suite starts a scoped local LLM broker, reaches it from an isolated `runsc`
capsule, and receives a fixture response; this validates the permitted network
path but not external provider TLS. A full VM or host test must additionally
prove the OCI backend's internal network, nftables rules, systemd credentials,
live provider TLS, restart behavior, and cross-agent denial. The repository VM
declares Docker and Podman internal bridges, checks their deterministic
interface names, exercises `runsc` with both backends, and checks denial from a
separate external node; it still does not prove a real provider, authoritative
rebinding DNS server, or tailnet. A successful source evaluation is not
live-runtime proof.

---

# Disposable tool worker and approval boundary

> Run host tool work in a disposable worker behind approval.

Canonical page: https://docs.tentaflake.dev/configuration/disposable-worker/

> Source: [`docs/13-disposable-worker.md`](https://github.com/timfewi/tentaflake/blob/e21be475a3afc6e95c5087003edd0b7f780e4df4/docs/13-disposable-worker.md) at `e21be475a3af` · docs version **0.4.37**

## Result

`tentaflake.worker.agents.<container>` creates a per-controller host queue for
untrusted shell, build, and foreign-code jobs. The controller writes only a
small JSON request. A host-side Rust orchestrator captures the request into
private state, takes a bounded workspace snapshot, and creates a short-lived
OCI container with `runsc`, `network=none`, numeric non-root user,
`cap-drop=ALL`, `no-new-privileges`, read-only root, bounded tmpfs, and
CPU/RAM/swap/PID/runtime limits. Docker workers explicitly select
`docker-default` AppArmor; Docker/Podman retain their default seccomp policy
and unconfined overrides are never exposed through the request. The worker
receives no agent, provider, Git, backup, or infrastructure credential.

The worker is a second containment boundary. It does not make the controller,
OCI daemon, kernel, gVisor, worker toolchain, or submitted code trustworthy.
It is optional for a deliberately stopped scaffold, but an automatically
started balanced controller fails evaluation without its matching worker.

## Enable one worker

The key must exactly match the OCI container generated by the builder. The
workspace and numeric UID/GID must also match; evaluation fails otherwise.

```nix
tentaflake.worker.agents.hermes-coding = {
  enable = true;
  workspace =
    "/var/lib/hermes-coding/workspace";
  containerUid = 10000;
  containerGid = 10000;

  maxSnapshotBytes = 1024 * 1024 * 1024;
  maxSnapshotEntries = 200000;
  maxTimeoutSeconds = 900;
  memory = "2g";
  memorySwap = "2g";
  cpus = "2.0";
  pidsLimit = 512;
  workspaceTmpfsSize = "2g";
  tmpTmpfsSize = "256m";
};
```

ZeroClaw uses `/var/lib/zeroclaw-<name>/data` with UID/GID 65534.

The host service runs with a real group-database entry matching the numeric
container GID. Tentaflake creates one deterministic `tfw-gid-<gid>` group per
non-nogroup GID and reuses `nogroup` for GID 65534. If a deployment already
owns a custom container GID, set `hostGroup` to that existing group; its
explicit host GID must match or evaluation fails.

The default image is built by Nix from the pinned nixpkgs input. It contains an
offline Rust/C/build toolchain. `CARGO_NET_OFFLINE=true` is set, and the
worker has no network namespace route regardless of tool configuration. A
deployment may replace `worker.image`; its default reference follows the
image's `imageName` and `imageTag`. Set `worker.imageReference` explicitly for
images without that metadata. The module loads the Nix derivation locally
without pulling the reference from a registry.

## Submit a job

Write one complete file atomically when practical:

```json
{
  "version": 1,
  "id": "cargo_test_1",
  "action_class": "local-reversible",
  "argv": ["cargo", "test", "--offline"],
  "timeout_seconds": 300
}
```

The path is:

```text
<workspace>/.tentaflake-worker/inbox/
  cargo_test_1.json
```

IDs are bounded lowercase ASCII identifiers. The parser rejects unknown
fields, oversized requests, empty or oversized argument lists, NUL bytes,
unsupported versions, duplicate IDs, and timeouts above policy.

The snapshot walker opens the configured workspace through Linux `openat2`
with `RESOLVE_NO_MAGICLINKS | RESOLVE_NO_SYMLINKS`. On kernels or filtered VM
environments that return `ENOSYS`, it falls back to descriptor-relative,
component-by-component `openat` calls with `O_DIRECTORY | O_NOFOLLOW` and
rejects relative paths plus dot or parent components.
It then traverses by directory/file descriptors, does not follow file
symlinks, excludes the queue directory, rejects special files, and stops at
byte and entry ceilings. This prevents an agent-controlled top-level symlink
or rename race from turning the root worker into a reader of an unrelated host
path. The private host snapshot stays root-owned, while the source owner's read
and execute bits are mirrored to the already-declared capsule GID before the
read-only bind mount. Private workspace entries therefore remain readable
without granting the host worker `CAP_CHOWN`. Inside the capsule, the job user
recreates the snapshot without preserving host ownership metadata as
additional authority.

## Action classes and approval

| Class | Behavior |
|---|---|
| `local-reversible` | runs automatically in the offline disposable capsule |
| `external-reversible` | requires host approval |
| `irreversible` | requires host approval |
| `financial` | requires host approval |
| `production` | requires host approval |
| `communicative` | requires host approval |
| `privileged` | requires host approval, but gains no extra worker authority |
| `forbidden` | always rejected |

The host captures a validated request below
`/var/lib/tentaflake-worker-<container>/pending/` before approval. The agent
cannot mutate that private copy. Approve or deny the exact ID from an operator
session:

```text
sudo tentaflake-worker --config \
  /etc/tentaflake/workers/hermes-coding.json \
  approve message_1

sudo tentaflake-worker --config \
  /etc/tentaflake/workers/hermes-coding.json \
  deny message_1
```

The root operator command adopts the worker's declared capsule group before it
creates a snapshot. Direct approval therefore uses the same read-only snapshot
permissions as the systemd queue service and does not require `CAP_CHOWN`.

Approval is an authorization decision outside the LLM. It does not change the
capsule: approved jobs still have no network, secrets, host devices, runtime
socket, added capabilities, or writable host bind. Therefore an approved
`communicative` or `financial` job cannot itself perform that external action.
Such effects require a separate narrow broker; no generic external-side-effect
broker is implemented yet.

## Results and cleanup

The worker captures bounded runtime logs and exit status. Docker capsules use
the `local` driver with a one-file size cap and compression disabled; Podman
capsules use bounded `k8s-file` logging. The worker separately caps the
published `job.log`. A job may place files below `artifacts/` in its tmpfs
workspace. After the job command returns, the capsule wrapper records its real
exit status and keeps the container alive briefly. The worker streams a tar
archive through `docker exec` or `podman exec` from inside the `runsc` mount
namespace while the bounded tmpfs is still mounted. Its host importer enforces
the snapshot byte and entry ceilings, accepts only relative regular files and
directories, rejects links and special files, and normalizes result
permissions. The worker rechecks the completion marker to exclude a concurrent
handoff and only then signals the wrapper to exit. The import lands in the
worker's private result directory and is exposed read-only to the controller:

```text
/run/tentaflake-worker/results/<id>/
  result.json
  job.log
  artifacts/
```

The controller chooses whether to copy a reviewed artifact into its persistent
workspace. The worker never writes an artifact directly back into that
workspace, and the capsule receives no writable host bind for this handoff.
Result directories expire after 14 days; the prompt-free JSONL audit rotates
daily with 14 generations.

## Queue ownership, capacity and recovery

Worker drains and operator approval/denial commands hold one per-agent private
file lock. A competing command fails visibly with a busy diagnostic; it does
not modify approvals or start a second execution. Requests and approvals are
written to private staging files, synced, then published atomically without
replacing an existing record. The inbox can live on a separate quota filesystem;
capture therefore copies into private state before removing the inbox entry.

| Durable state | Transition and restart behavior |
|---|---|
| `pending/<id>.json` | Validated immutable request; approval remains outside the agent. |
| `running/<id>.json` | Atomically claimed and synced before any OCI dispatch. |
| `results/<id>/result.json` | Terminal result and artifacts synced before publication. |

After a crash, a running claim is never dispatched again. Recovery checks the
exact container name and ownership label before removing its disposable
container and private staging. If no terminal result exists, it publishes
`interrupted`: the execution outcome is unknown. Review it before submitting a
new ID. A completed result with a stale private request is retained and the
request retired without replay. Runtime listing/ownership errors preserve the
claim and fail closed. This is conservative crash recovery, not a promise of
exactly-once effects across operator rollback or deleted state.

`maxInboxEntries` and `maxInboxBytes` default to 256 entries and 8 MiB;
`maxQueueEntries` and `maxQueueBytes` apply the same defaults jointly to private
pending and running requests. Inbox scans count ignored names and staging files
before allocating an unbounded list. Summed entry sizes include links and
directories, but do not traverse directory trees; the workspace quota bounds
those trees. Admission rejects overflow visibly, records `queue-overload`, and
preserves the unaccepted inbox request. Existing ready jobs drain before new
admission, so an overflowing inbox cannot prevent completed work from retiring.
These ceilings cover requests, not retained results, snapshots or audit archives;
their existing separate limits and retention still apply.

Existing configurations without the new JSON fields use these defaults.
Upgrade with the worker and watcher stopped and preserve a quota-aware backup.
Complete legacy pending records and exact approvals remain valid. A legacy
`jobs/<id>` snapshot with pending state is treated as abandoned execution and
retired as `interrupted`, without replay. Unpublished private `.new` files are
discarded during recovery. Malformed legacy pending records fail closed and
preserve a complete inbox copy; inspect the exact job and restore or quarantine
its private record offline before restarting. If restored queues exceed the new
ceilings, explicitly adjust the ceilings or review and reduce the queue offline.
Review pending/running requests before activating a restored backup: a rollback
can predate work that already executed. No existing image is reformatted and
quota-aware backup sources are unchanged.

The path-activated queue service has a ten-second failure-restart delay but no
aggregate systemd start counter. systemd counts successful oneshot activations
against that counter as well, which would otherwise disable a healthy queue
after a small number of ordinary jobs. Per-request resource, timeout, snapshot,
and workspace ceilings remain enforced independently.

Tmpfiles establishes the result root as `02750`; the orchestrator preserves
that inherited setgid bit without trying to reapply it inside the hardened
systemd unit. Published result files therefore retain the controller's exact
numeric group while `RestrictSUIDSGID=true` remains enforced.

The OCI container and temporary host snapshot are removed after every normal
completion, failure, or timeout. A stale container is removed only when its
Tentaflake ownership label exactly matches the configured agent.

## Verification boundary and remaining risks

Unit tests cover parser/action policy, bounded reads and snapshots, symlink
rejection, control-directory exclusion, and log truncation. Module-evaluation
tests cover builder matching and fail-closed configuration. The NixOS VM test
submits real local, approval-required, and forbidden jobs, checks no network or
runtime socket, validates the numeric user and artifacts, and proves container
and snapshot cleanup. That runtime test uses Docker; Podman evaluation is not
live Podman/AppArmor proof.

A derivation evaluation is not proof that Docker/Podman, `runsc`, systemd path
units, tmpfs ceilings, or cleanup worked on an activated host. Run the VM check
and then a deployment-specific live drill before relying on it. The module
also cannot transparently intercept every runtime's built-in terminal tool;
configure the agent/tool policy to submit untrusted execution to this queue.
The posture doctor reports `TFSEC-020` when a secure-profile controller has no
worker, but it cannot prove that every model-selected command used it.

---

# Persistent workspace and state quotas

> Bound and migrate agent workspaces and optional private state.

Canonical page: https://docs.tentaflake.dev/configuration/workspace-quota/

> Source: [`docs/14-workspace-quota.md`](https://github.com/timfewi/tentaflake/blob/e21be475a3afc6e95c5087003edd0b7f780e4df4/docs/14-workspace-quota.md) at `e21be475a3af` · docs version **0.4.37**

## Result

`tentaflake.workspaceQuota.agents.<container>` mounts a fixed-size Btrfs
filesystem at one controller workspace, and optionally a separate one at its
private state source. Each backing file has an exact declared
size of at least 128 MiB, so both controller writes and worker-inbox writes receive a real
filesystem `ENOSPC` instead of consuming the host root filesystem without a
per-agent ceiling.

A deliberately stopped balanced scaffold may omit this disk mutation. An
automatically started balanced controller must declare its matching quota;
evaluation fails until the path and numeric owner match the runtime builder.

Private state is opt-in through `state`; omitting it preserves existing state
and workspace behavior. It bounds sessions and caches written to the adapter's
state mount. Broker/worker audit, backups, other host paths and aggregate sparse
image allocation still require host free-space monitoring. This change does not
relax startup admission or establish acceptance of a new agent runtime.

## Declaration

The key, path, and owner must match the runtime builder:

```nix
tentaflake.workspaceQuota.agents.hermes-coding = {
  enable = true;
  workspace =
    "/var/lib/hermes-coding/workspace";
  sizeMiB = 8192;
  ownerUid = 10000;
  ownerGid = 10000;
  state = {
    path = "/var/lib/hermes-coding";
    sizeMiB = 1024;
  };
};
```

ZeroClaw uses numeric UID/GID 65534 by default. Builder assertions
reject a path or owner mismatch. The agent unit requires the mount/ownership
unit, so a missing image, failed filesystem check, mount failure, or ownership
failure prevents the controller from starting.

`state.path` must exactly match the adapter's declared `stateStorage`, with the
same numeric ownership as the agent. Hermes and ZeroClaw use their existing
state directory; generic workloads use `/var/lib/generic-NAME/state` beneath
the unmounted root-owned instance parent. State sizes default to 1024 MiB and
must be at least 128 MiB. Paths must be normalized, outside backing images,
and outside the workspace. Mounts may not overlap another agent's filesystems.

## First activation is a disk mutation

Enabling the option and activating the resulting NixOS generation creates:

```text
/var/lib/tentaflake-workspace-volumes/
  hermes-coding.img
  state/hermes-coding.img
```

The prepare unit creates the exact-size sparse file, formats it as Btrfs, and
mounts it with `loop,nodev,nosuid,noatime`. This is an intentional filesystem
mutation. A source build or evaluation does not perform it and does not
authorize activation.

The managed mount starts after the host's ordinary local filesystems and
tmpfiles setup. Its ownership unit then recreates the private worker-control
directories inside the mounted filesystem before the controller or worker
watcher may start. The inbox is writable only by the exact agent UID/GID and
the host worker's statically declared matching service group. Unsafe symlink
or non-directory control paths fail closed. Private state ownership is a
dependency of workspace ownership, so its failure or explicit stop also stops
dependent controllers. For legacy layouts with a workspace below state, the
state mount and directory initialization complete before workspace preparation.
Seeds and UID healing wait for these ownership units.

The module checks each mount source and its parents for symlinks/non-directories
before changing permissions or preparing an image. State initialization also
checks adapter-owned child directories before creating or changing them.
The module refuses to mount over non-empty state/workspace sources, including
files written below an unmounted path while its backing image already exists.
It tolerates only the empty `.tentaflake-worker/inbox` scaffolding that NixOS
tmpfiles may create during workspace activation; private state must be empty.
It also rejects a symlink/non-regular
backing path, a leftover `.new` file during image creation, or any size mismatch. It never mounts
over existing data, shrinks, grows, or reformats an existing image silently.
Existing ext4 images are rejected without modification and require the explicit
backup/restore migration below; this update does not convert them in place.

## Existing workspace migration

The Restic backup module includes enabled quota mounts inside its selected
`paths` as separate sources and requires them to be mounted before backup.
Selecting the parent agent-state directory covers both state and its quota
workspace; generic instance parents also include their separate state mount.
Only mounts within selected trees are included. Other nested filesystems need
explicit backup paths. Restore files
into the mounted state/workspace with the agent stopped; do not overwrite a live
backing image. A live file backup does not provide application-level snapshot
consistency.

Perform this only in an approved maintenance window and adapt paths to the
exact stopped agent:

1. Back up and verify the existing workspace.
2. Stop the exact agent and its brokers.
3. Move the old workspace to a reviewed temporary path on the same host.
4. Activate the generation with the quota declaration; confirm the empty
   filesystem mounted and has the configured byte size.
5. Copy reviewed data into the mounted workspace without crossing its limit.
6. Verify numeric ownership, run the security doctor, and start the agent.
7. Keep the verified backup until a restore/start drill passes.

Tentaflake intentionally provides no automatic migration command because a
wrong source, destination, or size would be destructive.

## Adding a private state quota to an existing agent

Use the same explicit backup/restore procedure for `state.path`. Stop the
controller, worker, seeds and any other state writers first. For a workspace
already mounted below that state path, unmount it before moving the old state
directory. Preserve both its backing image and old state data. Do not copy a
mounted child filesystem or recreate worker scaffolding in the empty state
mount point. After activation, verify state mounted first and the existing
workspace image mounted beneath it; restore only reviewed state files, excluding
the separately backed-up workspace. Verify ownership, state and workspace
contents, both limits and a restore/start drill before removing originals.
No automatic migration or host activation is provided.

## Migrating existing ext4 quota images

Stop the controller and worker, back up the mounted workspace files, and verify
that backup. Unmount the workspace and preserve the exact old backing image at
a separate reviewed path. Declare at least 128 MiB, then activate the Btrfs
configuration to create a new empty image. Restore the files into that mounted
workspace and verify ownership, restored contents, the size ceiling, and agent
startup before discarding any original image or backup. Copy files, not an ext4
image, into the new Btrfs workspace. No automated in-place conversion is provided.

## Resize and recovery

Changing `sizeMiB` while the image exists fails closed. Resize requires an
explicit backup/restore into a new Btrfs image of the declared size, with the
controller and worker stopped and the old image preserved until verification.
The module never grows or shrinks a live image automatically.

At boot, the prepare service checks the filesystem type and runs
`btrfs check --readonly` only while the expected image is not mounted at its
target. A failed check prevents the mount and therefore prevents controller startup. It performs no
automatic repair; preserve the image and restore from a verified backup rather
than running `btrfs check --repair` as an automatic recovery step.

`ReadWritePaths` creates a private bind mount in the service namespace even
when the quota is absent. Both preparation and ownership identify the exact
loop backing image; a mount-point or filesystem-type check alone is insufficient.
Ownership refuses an absent or mismatched image and verifies its configured
byte size before modifying directories. Changing a size also changes the
ownership unit, so an active mount cannot retain stale size admission.

## Verification boundary

Module tests prove declaration matching, mount dependencies and fixed path/options.
The adapter check executes generated quota helpers against temporary symlink,
non-empty state, incomplete/unsafe image, size and filesystem-format fixtures;
unrelated permissions and contents must stay unchanged. The VM test boots state
and workspace mounts, verifies the
exact backing-file size and Btrfs type, and confirms a write beyond the limit
fails. It also checks reboot persistence, backup/restore across the mount, and
rejection of legacy ext4 images with checksums unchanged. It also exercises
unit stop propagation and state-directory symlink refusal inside the unit
namespace. Without a
completed VM/live run, evaluation alone is not quota proof.

---

# Exclusive web research

> Configure the pinned research relay and optional VPN readiness evidence.

Canonical page: https://docs.tentaflake.dev/configuration/research/

> Source: [`docs/16-research.md`](https://github.com/timfewi/tentaflake/blob/e21be475a3afc6e95c5087003edd0b7f780e4df4/docs/16-research.md) at `e21be475a3af` · docs version **0.4.37**

Tentaflake uses the public MIT-licensed
[tentaflake-research](https://github.com/timfewi/tentaflake-research) service for
web access. Its stdio MCP server is named `secure-research-tool`; it exposes
`research_job`, `research_search`, `research_fetch`, `research_browser` and
`research_read`. Model requests continue through the separate LLM broker.

The service is a pinned flake input, imported by `nixosModules.default`. An
operator must configure the VPN and service before enabling a research agent.
The template does not create an Internet route or obtain provider credentials.

## Configuration

Declare a registered Research-capable adapter with `mkAgent` or either
compatible runtime builder, then use its complete OCI container name:

```nix
tentaflake.research.agents = {
  hermes-assistant.uid = 62101;
  zeroclaw-assistant.uid = 62102;
};
services.secureResearch = {
  serviceUid = 4201;
  egressUid = 4202;
  vpnInterface = "wg0";
  resolvers = [ "9.9.9.9" ];
};
```

Choose distinct unused host UIDs. Each relay has its own host UID, even when
two containers use the same internal UID. The service authorizes job ownership
using the relay's peer credentials. These UIDs are service identities, not host
user-namespace remapping for the containers.

This fragment does not configure WireGuard or VPN readiness. Select the reference
observer below, or supply fresh, trusted observations under the same lease contract
described in the upstream
[operations guide](https://github.com/timfewi/tentaflake-research/blob/main/docs/operations.md)
and [egress control](https://github.com/timfewi/tentaflake-research/blob/main/docs/egress-control.md).
Missing or stale readiness prevents public requests. Optional search providers,
Chromium and OCR require their own explicit configuration. Provider keys and
data grants belong in the research service; never put keys in agent settings or
the Nix store. Research summarization remains disabled in this topology.

Keep the agent's [LLM broker](https://docs.tentaflake.dev/configuration/brokered-egress/),
[disposable worker](https://docs.tentaflake.dev/configuration/disposable-worker/) and
[workspace quota](https://docs.tentaflake.dev/configuration/workspace-quota/) declarations. A secure automatically
started controller requires all four policies. A stopped scaffold can remain
offline without research. The old fetch broker must be disabled for every
research-enabled agent; its configuration fails evaluation otherwise.

## VPN readiness selection

`nixosModules.default` already imports the pinned Research module. Configure its
`services.secureResearch.vpnObserver` options directly; there is no separate
Tentaflake observer wrapper. The observer is disabled by default, and its extra
evidence selections are opt-in. An operator-owned producer of the generic
observation lease remains valid; no VPN vendor is required.

For an existing, operator-configured WireGuard exit, extend the service declaration:

```nix
services.secureResearch = {
  vpnInterface = "wg0";
  resolvers = [ "9.9.9.9" ];
  drainSeconds = 60;
  vpnObserver = {
    enable = true;
    linkKind = "wireguard";
    egressPathEvidence = true;
    handshakeWithinSeconds = 180;
    # Illustrative public key: replace with the selected exit's exact peer key.
    peerPublicKeys = [ "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=" ];
    firewallMarker = "/run/research-vpn/firewall-ready";
    drainMarker = "/run/research-vpn/drain";
  };
};
```

This does not create a tunnel, enroll management, install VPN credentials or
declare that an exit is ready. Keep VPN credentials on the host. Only the
operator's firewall installer creates `firewallMarker`, after establishing the
independent egress firewall; creating the marker alone proves no kernel rule.
Markers must be distinct regular root-owned files without group/world write
access. Their ancestors must be real root-owned directories, without symlinks
or write access for other identities. A missing drain marker means no drain;
an unsafe marker makes readiness fail closed.

| Selection | Evidence and limits |
|---|---|
| `linkKind = "wireguard"` or `"tun"` | Checks the selected link kind and that the interface is up. The default is `null`; interface presence alone does not prove a usable exit. |
| `egressPathEvidence = true` | Evaluates the configured `egressUid` through policy-routing rules: unmarked IPv4 must use the tunnel, IPv6 must use it or be denied, and every `services.secureResearch.resolvers` address must use it. The default is `false`. |
| `handshakeWithinSeconds` | Selects WireGuard handshake-age evidence. Use 180 seconds or more; loss can take the selected window to detect. The default is `null`. |
| `peerPublicKeys` | Pins the exact WireGuard peer set, at most four base64 public keys. These values are public, not VPN secrets. The default is `[]`. |
| `drainMarker` | Requests a planned exit change. The default is `null`; it must differ from `firewallMarker`. |

Handshake and peer-key selections require `linkKind = "wireguard"`. They grant
only the root observer unit `CAP_NET_ADMIN` to read kernel WireGuard evidence;
agents receive no added capability, credential or readiness hook. Link/path-only
observation has no capabilities. The observer retains only public peer keys and
handshake times and must never log the kernel dump or secret attributes.

Tailscale or Headscale management membership supplies neither an Internet exit
nor proof of one. For a separately configured exit using a layer-3 tun interface,
select `linkKind = "tun"` and `egressPathEvidence = true`, leaving WireGuard
handshake/key options unset. Tun exit identity and liveness are not observed or
accepted by this reference implementation. `region` is an optional operator
assertion, as is `firewallMarker`; neither substitutes for observed link/path
evidence. Conservative or unknown routing evidence withholds readiness. The
independent kernel firewall remains responsible for direct IPv4, IPv6 and DNS
denial if the observer, controller or both fail.

See the pinned upstream
[VPN evidence contract](https://github.com/timfewi/tentaflake-research/blob/91ce54005c3c69a3eab9d8e2730ca9d2802d2c96/docs/vpn-adapters.md)
for route evaluation and its limits. Agents cannot select exits, write
observations or control leases, execute readiness hooks, or access VPN credentials.

## Planned exit change

An exit change is an operator action requiring its own authorization. Keep the
old tunnel and its firewall protection intact throughout draining. The observer
renews observations for at most ten seconds; the controller publishes leases for
at most three seconds. Repeated drain observations cannot extend the controller's
monotonic `drainSeconds` deadline (1–300 seconds, default 60).

1. Read the controller's control lease, not just the observer's input, and record
   the current `ready` generation. The default path contains no credentials,
   queries or exit address. Bound every read by time and bytes:

   ```sh
   sudo timeout 2s head -c 4097 -- /run/agent-research-egress/control/state.json
   ```

   Require a regular, protected root-owned lease of at most 4096 bytes, `version`
   1, a non-nil UUID `generation`, and `valid_until` later than now but no more
   than three seconds ahead. A timeout, missing, oversized, malformed or expired
   lease is unknown/offline, never permission to declare readiness.
2. Verify the marker parent protections above, then create the configured drain
   marker as root. For the example path, refuse an existing path or symlink and
   use noclobber; do not replace another operator's marker:

   ```sh
   sudo bash -c '
     set -eC
     marker=/run/research-vpn/drain
     test ! -e "$marker"
     test ! -L "$marker"
     umask 077
     : > "$marker"
   '
   ```

3. Repeat the bounded control read until a fresh lease reports `draining` with
   the saved generation. Allow at most `vpnObserver.refreshSeconds + 3` seconds
   for this transition (refresh defaults to five seconds). If it does not occur,
   leave the old exit protected and investigate; a marker alone does not prove
   controller drain admission. New work is refused once the controller drains.
4. Keep the old exit and firewall protected until existing work finishes or the
   drain deadline expires. Without independent completion evidence, wait for a
   fresh `offline` control lease; after a stalled controller, wait at least the
   configured `drainSeconds` from the first observed drain and for every last
   valid control lease to expire. Bound the wait to `drainSeconds + 3`; if the
   condition cannot be established, stop and inspect rather than changing exits.
5. Only after that completion/offline/deadline condition, change the host-owned
   exit configuration while retaining fail-closed firewall protection. Preserve
   the firewall marker and controller lock; do not fabricate a readiness lease.
6. Remove only the selected drain marker:

   ```sh
   sudo rm -- /run/research-vpn/drain
   ```

   Confirm a fresh `ready` controller lease with a generation different from the
   one saved before draining. The observer must re-prove all selected evidence,
   even if the exit ultimately stayed the same. If proof fails, remain offline.

## Enforced boundary

The shared projection installs the pinned `research-client` and its exact runtime closure
as read-only mounts. Only `/run/tentaflake-research/CONTAINER` is projected into
that agent as `/run/tentaflake-research`. The root-only host parent hides other
agents' sockets. The container receives no research credentials or upstream
service socket. Mounts must use normalized paths and retain these exact
read-only capabilities after caller overrides.
The security manifest records the declared mounts. `doctor --security` accepts
the research socket only when its source, destination and read-only mode match
that declaration; broad `/run`, writable and other-agent mounts remain unsafe.

gVisor runs with `--host-uds=open` to connect to projected sockets; host socket
creation remains disabled. Agent IP networking still permits only its declared
LLM broker, or remains `network=none`. Browser, HTTP and shell networking cannot
gain general egress by changing a tool setting.

Adapter hooks generate upstream-specific settings after shared socket/closure
projection. An arbitrary OCI container with a runtime-like prefix cannot opt
into Research; validation checks explicit registered instance metadata and the
adapter's accepted capability. OpenClaw's stopped scaffold has no accepted
Research discovery integration. See [agent adapters](https://docs.tentaflake.dev/configuration/agent-adapters/).

Hermes settings disable the `web` and `browser` toolsets and register
`mcp_servers.secure-research-tool`. Additional MCP entries must be local stdio
commands. ZeroClaw settings disable `browser`, `http_request`, `web_fetch` and
`web_search`, and replace the MCP server list with this stdio service. These
settings are applied after caller settings. The LLM broker permits local
function/custom tool definitions but rejects provider-hosted tools, including
web search and remote MCP, and known provider extension fields. Operators must
select inference-only models; this does not constrain a provider's internal
implementation or make an arbitrary vendor extension safe.

Each relay has at most four connections and no IP networking. Healthy sessions
can remain idle between tool calls. Stop one relay socket to revoke accepted
connections immediately:

```sh
sudo systemctl stop tentaflake-research-hermes-assistant.socket
```

Stopping `agent-research.socket` also stops all relay sockets. Restart the
selected socket and the agent to establish a fresh client session. Upstream
readiness, budgets and source-retention policy remain enforced by the service.

## Verification and limits

The observer option fixture evaluates without building Research closures:

```sh
nix eval --option allow-import-from-derivation false --no-write-lock-file \
  --impure --json --expr '
    let flake = builtins.getFlake (toString ./.);
    in import ./tests/research-vpn-observer.nix {
      self = flake;
      pkgs = flake.inputs.nixpkgs.legacyPackages.x86_64-linux;
    }'
```

The broader policy and runtime gates are separate workloads:

```sh
nix build --no-link .#checks.x86_64-linux.research-policy
nix build --no-link -L .#checks.x86_64-linux.research-integration
```

The policy check exercises remote MCP rejection, missing capability mounts,
duplicate relay UIDs, legacy fetch rejection and disabled summarization. The VM
runs the real stdio client and service inside real Docker/gVisor capsules with
Hermes and ZeroClaw UID values, verifies the generated settings, performs MCP
initialization and job creation, and checks direct IP and offline-fetch denial.
Its container image is a synthetic Python probe. It does not run the actual
vendor agent images or prove their MCP discovery behavior.

Observer option evaluations cover direct module exposure, defaults, selected
link/path/handshake/peer/drain arguments and refusal of invalid combinations.
They establish configuration wiring, not a working tunnel or live firewall.
The pinned upstream separately records a real WireGuard observer/controller VM
fixture; its evidence does not establish Tentaflake host integration or a
non-WireGuard exit node. Issue [#141](https://github.com/timfewi/tentaflake/issues/141)
retains real-tunnel integration and production acceptance. That VM work was
explicitly excluded from the current lightweight-only run; no runtime acceptance
or production activation is claimed here.

The upstream synthetic VM suite separately checks encrypted WireGuard egress,
direct-path denial, parser credentials, rendering, resources, cross-agent
ownership, idle sessions and socket revocation. See its
[verification evidence](https://github.com/timfewi/tentaflake-research/blob/main/docs/verification.md).
Actual Hermes/ZeroClaw sessions, Podman socket transport, paid-provider billing,
production VPN policy, ARM and arbitrary kernel escapes still require deployment
acceptance. No production host is activated by these checks. Retrieved content
remains untrusted; the network boundary does not prevent prompt injection or
disclosure of sensitive text intentionally sent to an allowed destination.

## Client recovery

The pinned stdio client reconnects future calls after a lost upstream Unix RPC
session. Each fresh session repeats protocol negotiation and host peer-UID
authorization. Interrupted calls fail without replaying dispatched operations;
caller cancellation and explicit client close remain enforced. If the service
is unavailable, connection retries remain bounded. Socket revocation still
closes accepted relays and prevents research access.

The public research service fixture verifies a real upstream SIGKILL followed
by a successful new call from the same stdio process, plus the idle-session and
live socket-revocation regressions. Tentaflake's separate Docker/gVisor fixture
checks the packaged MCP interface, exact closure/socket mounts and UID policy.
These synthetic fixtures do not establish actual vendor-agent workloads or
production VPN/provider acceptance.

---

# Agent adapters

> Use the common adapter API and review runtime support and acceptance limits.

Canonical page: https://docs.tentaflake.dev/configuration/agent-adapters/

> Source: [`docs/agent-adapters.md`](https://github.com/timfewi/tentaflake/blob/e21be475a3afc6e95c5087003edd0b7f780e4df4/docs/agent-adapters.md) at `e21be475a3af` · docs version **0.4.37**

Status: common schema-v1 OCI definitions and preset discovery are available.
Generic workloads remain stopped pending capability-specific admission. OpenClaw
remains a stopped scaffold; vendor runtime acceptance is separate.

The requirement is one discoverable adapter per integrated runtime, a common
consumer builder, unchanged Hermes/ZeroClaw state and public arguments, and
explicit instance identities. Verification uses `checks.*.agent-adapters`,
`checks.*.research-policy`, the installer-generated flake regression, Rust CLI
fixtures, and `just fast`. Vendor runtime acceptance is a separate workload.

## Consumer API

```nix
{ mkAgent }:
[
  (mkAgent {
    adapter = "openclaw";
    name = "assistant";
    autoStart = false;
  })
]
```

This creates private state/workspace directories and an intentionally stopped
refusal unit. It creates no OCI container, downloads no image, and grants no
network or Research capability. An explicit start fails with the missing
acceptance requirements. `autoStart = true`, runtime settings, and image/tool
overrides are unavailable until those requirements are verified. This is partial
delivery, not a working secure OpenClaw deployment.

For current runtimes use `adapter = "hermes"` or `"zeroclaw"`. Their supported
arguments and defaults are those of `mkHermesAgent` and `mkZeroClawAgent`.
Both legacy entrypoints remain available. The library exports `mkAgent` and the
lazy `adapters` registry alongside those helpers. Installed and consumer flakes
pass them through `specialArgs`; the installer copies the adapter directory.
An older `my-agents.nix` accepting only `{ mkHermesAgent }` still works because
the importer passes only the helpers that its function requests.

See [the complete secure consumer example](https://github.com/timfewi/tentaflake/blob/e21be475a3afc6e95c5087003edd0b7f780e4df4/examples/adapter-secure.nix) for
Hermes with matching LLM broker, offline worker, Btrfs workspace and Research
relay. Its operator inputs are runtime credential paths, exact model prices,
VPN interface and readiness configuration. There is no operational OpenClaw
example yet: copying those host declarations would not solve tool routing or
model compatibility.

## Generic isolated workload

A local definition needs no new Rust adapter or upstream security module:

```nix
{ mkAgent }:
[
  (mkAgent {
    adapter = "generic";
    name = "coding";
    definition = {
      schemaVersion = 1;
      # Replace with an independently reviewed artifact, never a mutable tag.
      image = "registry.example.invalid/agent@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa";
      command = [ "agent" "--interactive" ];
      ownership = { uid = 10000; gid = 10000; };
      workspace = "/workspace";
      resources = { memory = "2g"; memorySwap = "2g"; cpus = "2.0"; pidsLimit = 512; };
      lifecycle = "stopped";
      capabilities = [ "files" "shell" "terminal" ];
    };
  })
]
```

The same `definition` object is accepted in a schema-v1 JSON `agents` entry.
On installed hosts, [CLI onboarding](https://docs.tentaflake.dev/configuration/agents/#add-an-agent-on-an-installed-host)
creates templates, validates plans and imports stopped additions without activation.
The argument vector follows OCI semantics and preserves the image entrypoint.
Image review/provenance remains operator-owned; a digest alone is not proof of
trust. Evaluation neither downloads the image nor activates the host.

The instance root `/var/lib/generic-NAME` belongs to root and is not mounted
into the capsule. Its separate `state` and `workspace` children belong to the
container UID/GID; `state` mounts at `/state` with `HOME=/state`. The agent cannot
replace its mount sources through a writable parent. Startup refuses symlink
sources or a parent without root ownership and mode 0700. The private `workspace`
child is mounted at the requested workspace path and becomes the working
directory. That destination must be `/workspace` or a
normalized child. Non-root ownership defaults to 10000:10000. Resource fields
inherit host security limits; allowed overrides are `memory`, `memorySwap`,
`cpus`, `nofile`, `pidsLimit`, `tmpfsSize` and `runTmpfsSize`. Values must be
positive. Host administrators remain responsible for appropriate aggregate bounds.

An optional [private state quota](https://docs.tentaflake.dev/configuration/workspace-quota/) uses the same managed
Btrfs mechanism as the workspace, with `state.path` matching the adapter's
`stateStorage`. For this generic instance the path is
`/var/lib/generic-coding/state`; legacy runtimes retain their existing state
roots. Quota activation and existing-data migration remain explicit operator
operations. A quota does not make a stopped workload operational.

Unknown versions/fields, root ownership, unpinned images, extra mounts,
environment/credential inputs, malformed commands and unsupported capabilities
fail before a module is produced. Definitions are non-secret, including command
arguments. Native shell/files run inside the ordinary gVisor capsule; this
contract does not redirect native execution to the worker. `worker` is an
additional declaration requiring matching host workspace/identity policy; no
automatic vendor routing is claimed.

The workload uses the existing host-selected Docker/Podman backend, gVisor,
read-only root, no-new-privileges, dropped capabilities, bounded resources and
`network=none`. It exposes no host/other-agent socket or model credentials.
Generic model and Research hooks are not yet accepted. `lifecycle = "service"`
or forcing `autoStart` still fails current broker/worker/quota/Research gates;
capability-specific admission is tracked in #116. `dev` is unavailable for the
new generic contract. After an explicitly authorized host update, an operator
may manually start an intentionally stopped offline workload and attach with
`tentaflake shell generic-NAME` or `tentaflake exec generic-NAME -- ...`.
These source fixtures do not establish vendor support or live isolation evidence.

### OpenShell reuse decision

Keep the existing OCI/gVisor backend for this contract. The investigated NVIDIA
OpenShell 0.1.x architecture supports separating agent commands from trusted
policy and credentials, which this design adopts. There is no verified NixOS,
gVisor, exact broker or isolated Research compatibility, so OpenShell is not a
required dependency. A future backend needs its own bounded compatibility and
isolation acceptance; selecting another agent must not select weaker containment.
See [the comparison and evidence limits](https://github.com/timfewi/tentaflake/blob/e21be475a3afc6e95c5087003edd0b7f780e4df4/docs/roadmap.md#inspiration-and-evidence-limits).

## Schema version 1

`adapters/default.nix` explicitly imports the generic workload and the Hermes,
ZeroClaw and OpenClaw presets.
Imports are lazy; discovering names requires no packages, image downloads or
services. `mkAgent` validates only the selected adapter. Unsupported options,
invalid settings shapes and unknown adapters fail during evaluation.

| Field | Contract |
| --- | --- |
| `schemaVersion` | `1` |
| `identity` | Runtime ID, investigated version and `compatibility`, `scaffold`, `definition` or `operational` status |
| `artifact` | Reviewed digest-pinned OCI reference, unselected scaffold, or operator-supplied generic instance artifact |
| `command` | Upstream default argument vector |
| `configuration` | Configuration format (`none` for generic), read-only generation and `generate name settings` hook |
| `ownership` | Default positive numeric UID/GID |
| `layout` | Default state pattern, workspace suffix, writable areas and relative initialization directories |
| `lifecycle` | Managed service or stopped scaffold |
| `model` | Protocol families and `required`, `optional` or `unknown` streaming requirement |
| `research` | Accepted capability and upstream settings hook; tool discovery needs separate runtime evidence |
| `execution` | Worker queue interface and whether automatic routing is verified |
| `build` | Runtime module generator with its precise supported argument set |
| `metadata` | Effective instance, container, unit, logical state root, actual `stateStorage` source, initialization directories, workspace and numeric ownership |
| `capabilities` | Declared available hooks; declarations do not prove vendor acceptance |
| `evidence` | Source/refusal regression level and separately recorded vendor acceptance |

Adapter facts do not grant host authority. Shared `containerSecurity`, Research
socket projection, broker, worker, quota, image provenance and recovery modules
retain policy ownership. The shared `containerSecurity.apply` builder validates
the final image and owns containment for generic, Hermes and ZeroClaw workloads. Legacy native features
remain in their existing builders.

`tentaflake.agentInstances` is a read-only map derived from adapter declarations.
Inventory consumes explicit adapter/name/container/unit/state fields, keeps the
five-column TSV contract and retains the original prefix/first-volume fallback
for unmanaged OCI containers. Duplicate effective container identities fail,
including duplicates across Nix imports and JSON input.
Different runtimes may retain the same short name; use their complete container
identities in CLI commands to select one unambiguously.

## Compatibility and state

| Runtime | Container / unit | State / workspace | Container owner |
| --- | --- | --- | --- |
| Generic workload | `generic-NAME` / `BACKEND-generic-NAME.service` | `/var/lib/generic-NAME` / `workspace` | 10000:10000, with definition ownership |
| Hermes | `hermes-NAME` / `BACKEND-hermes-NAME.service` | `/var/lib/hermes-NAME` / `workspace` | 10000:10000, retaining overrides |
| ZeroClaw | `zeroclaw-NAME` / `BACKEND-zeroclaw-NAME.service` | `/var/lib/zeroclaw-NAME` / `data` | 65534:65534 |
| OpenClaw scaffold | Reserved `openclaw-NAME` identity / refusal unit | `/var/lib/openclaw-NAME` / `workspace` | 1000:1000 |

Custom existing state directories, host users, seed destinations, no-clobber
copying, numeric owners, commands and optional operational services stay intact.
There is no automatic state move, deletion, conversion or reset. OpenClaw's
upstream `/home/node/.openclaw` mount is only a future runtime layout; the current
scaffold mounts nothing. Existing ext4 workspace migration remains explicit;
new managed workspaces are Btrfs and at least 128 MiB.

## JSON input

```json
{
  "schemaVersion": 1,
  "agents": [
    { "adapter": "hermes", "name": "coding", "autoStart": false },
    { "adapter": "openclaw", "name": "assistant", "autoStart": false }
  ]
}
```

The generic `agents` array requires the declared `schemaVersion`. Entries use
the selected builder's arguments. Legacy `hermes` and `zeroclaw` arrays remain
supported without a version; their existing fields and generated settings are
retained. Generic and legacy arrays are additive, with no replacement or hidden
precedence. Effective container collisions fail. Unknown root/entry fields,
unknown adapters, malformed entries and unsupported schema versions fail with
the relevant input location. Input remains non-secret; real provider keys are
host runtime files, never settings or environment values in the Nix store.

## Research and execution

`researchClient.nix` projects only the shared pinned client closure and exact
per-agent read-only relay socket. Upstream settings are adapter hooks. Research
validation requires an explicit registered adapter with accepted capability,
rather than trusting an OCI name prefix. Hermes disables native web/browser
toolsets and accepts only local stdio MCP additions. ZeroClaw disables its
native web tools and replaces its MCP list with the isolated stdio server.
Remote additions, legacy fetch, missing read-only mounts and other-agent sockets
remain rejected. Revocation and peer-UID isolation remain service-owned.

Declaring a worker is not automatic mediation. Current adapters expose the
existing queue/results interface; this change adds no automatic interception
of built-in shell, build, plugin or foreign-code tools. The threat model and
[worker controls](https://docs.tentaflake.dev/configuration/disposable-worker/) retain that limitation.

## Support and upstream evidence

`adapters/catalog.json` is the versioned source of preset, capability and support
facts. Nix imports it, `tentaflake runtimes [--json]` embeds it in the CLI, and
`python3 scripts/runtime-catalog-docs.py` renders this table. Its `--check` mode
runs in `checks.*.agent-adapters`. A declaration or source regression is not
actual vendor runtime acceptance. The generic definition's artifact is chosen
and reviewed by the operator; its catalog entry selects no image.

<!-- runtime-catalog:start -->

| Preset | Version | Declaration | Fixture | Vendor acceptance |
| --- | --- | --- | --- | --- |
| generic | 1 | definition | source-regression | operator-owned |
| hermes | image-snapshot-2026-07-18 | compatibility | source-regression | pending |
| openclaw | 2026.9.7 | scaffold | refusal-regression | pending |
| zeroclaw | 0.8.2 | compatibility | source-regression | pending |

<!-- runtime-catalog:end -->

OpenCode and Goose are the next investigations. Codex, Claude Code, Aider, Pi
and OpenHands remain candidates. No adapter files or support claims are added
for them.

Primary OpenClaw sources read through the isolated Research service:

- [v2026.9.7 Dockerfile](https://github.com/openclaw/openclaw/blob/v2026.9.7/Dockerfile): Node user UID 1000, private `/home/node/.openclaw` state, default gateway command and image entrypoint.
- [v2026.9.7 license](https://github.com/openclaw/openclaw/blob/v2026.9.7/LICENSE): MIT, OpenClaw Foundation; third-party notices require review before packaging.
- [v2026.9.7 configuration reference](https://github.com/openclaw/openclaw/blob/v2026.9.7/docs/gateway/configuration-reference.md): JSON5 configuration and actual schema/parser validation commands.
- [v2026.9.7 custom providers](https://github.com/openclaw/openclaw/blob/v2026.9.7/docs/gateway/config-tools/custom-providers.md): OpenAI Completions/Responses protocol choices; no verified non-streaming mode. `supportsUsageInStreaming` concerns streaming usage metadata and does not establish a non-streaming fallback.
- [v2026.9.7 MCP/extensions](https://github.com/openclaw/openclaw/blob/v2026.9.7/docs/gateway/config-extensions.md): native `mcp.servers` stdio declarations and session discovery. This is documentation evidence, not proof of discovery in our image.
- [v2026.9.7 exec tool](https://github.com/openclaw/openclaw/blob/v2026.9.7/docs/tools/exec.md): exec targets gateway/sandbox/node and can mutate permitted paths; none constitutes the Tentaflake disposable-worker queue.
- [v2026.9.7 stream selection](https://github.com/openclaw/openclaw/blob/v2026.9.7/src/agents/embedded-agent-runner/stream-resolution.ts) and [transport factory](https://github.com/openclaw/openclaw/blob/v2026.9.7/packages/ai/src/transports/provider-transport-stream.ts): the embedded runner selects built-in OpenAI stream functions. Provider overrides are extension code, not a verified non-streaming setting.
- [v2026.9.7 Completions request builder](https://github.com/openclaw/openclaw/blob/v2026.9.7/packages/ai/src/transports/openai-completions-params.ts#L309): both direct and managed modes use a request with `stream: true`; `supportsUsageInStreaming` only controls `stream_options.include_usage`.
- [v2026.9.7 Responses provider](https://github.com/openclaw/openclaw/blob/v2026.9.7/packages/ai/src/providers/openai-responses.ts#L143) and [stream lifecycle](https://github.com/openclaw/openclaw/blob/v2026.9.7/packages/ai/src/providers/openai-responses-shared.ts#L172): the direct provider constructs `stream: true` and consumes a Responses event stream.

No runtime artifact was downloaded, built or pulled. Repository/release HTML exceeded
the bounded extraction limit; the older embedded-runner source path returned
404. The versioned directory index identified the current source paths above.
The inspected built-in OpenAI request builders require streaming. The broker
now offers opt-in bounded SSE for Chat Completions/Responses; Anthropic Messages
remain rejected. See [streaming policy](https://docs.tentaflake.dev/configuration/brokered-egress/#streaming-model-responses). The scaffold therefore declares streaming
as required for those built-in transports and refuses activation. This is a
source-level incompatibility finding; no actual pinned executable was exercised.
It does not rule out a separately reviewed custom provider or an upstream change.
Setting `stream: false` in a payload hook alone does not establish compatibility
with an event-stream consumer. Broker streaming does not establish acceptance with that pinned executable.

The existing broker credential-substitution regression also sends source-derived
streaming request shapes to both broker routes. It requires HTTP 400 with
`streaming is disabled`, unchanged budget state, and an audit entry without
prompts or credential values. Its upstream fixture is already closed, so an
accidental dispatch cannot satisfy the expected result. These deterministic
probes preserve the default denial. Separate live loopback-provider tests cover
the opt-in SSE transport, not vendor compatibility.

## OpenClaw acceptance and next design

Before enabling OpenClaw:

1. Review the exact OCI release digest and provenance, or separately review a
   local image with exact source/dependency hashes. Retain secure image checks.
2. Parse generated read-only configuration with the actual pinned executable;
   exercise startup, intentional stop, crash recovery and state after restart.
3. Capture a real bounded request against a deterministic broker fixture. Prove
   transport compatibility using the opt-in broker SSE path. Repeat bounded
   partial-output, cancellation, timeout/audit failure and conservative budget
   checks with that executable.
4. With only the projected stdio relay, prove actual tool discovery and one
   Research call. Verify native web tools, remote MCP, provider-hosted tools,
   channels, remote plugins and background tools cannot create another path.
5. Implement a reviewed upstream execution extension mapping commands to
   worker-queue v1 requests, await exact-ID results and propagate rejection,
   timeout and cancellation. Disable built-in exec/process alternatives and
   prove a build runs in the disposable no-egress capsule.
6. Wire the exact broker/worker/Btrfs quota/Research declarations, matching paths
   and UID/GID. Operator-provisioned credentials and VPN readiness stay on the
   host. Repeat negative policy checks after overrides and provenance gates.

The broker streaming prerequisite is implemented separately from the adapter
refactor, with bounded SSE events/output, first/idle/total deadlines, cancellation,
no replay, pre-dispatch audit/budget admission and conservative reservation.
Actual OpenClaw transport acceptance remains unverified. Keep provider credentials
host-only and retain model/tool allowlists; repeat partial-output, usage,
cancellation and audit-failure checks before any adapter activation.
Heavy VM/image/system builds and paid-provider probes need separate workload
authorization. Synthetic policy fixtures and package builds never establish
vendor operational compatibility.

## Implementation and evidence status

The adapter foundation, compatible wrappers, explicit inventory, versioned
JSON and opt-in broker SSE are implemented. Focused policy and protocol
fixtures cover shared admission, default streaming denial and bounded streaming
failure paths. See [brokered egress](https://docs.tentaflake.dev/configuration/brokered-egress/) for transport details
and [build boundaries](https://docs.tentaflake.dev/reference/builds/) for the verification entrypoints.

OpenClaw has no selected OCI artifact. Actual native configuration parsing,
model/Research calls, worker mediation, lifecycle and persistence acceptance
remain pending. Hermes/ZeroClaw retain their existing integrations; synthetic
fixtures do not establish actual vendor workloads. No host activation is
implied by source checks.

---

# Operator shell and Rust CLI

> Use the operator shell and tentaflake CLI.

Canonical page: https://docs.tentaflake.dev/operations/cli-shell/

> Source: [`docs/06-shell.md`](https://github.com/timfewi/tentaflake/blob/e21be475a3afc6e95c5087003edd0b7f780e4df4/docs/06-shell.md) at `e21be475a3af` · docs version **0.4.37**

`modules/shell.nix` installs a small Rust CLI and optional operator comforts.
It does not contain the CLI implementation; the workspace source lives in
`crates/tentaflake-cli`.

## Generated inputs

Nix writes two non-secret files:

- `/etc/tentaflake/cli.conf` selects the backend, host, flake directory, and
  agent inventory path.
- `/etc/tentaflake/agents.tsv` uses explicit `tentaflake.agentInstances`
  metadata for adapters and retains a fallback for unmanaged OCI containers.
  Its five fields are adapter, instance name, container name, systemd unit,
  and state directory. Stopped scaffolds can appear without an OCI container.

Do not hand-edit them. Change the Nix configuration and rebuild explicitly.

## Commands

```text
tentaflake help
tentaflake runtimes [--json]
tentaflake agent template <preset> <name>
tentaflake agent validate|plan|import <file> [--json] [--hide]
tentaflake status [--json] [--hide]
tentaflake health [--json] [--hide]
tentaflake doctor [--json] [--hide]
tentaflake doctor --security [--json] [--hide]
tentaflake stats
tentaflake logs <agent>
tentaflake restart <agent>
tentaflake start <agent>
tentaflake stop <agent>
tentaflake shell <agent>
tentaflake exec <agent> -- <command>
tentaflake ps
tentaflake backup <agent>
tentaflake rebuild
tentaflake update
```

`status` is the default command. `--json` provides machine-readable status;
`--hide` redacts the host and agent names. `tentaflake-status` invokes the same
status renderer and is used by the login banner.

`runtimes` lists the pinned preset catalog and its support/evidence levels;
`--json` emits the complete schema-v1 catalog. It performs no host reads or
image downloads. See [generic workloads](https://docs.tentaflake.dev/configuration/agent-adapters/#generic-isolated-workload).

`help`, `--help`, and `-h` display usage even when the generated host
configuration or inventory is missing or invalid. `agent template` is also
configuration-free; other management commands require the generated inputs.
See [installed-host onboarding](https://docs.tentaflake.dev/configuration/agents/#add-an-agent-on-an-installed-host)
for generic image/command templates, additive imports and explicit host updates.

`health` adds host load to the same diagnosis used by `doctor`. Both check
failed systemd units, root-disk usage of at least 90%, and failed or unknown
agent states. They return `0` for a healthy diagnosis, `1` for detected
problems, and `2` when systemd or disk evidence cannot be obtained. An
intentionally stopped agent is valid. JSON reports include `failed_agents`
and `unknown_agents`; `--hide` redacts names in both text and JSON. A
successful diagnosis is host evidence only; it does not prove external
provider access.

`doctor --security` reads the Nix-generated desired-state security manifest.
It emits stable `TFSEC-*` findings with severity and remediation; critical or
high findings return non-zero. JSON output is CI-friendly. The desired-state check
covers the declared profile, capsule flags, mounts, images, credential files,
OpenSSH, AppArmor, Docker-group membership, backup declaration, workspace
quota, missing broker, and the declared private management/SSH capabilities.
Manifest v2 carries a separate typed management record; unversioned legacy or
incomplete records report `TFSEC-038` unknown management evidence. Configured
capabilities do not prove enrollment or effective remote grants. It also
inspects root-disk pressure, the last
successful Restic timestamp, and live Tailscale Serve/Funnel JSON. If a live
command is unavailable or blocked, the result is an explicit warning rather
than green. A non-interactive backend inspect also compares a running Docker or
Podman container's privilege, user, root filesystem, runtime, network,
capabilities, AppArmor, security options, resource limits, and mounts with the
exact manifest values, including every bind mount, hardened tmpfs size, and
CPU/memory/PID/ulimit setting. Brokered networks are separately inspected for
the exact internal bridge. Each OCI inspection captures both output streams
and has a five-second deadline and an 8 MiB combined stdout/stderr limit;
raw Inspect environment data never reaches
the report. A stopped/missing container, incomplete backend schema, or
unavailable approved sudo path is reported as unknown. Configured broker
`/healthz` endpoints are probed directly: connection failures stay unknown,
while an explicit non-ready response is high severity because broker health
checks credentials and audit persistence. Complete controller-tool audit
coverage and cross-agent network denial remain separate runtime evidence.

`backup` creates a mode-`0600`, caller-owned archive of one state directory in
the current directory. It does not back up secret-provider identities.

`rebuild` and the apply step offered by `update` activate NixOS. Invoke them
only when the target is resolved and activation is intended.

The deprecated `hermes` shim remains for command-name compatibility. The old
`top`, `console`, and interactive agent wizard commands were removed with the
Auditd, SQLite, and custom web-console stack.

## Shell options

All options live below `tentaflake.shell`:

| Option | Default | Purpose |
|---|---:|---|
| `enable` | `true` | Shell module |
| `tentaflakeCli.enable` | `true` | Rust CLI |
| `motd.enable` | `true` | Login status |
| `tools.enable` | `true` | Curated terminal tools |
| `starship.enable` | `true` | Prompt |
| `zsh.enable` | `false` | Zsh and completion |
| `zoxide.enable` | `true` | Directory jumping |
| `lazygit.enable` | `false` | Git TUI |
| `tmux.enable` | `false` | Terminal multiplexer |

## Physical console

`tentaflake.modernConsole.enable` selects kmscon for Unicode-capable output.
Set it to `false` when hardware cannot use kmscon; `consoleFont` then controls
the legacy VT font. The installer disables font replacement to avoid display
reconfiguration on sensitive hardware.

---

# Agent management

> Manage agent lifecycle and routine operator workflows.

Canonical page: https://docs.tentaflake.dev/operations/agent-management/

> Source: [`docs/02-agent-tips.md`](https://github.com/timfewi/tentaflake/blob/e21be475a3afc6e95c5087003edd0b7f780e4df4/docs/02-agent-tips.md) at `e21be475a3af` · docs version **0.4.37**

Manage agents through the operator CLI and declarative configuration. Installed
hosts default to `balanced`; direct provider files, host networking, published
ports, dashboards, and mutable images require an explicit `dev` exception.
See [security profiles](https://docs.tentaflake.dev/security/profiles/) before migrating an old setup.

## Identify the agent

```text
tentaflake status
tentaflake ps
tentaflake doctor
tentaflake doctor --security
```

Use the exact identifier shown by the inventory. Adapter metadata supplies the
runtime, instance, container, unit, and state path; unmanaged OCI containers
retain a fallback. The [adapter guide](https://docs.tentaflake.dev/configuration/agent-adapters/) distinguishes existing
Hermes/ZeroClaw integrations from OpenClaw's stopped scaffold. A scaffold can
appear in the inventory without a container and refuses an explicit start.

## State and configuration

| Runtime | Default host state | Managed workspace | Container identity |
|---|---|---|---|
| Hermes | `/var/lib/hermes-<name>` | `workspace/` | UID/GID 10000 |
| ZeroClaw | `/var/lib/zeroclaw-<name>` | `data/` | UID/GID 65534 |

Hermes mounts its state at the same path inside the capsule and sets
`HERMES_HOME` there. ZeroClaw mounts its state at `/zeroclaw-data`; its generated
TOML configuration lives at `/zeroclaw-data/.zeroclaw/config.toml`. State paths
and permitted ownership overrides come from the selected builder. Confirm them
before copying data; do not infer a path from an instance name alone.

Hermes generates read-only configuration when effective `settings` is non-empty;
Research adds settings even when callers supply none. ZeroClaw always generates
and mounts its configuration read-only. Edit the declarative input and rebuild
deliberately. Interactive configuration writes can fail against that mount;
they are not a persistent configuration workflow. Seed files are copied without
overwriting existing
state, so changing `seedDir` is not an automatic update of existing skills or
personality files.

[Workspace quotas](https://docs.tentaflake.dev/configuration/workspace-quota/) cap the declared workspace, not every
session, cache, log, or broker directory. Monitor host free space separately.
New managed images use Btrfs and require at least 128 MiB; existing ext4 images
need the documented explicit backup/restore migration.

## Start, stop, restart and logs

```text
tentaflake logs <agent>
tentaflake stop <agent>
tentaflake start <agent>
tentaflake restart <agent>
tentaflake shell <agent>
tentaflake exec <agent> -- <command>
```

Resolve the target before a lifecycle mutation. `stop` stops the controller and
its loaded LLM/fetch brokers together. A direct container-runtime restart skips
that coordinated systemd operation; use the CLI. See [operations and recovery](https://docs.tentaflake.dev/operations/recovery/)
for crash backoff, broker readiness, and the combined kill switch.

For host-side unit evidence, use the exact unit from the inventory:

```bash
sudo systemctl status docker-hermes-coding.service
sudo journalctl -u docker-hermes-coding.service --since '1 hour ago'
```

Podman units use the `podman-` prefix. NixOS manages these containers as root;
selecting Podman does not make this a rootless deployment. The balanced
operator is not placed in the root-equivalent Docker group.

Do not print raw OCI environment data: compatibility containers can contain
real credentials, and balanced capsules contain virtual keys. The security
doctor provides bounded, redacted runtime evidence. `--hide` removes host and
agent names from diagnostic text and JSON; review other operational logs before
sharing them. Optional [observability](https://docs.tentaflake.dev/operations/observability/) adds local metrics
and journal queries.

## Add or remove an agent

Use the module list in `my-agents.nix` or the versioned generic JSON shape in
[declarative configuration](https://docs.tentaflake.dev/configuration/agents/). For example:

```nix
{ mkHermesAgent, ... }:
[
  (mkHermesAgent {
    name = "assistant";
    autoStart = false;
  })
]
```

This creates a stopped capsule. Under balanced, automatic start requires its
exact LLM broker, disposable worker, workspace quota, and research relay. Model
access follows [brokered egress](https://docs.tentaflake.dev/configuration/brokered-egress/); web access follows
[secure research](https://docs.tentaflake.dev/configuration/research/). The legacy fetch broker must be disabled for
a research-enabled agent. [The secure adapter example](https://github.com/timfewi/tentaflake/blob/e21be475a3afc6e95c5087003edd0b7f780e4df4/examples/adapter-secure.nix)
combines these declarations while retaining the documented runtime evidence gaps.

First evaluate and build the intended host, review assertions and the diff,
then activate in a planned window. `autoStart = false` does not prevent an
activation from creating declared quota filesystems or worker services.

Remove the agent's builder or JSON entry and its matching broker, research,
worker, quota, and provenance declarations. Review backup paths before
activation. State, backing images, and external secret files are not an
automatic deletion target; retain or archive them deliberately. Do not manually
remove Nix-managed users or groups as part of routine agent removal.

## Rotate credentials

For balanced agents, rotate the broker's provider file through the host's
runtime secret mechanism and restart only the exact LLM broker. Verify its
`/healthz` readiness and audit persistence without printing credentials. The
controller never receives the real provider value; the virtual key survives
broker unit restarts and changes on host reboot.

Research credentials belong to the separate Research service. Git auto-push,
backup, and observability credentials also stay host-side. Direct environment
files in an agent are a dev-only compatibility path; consult
[Agenix](https://docs.tentaflake.dev/configuration/secrets/). Runtime files below `/run` must be provisioned
again at boot by the deployment's secret mechanism.

## Update images

Default controller image digests live in `adapters/catalog.json`;
`lib/constants.nix` retains the existing exported aliases. They are independent
of `flake.lock`:

```bash
./scripts/update-agent-images.sh
```

The script reports upstream digests; it does not rewrite pins. Review the image
source and provenance, update the exact digest and regenerate catalog docs with
`python3 scripts/runtime-catalog-docs.py`, run the appropriate checks, and
build before activation. Use `registry/repository@sha256:<digest>`; references
combining a tag and digest are rejected for backend compatibility. The
`allowMutableImage` escape hatch is dev-only.

A successful source evaluation or fixture test does not prove real vendor
startup, model compatibility, MCP discovery, or routing through the worker.
Validate those against the exact deployed image.

## Tune resources and execution

Balanced resource limits are enforced after caller configuration:

```nix
tentaflake.security.resources = {
  memory = "4g";
  memorySwap = "4g";
  cpus = "0.5";
  nofile = 4096;
};
```

Both current runtime builders default to `pidsLimit = 512`. Secure controllers
require a positive ceiling; `null` is dev-only. `extraContainerConfig` cannot
weaken the final limits or restore capabilities. See
[security profiles](https://docs.tentaflake.dev/security/profiles/) for tmpfs and other bounds.

Use the [offline disposable worker](https://docs.tentaflake.dev/configuration/disposable-worker/) for untrusted code.
Declaring a worker does not prove that every native runtime tool submits work to
its queue. Approval of a worker job never grants network, credentials, or a
writable host bind.

## Back up and recover

Use the opt-in encrypted Restic policy in [operations and recovery](https://docs.tentaflake.dev/operations/recovery/)
for unattended backups. Selected quota mounts are added as separate sources
and must be mounted before backup. Keep configuration, runtime credentials,
recipient identities, and state recovery plans distinct.

`tentaflake backup <agent>` instead creates a mode-0600, caller-owned state
archive in the current directory. It is not encrypted, does not back up secret
identities, and does not replace a real restore drill. Plan space and access
before creating an archive. Restore with the controller stopped, verify numeric
ownership and quota mounts, then test application behavior.

---

# Operations and recovery

> Operate, troubleshoot, and recover a tentaflake host.

Canonical page: https://docs.tentaflake.dev/operations/recovery/

> Source: [`docs/07-operations.md`](https://github.com/timfewi/tentaflake/blob/e21be475a3afc6e95c5087003edd0b7f780e4df4/docs/07-operations.md) at `e21be475a3af` · docs version **0.4.37**

## Evidence boundaries

Nix evaluation proves option types and assertions. A build proves the closure.
Activation changes the host, and live checks prove only the activated runtime.
Do not treat one boundary as proof of another. Tentaflake never activates a
configuration merely because a check or build succeeded.

## Daily commands

```text
tentaflake status
tentaflake health
tentaflake doctor
tentaflake doctor --security
tentaflake logs <agent>
tentaflake stop <agent>
tentaflake start <agent>
tentaflake restart <agent>
```

Under `balanced`, Docker daemon access is root-equivalent and is not granted
through the `docker` group. Container commands use the CLI's sudo-backed path.
Podman containers are still root-managed by the NixOS OCI module unless a
separate rootless design is implemented; selecting Podman alone is not proof
of rootless operation.

## Safe update flow

1. Review input and image changes. A digest pin is reproducible identity, not
   evidence that the image is trustworthy. If the upstream signs releases,
   update and review its exact `tentaflake.imageProvenance` identity/key policy
   too; a mismatched signature gate blocks the controller before start.
2. Run formatting, lint, unit, evaluation, and VM checks.
3. Build the selected system without activation.
4. Review the exact target and build result.
5. Activate only in an approved maintenance window.
6. Run live status, security posture, service, and application health checks.

The previous NixOS generation remains the rollback path. Select it in the boot
menu or run the explicit rollback command from an operator session after
resolving the target. Source changes and successful builds do not authorize
`switch` or rollback.

## Git auto-push

The agent commits inside its workspace; a host unit performs the push with a
credential the balanced agent never receives:

```nix
gitAutoPush = {
  tokenEnvFile = "/run/agenix/github-coding";
  allowedRemotes = [
    "https://github.com/example/coding.git"
  ];
  allowedBranches = [ "main" ];
  interval = "2min";
};
```

The helper requires an exact canonical HTTPS GitHub repository and exact
branch. It rejects altered remotes and emits `ALLOW`, `DENY`, or `ERROR` to
journald. Use a fine-grained repository token or GitHub App credential with
only the required contents-write permission. The token file stays on the host.

## Legacy egress option

`tentaflake.networking.egress` is renamed to
`tentaflake.networking.legacyPortEgress` and is allowed only in `dev`:

```nix
tentaflake.networking.legacyPortEgress = {
  enable = true;
  allowedTCPPorts = [ 443 ];
};
```

This is a host `OUTPUT` destination-port filter. It does not constrain Docker
bridge `FORWARD`, identify destination hosts, control DNS, or constitute a
secure allowlist. Balanced agents use `network=none` or the dedicated broker
path documented in [brokered egress](https://docs.tentaflake.dev/configuration/brokered-egress/). That path adds an
internal network, subnet-scoped INPUT/FORWARD policy, disabled container DNS,
and host-side destination resolution.

## Crash and reboot behavior

The upstream NixOS OCI module generates systemd units with
`Restart=on-failure`. Secure controllers and brokers share a recovery policy:
the restart delay increases exponentially from 10 seconds to one minute over
five steps, then stays at one minute. The start-limit window is disabled so a
prolonged transient failure does not permanently disable a 24/7 service.
Each attempt still runs the configured credential, mount, provenance, and
isolation gates. Persistent errors remain failed attempts; alert on restart
flapping and investigate their cause. An explicit systemd stop prevents
automatic restarts; declarative `autoStart` controls boot startup.

Brokers expose credential/policy-aware `/healthz`; this is not a provider
end-to-end probe. Backoff does not detect a hung or unhealthy process that
keeps running, or restart a controller after a failed dependency start job.
Use live health checks and the optional observability profile for those cases.

Hosts with a known, tested watchdog device can also opt in to PID 1 hardware
watchdog supervision:

```nix
tentaflake.hardening.watchdog = {
  enable = true;
  device = "/dev/watchdog";
  runtimeSec = "30s";
  rebootSec = "10min";
};
```

This is disabled by default because the template cannot know whether a target
has a functional watchdog or whether its firmware reset behavior is safe.
Validate the exact device and a controlled reset/recovery drill on the target
before relying on it.

## Backup and restore

Back up declarative configuration separately from mutable state. Per-agent
state defaults to `/var/lib/<runtime>-<name>`. Provider/Git/backup credentials
must remain outside the agent and outside backup logs.

The core module provides an opt-in Restic policy while keeping repository and
password values in runtime files owned by the deployment fork:

```nix
tentaflake.backup = {
  enable = true;
  paths = [
    "/var/lib/hermes-coding"
    "/var/lib/tentaflake-broker-llm-hermes-coding"
  ];
  repositoryFile =
    "/run/agenix/restic-repository";
  passwordFile =
    "/run/agenix/restic-password";
  pruneOpts = [
    "--keep-daily 7"
    "--keep-weekly 4"
  ];
};
```

Enabled managed quota state/workspace mounts below a selected path are automatically
added as separate Restic sources. This preserves `--one-file-system` without
silently skipping a mounted state or workspace filesystem. Only explicitly selected trees are
covered; unrelated agents and disabled quotas are excluded. The backup unit
requires the selected filesystems and asserts that each included quota
filesystem is mounted before it runs. A failed mount prevents a new backup and
success timestamp. Other nested filesystems require their own explicit entry
in `paths`.

The job encrypts through Restic, prunes after backup, and runs an integrity
check. On success, a separate hardened oneshot updates
`/var/lib/tentaflake-backup/last-success` in a systemd-managed state directory;
the security doctor compares that timestamp with `lastSuccessMaxAgeHours` (36
by default). Its default timer is persistent, scheduled around 03:00 with
randomized delay. Alert on failed/stale `restic-backups-tentaflake.service`
runs.
For a restore drill: install a fresh test host, keep the agent stopped, restore
its state, verify ownership and file permissions, start the exact unit, and run
application-level checks. The VM test restores mounted state and a mounted
quota workspace into a fresh directory, rejects an unavailable mount, and
verifies recovery afterward. Production backend credentials, retention,
capacity, and a real fresh-host drill remain operator evidence. Backup freshness and restore
readiness beyond freshness still requires an actual restore drill.

## Incident response and kill switch

1. Resolve and stop the exact agent with `tentaflake stop <agent>`.
2. Revoke its Git/provider/broker credentials outside the agent.
3. Preserve journald, optional Loki/Falco evidence, and a read-only state copy.
4. Inspect declared image, mounts, policy, workspace changes, and host events.
5. Build a repaired configuration and review it before activation.
6. Restore only reviewed state and rotate every possibly exposed credential.

`tentaflake stop <agent>` asks systemd to stop the container and every loaded
LLM/fetch broker for that exact container in one transaction. The internal
network remains declared but has no allowed service endpoint, so the virtual
key is inert. Provider/Git credential revocation remains an external operator
step.

Pending worker actions are separate private state. Inspect, approve, or deny
them with the commands in
[the disposable-worker guide](https://docs.tentaflake.dev/configuration/disposable-worker/). Stopping an agent does
not authorize a pending action and an approval never grants network or secret
access to the worker.

## Disk and logs

The secure capsule limits RAM, swap, CPU, PIDs, ulimits, and tmpfs size. The
fixed-size workspace filesystem provides a hard per-controller code workspace
ceiling; optional private state uses a separate bound. See
[workspace/state quotas](https://docs.tentaflake.dev/configuration/workspace-quota/). Other host state and aggregate
sparse backing images still require host free-space monitoring.
journald rotation follows the host configuration; the
optional Alloy/Loki profile is loopback-only and does not itself define a
retention policy suitable for every deployment. It does provide baseline
Prometheus rules for disk pressure, restart flapping, broker denials, unusual
fetch-denial bursts, and near-exhausted request/token/cost budgets. Alert
delivery remains an explicit deployment-fork integration.

## Continuous-runtime follow-ups (2026-10-03)

The source review for a modular 24/7 host identified these remaining issue
candidates. Restart backoff, broker health, quota-aware backups and optional
metrics already exist; the gaps below should not be described as missing all
supervision or backup support.

| Priority / proposed issue | Current limitation | Acceptance criteria |
| --- | --- | --- |
| [P1: Application readiness and bounded hang recovery](https://github.com/timfewi/tentaflake/issues/112) | `on-failure` recovers crashes, but a running hung agent remains running. Broker `/healthz` is not an end-to-end agent or Research probe. | Add adapter-defined, non-billable readiness/liveness checks with bounded timeouts, cooldown and explicit restart authority. Exercise hung processes, dependency recovery and operator stop without replaying operations. |
| [P1: Actual vendor-agent acceptance](https://github.com/timfewi/tentaflake/issues/111) | The Research integration VM uses a synthetic probe, not the pinned Hermes/ZeroClaw applications. Worker automatic routing and state survival also need real runtime evidence. | Run each reviewed image through model, Research discovery, isolated execution, restart/reboot and persistence scenarios with synthetic upstreams. Keep unsupported adapters stopped. See `16-research.md` and `agent-adapters.md`. |
| [P1: Worker queue crash recovery and capacity](https://github.com/timfewi/tentaflake/issues/113) | The selective archive review already identifies atomic claims, abandoned-work recovery and bounded pending/inbox capacity as follow-ups. | Implement queue lifecycle and negative crash/overload tests together, define migration, and retain quota-aware backup/restore. See `archive-comparison.md`. |
| [P2: Research/management metrics and tested alert delivery](https://github.com/timfewi/tentaflake/issues/114) | Optional observability covers host and broker metrics; notification destinations are deployment-owned. No source proof of end-to-end paging or shared Research/VPN readiness metrics was found. | Export categorical readiness/staleness/renewal/restart and usage metrics without queries, credentials or source text. Test a configured generic notification route in a deployment fixture. |

Private management and Headscale issue candidates are recorded in
[the management review](https://docs.tentaflake.dev/operations/tailscale/#modular-management-review-2026-10-03).
The linked tracking issues record scope, acceptance criteria and dependencies.
The reviewed Research input update is tracked in
[issue #110](https://github.com/timfewi/tentaflake/issues/110).

---

# Private management contract and Tailscale policy

> Configure private management and SSH policy with the reviewed Tailscale backend.

Canonical page: https://docs.tentaflake.dev/operations/tailscale/

> Source: [`docs/11-tailscale-management.md`](https://github.com/timfewi/tentaflake/blob/e21be475a3afc6e95c5087003edd0b7f780e4df4/docs/11-tailscale-management.md) at `e21be475a3af` · docs version **0.4.37**

## Host contract and migration

Configure private operator connectivity and SSH authorization separately from
agent capabilities and Research Internet egress:

```nix
tentaflake.management = {
  enable = true;
  transport = "tailscale";
  ssh.policy = "tailnet-policy";
};
```

These are the existing defaults. `tailscale` is the only currently reviewed
transport, using the Tailscale client; the contract does not require a hosted
control-plane account or automatically configure Headscale. Other transport
names fail validation until their implementation and acceptance are added.
`tailnet-policy` selects Tailscale SSH and requires the operator to maintain
restrictive remote grants and SSH rules. `ssh.policy = "disabled"` is available
only for development configurations; it cannot satisfy `balanced`.

The existing `tentaflake.tailscale.enable` option is an alias for
`tentaflake.management.enable`. Existing consumers retain their behavior; migrate
declarations to the new name when convenient. The two names are the same option,
not independent switches, so do not give them conflicting values. Disabling
management through either name still fails `balanced`; it does not waive the
private management requirement. The public OpenSSH module remains forbidden.

The internal, read-only `management.capabilities.privateConnectivity` is derived
from the enabled contract and actual configured Tailscale service.
`management.capabilities.sshAuthorization` is derived from that connectivity,
the selected policy and the final SSH flags after overrides. It becomes
`disabled` if SSH is not enabled in `extraSetFlags`, if a disabling SSH argument
appears in the set/up flags, or if an argument separator prevents reliable
interpretation. Balanced requires both configured capabilities. They cannot be
set by an agent or replaced with operator-written readiness booleans.

These capabilities describe configuration. They do not establish enrollment,
live connectivity, operator-tested access, remotely effective grants or a usable
recovery path. Host configuration remains operator-owned; importing agent data
does not choose transports, control servers, firewall policy or credentials.
Keep console access and a separately tested operator session before any
explicitly authorized management change.

The generated `/etc/tentaflake/security.tsv` now starts with `manifest` and
version `2`, separated by a tab. Its `management` record carries the transport,
enabled state, configured private connectivity and derived SSH authorization.
The host record no longer carries the Tailscale-specific enable boolean. The CLI
reads legacy unversioned manifests but treats their management contract as
unknown; a missing version-2 management record is also unknown. Under balanced,
`TFSEC-038` reports that gap, `TFSEC-019` reports disabled/missing private
connectivity, and `TFSEC-039` reports disabled configured SSH authorization.
Unsupported versions/transports/policies and duplicate records fail parsing.
Regenerate the manifest through an authorized host update; never hand-edit it
to manufacture evidence.

## Remote authorization

Tailscale connectivity is private transport, not authorization by itself. A
new tailnet's permissive default policy must be replaced before treating it as
the only management path.

`tailscale-policy.example.json` is syntactically valid JSON and separates:

- a network grant from one operator group to TCP 22 on `tag:agent-host`; and
- a Tailscale SSH rule with `action: check`, including privileged root access.

Replace `operator@example.invalid` with one controlled identity and validate
the policy in the Tailscale admin console before saving it. Add other ports
only for an explicitly authenticated operator service. Do not use Funnel for
a secure host, and do not automatically publish agent APIs with Tailscale
Serve. Tailnet-visible is still reachable by every identity the policy allows.

Protect the identity provider with phishing-resistant hardware-backed MFA.
Enable Tailnet Lock only after provisioning at least two independent signing
nodes and testing the recovery procedure. Keep auth keys and lock signing keys
out of Nix expressions and the Nix store.

The template cannot confirm the remote admin-console policy from local Nix
evaluation. For an enabled Tailscale declaration or an unknown legacy/incomplete
management contract, the security doctor checks local `tailscale serve status --json`
for active Serve/Funnel state and reports a blocked local API as unknown, but
the operator must still inspect and validate the remotely active policy;
neither restrictive grants nor SSH rules are inferred from an enabled management
contract or its configured capabilities.

## Enrollment and local preferences

The template applies `--ssh` and the host name through
`services.tailscale.extraSetFlags`. NixOS creates `tailscaled-set` even without
`authKeyFile`. `--advertise-tags` is accepted only by `tailscale up`, so
`extraUpFlags` retains the tag and these preferences for automatic auth-key
enrollment. NixOS does not apply `extraUpFlags` without `authKeyFile`.

For a new manually enrolled node, supply the tag explicitly using the configured
host name:

```sh
sudo tailscale up --advertise-tags=tag:agent-host --ssh --hostname='<host-name>'
```

On an already configured node, preserve all existing non-default preferences;
follow the CLI's required flag list rather than resetting them. Local settings
do not enroll a node or validate the remote policy. Verify the assigned tag,
live preferences and access from a separate authorized session.

## Modular management review (2026-10-03)

Requirement: keep Tentaflake a provider-independent host for agents running
continuously. These are source-review findings and issue candidates, not
implemented adapters or verified production networking.

Management connectivity and Research Internet egress are separate capabilities.
Tailscale/Headscale controls node membership and operator reachability. Research
uses the explicitly configured `services.secureResearch.vpnInterface`, resolver
policy, UID firewall and root-owned readiness leases; it neither creates the VPN
nor chooses the control server. A management tailnet does not automatically
provide an Internet exit node.

| Priority / proposed issue | Current evidence | Acceptance criteria |
| --- | --- | --- |
| [P1: Generic private management contract](https://github.com/timfewi/tentaflake/issues/106) | The typed management/SSH contract, legacy alias and version-2 manifest describe configured capabilities. Tailscale is the only reviewed transport; enrollment and remote policy remain unverified. | Preserve current consumers and public-access refusal. Authorized runtime fixtures still need to prove public-access denial and operator access before acceptance; source evaluation alone does not close this issue. |
| [P1: Optional Headscale control-server configuration](https://github.com/timfewi/tentaflake/issues/107) | The management contract selects Tailscale SSH, but login-server, enrollment-credential and configurable-tag support remain separate. Consumers can override upstream NixOS options; Headscale enrollment is not a template-tested workflow. | Select a control server explicitly while retaining the Tailscale client. Keep runtime credentials out of the store; test enrollment, logout, restart and unreachable server. Tags must be configurable and constrained by remote policy. No automatic control-plane hosting. |
| [P1: SSH policy compatible with the chosen control plane](https://github.com/timfewi/tentaflake/issues/108) | Balanced forbids every OpenSSH service and assumes Tailscale SSH. The example's `check` action and Tailnet Lock guidance target the hosted Tailscale policy. | Verify the chosen Headscale version's SSH capabilities, or support key-only OpenSSH restricted to the private interface. Test public denial and operator access; do not assume hosted policy features work on another control plane. |
| [P1: Management readiness and recovery evidence](https://github.com/timfewi/tentaflake/issues/109) | The doctor checks the typed configured contract and Serve/Funnel state; it does not verify enrollment, a usable operator path or the remote grants. | Report declared, enrolled, reachable and remotely unverified states separately with bounded probes. Test offline control plane, reboot and credential expiry; retain console/recovery access. Never mark remote authorization verified from a local flag. |
| [P1: Provider-independent Research VPN adapter acceptance](https://github.com/timfewi/tentaflake-research/issues/4) | Research already accepts an external observation adapter. Its optional reference observer checks interface-up, an IPv4 route across routing tables and a root-owned firewall marker; the marker is not verification of live rules or tunnel identity. | Document a reusable adapter contract and test selected tunnel/exit-node configurations for IPv4, IPv6, DNS, policy routing, tunnel loss and stale observations. Keep the independent UID firewall fail-closed. The observer's `region` is an operator assertion. |

The manual-enrollment preference bug above is addressed in
[PR #105](https://github.com/timfewi/tentaflake/pull/105). Its regression covers
manual and auth-key configurations and parses the actual flags with the pinned
Tailscale CLI. The adapter items remain follow-up work.
No enrollment, ACL publication, VPN switching or host activation is performed
by this review.

Verification checkpoint (2026-10-04): the pinned Tailscale 1.98.8 parser rejects
the former `set --advertise-tags` invocation before contacting a daemon. The
module-evaluation regression reproduces that failure with the generated flags;
`--help` validates parsing without enrollment. All fast-gate checks pass with
the corrected flags. Live enrollment, SSH access and
installed unit behavior require separate acceptance. No VM suite or host
activation was run; Research and lockfile pins remain unchanged.

The reviewed Research pin update is tracked separately in
[issue #110](https://github.com/timfewi/tentaflake/issues/110); it requires upstream review and
affected integration acceptance before deployment.

## Contract acceptance limits

Focused option and CLI regressions cover defaults, the old alias, service/SSH
overrides, unsupported selections, public OpenSSH refusal, versioned parsing and
legacy/missing management evidence. These are source-level checks. Issue
[#106](https://github.com/timfewi/tentaflake/issues/106) remains open for authorized
runtime fixtures proving public-access denial and existing-consumer operator
access. VM/runtime workloads were excluded from this lightweight-only run;
no enrollment, remote policy change, host activation or runtime acceptance is
claimed. Headscale enrollment and alternative private SSH remain separate
[#107](https://github.com/timfewi/tentaflake/issues/107) and
[#108](https://github.com/timfewi/tentaflake/issues/108) work.

---

# Observability and runtime detection

> Enable metrics, logs, dashboards, and runtime detection.

Canonical page: https://docs.tentaflake.dev/operations/observability/

> Source: [`docs/09-observability.md`](https://github.com/timfewi/tentaflake/blob/e21be475a3afc6e95c5087003edd0b7f780e4df4/docs/09-observability.md) at `e21be475a3af` · docs version **0.4.37**

Tentaflake exposes two independent profiles. Neither is imported by the core.

## Observability profile

`nixosModules.observability` enables:

- Prometheus for time-series storage;
- the Prometheus node exporter for host metrics;
- Loki for journal logs;
- Alloy to forward the local systemd journal to Loki;
- Grafana with provisioned Prometheus and Loki data sources.
- local textfile metrics for configured broker denial/budget state and bundled
  Prometheus alert rules.

Every listener binds to `127.0.0.1`; the profile opens no firewall port.
Grafana signup, update checks, analytics, Gravatar, and feedback links are
disabled.

Enable it in a consumer or fork:

```nix
imports = [
  inputs.tentaflake.nixosModules.observability
];

tentaflake.profiles.observability = {
  enable = true;
  retention = "14d";
  grafanaSecretKeyFile =
    "/run/agenix/grafana-secret-key";
  grafanaAdminPasswordFile =
    "/run/agenix/grafana-admin-password";
};
```

The two required files are loaded through systemd credentials. Their contents
must be stable across restarts and must not enter the Nix store. Evaluation
fails when either path is missing from the configuration.

Default loopback ports:

| Service | Port |
|---|---:|
| Grafana | 3000 |
| Prometheus | 9090 |
| Node exporter | 9100 |
| Loki | 3100 |
| Alloy diagnostics | 12346 |

Publishing Grafana through a tailnet or reverse proxy is a deployment decision.
Require authentication and TLS as appropriate; do not expose the other
listeners merely to make Grafana reachable.

The local rules cover root-disk pressure, repeated agent/service state changes,
recent broker policy denials, a burst of rejected fetches, and request, token,
or cost budgets above 90 percent. A hardened root oneshot reads only the exact
configured broker audit/budget paths once per minute and writes numeric metrics
for node exporter's textfile collector; it does not export prompt bodies or
credentials. Missing state files produce zero usage until a broker has written
state.

Prometheus evaluates these rules, but this generic profile deliberately does
not guess an email, pager, webhook, or external Alertmanager destination.
Configure and test notification delivery in the deployment fork. Until that
is done, a firing rule is visible in Prometheus/Grafana but is not proof that an
operator was paged. Thresholds are a baseline and should be tuned from observed
normal traffic.

## Falco profile

`nixosModules.falco` is separate because kernel event capture is a materially
different trust boundary. It runs Falco directly under systemd with the modern
eBPF engine and the capabilities `CAP_BPF`, `CAP_PERFMON`, `CAP_SYS_RESOURCE`,
and `CAP_SYS_PTRACE`.

The pinned nixpkgs revision does not provide Falco. Supply a reviewed and pinned
package whose layout includes `bin/falco` and the upstream configuration tree:

```nix
imports = [
  inputs.tentaflake.nixosModules.falco
];

tentaflake.profiles.falco = {
  enable = true;
  package = myPinnedFalcoPackage;
};
```

Modern eBPF generally requires a kernel at least 5.8 with BTF and the BPF ring
buffer. Check the exact Falco release requirements before deployment. The
profile does not download drivers or silently run a privileged container.

Falco defaults to the package's configuration. Use `configFile` and
`extraArgs` only with reviewed rules and outputs. Runtime alerts are written to
Falco's configured output; integrate them with journald or another explicitly
chosen destination.

Evaluation proves option wiring and systemd configuration. It does not prove
kernel compatibility, event capture, rule quality, or alert delivery. Verify
those on the target host after an explicitly authorized activation.

Upstream references:

- <https://falco.org/docs/concepts/event-sources/kernel/>
- <https://falco.org/docs/setup/download/>
- <https://grafana.com/docs/alloy/latest/reference/components/loki/loki.source.journal/>

---

# Security profiles and broker migration

> Choose and migrate between tentaflake security profiles.

Canonical page: https://docs.tentaflake.dev/security/profiles/

> Source: [`docs/10-security-profiles.md`](https://github.com/timfewi/tentaflake/blob/e21be475a3afc6e95c5087003edd0b7f780e4df4/docs/10-security-profiles.md) at `e21be475a3af` · docs version **0.4.37**

## Result

Installed hosts default to `balanced`. The common adapter builder and compatible
Hermes/ZeroClaw wrappers apply one policy to final merged OCI configurations.
OpenClaw remains a stopped scaffold without a container or operational support.
The policy rejects unsafe configurations during NixOS evaluation instead of
silently weakening them. `strict` also fails because no tested separate-kernel boundary
is implemented yet.

Phase A provides the containment baseline. A stopped balanced scaffold may
omit a broker and remains `network=none`. An automatically started balanced
controller must have an enabled broker policy, disposable worker, fixed
workspace quota, and research relay or evaluation fails. Brokered capsules join one dedicated
internal network. Neither path gives an agent a real provider credential or
general egress. [Web research](https://docs.tentaflake.dev/configuration/research/) uses only `secure-research-tool`;
model calls remain on the LLM broker. Retrieved content remains untrusted.
See [the worker guide](https://docs.tentaflake.dev/configuration/disposable-worker/).

## Profiles

```nix
tentaflake.security.profile = "balanced";
```

| Profile | Intended use | Network and credentials |
|---|---|---|
| `dev` | trusted local development and migration | legacy host networking, env files, and loopback ports may be configured |
| `balanced` | installed 24/7 baseline | stopped scaffolds may use `network=none`; auto-start requires one isolated LLM broker network, worker, quota, and research relay |
| `strict` | future high-risk workloads | evaluation error until a tested MicroVM/separate-kernel implementation exists |

There is no automatic fallback between profiles.

## Enforced balanced invariants

`lib/containerSecurity.nix` is applied after `extraContainerConfig`, so caller
overrides cannot bypass the final checks. It enforces:

- gVisor `runsc` selected explicitly for Docker or Podman;
- explicit non-root UID/GID and `privileged=false`;
- `cap-drop=ALL`, `no-new-privileges`, and read-only root;
- the runtime default seccomp policy with every unconfined override rejected;
  Docker additionally receives explicit `apparmor=docker-default`, while the
  Podman path requires host AppArmor and forbids an unconfined override;
- private tmpfs mounts for `/tmp`, `/run`, and `/var/tmp` with `nosuid`,
  `nodev`, `noexec`, and size ceilings;
- CPU, memory, swap, PID, open-file, and process limits; controller memory and
  total memory-plus-swap are bounded to 1 TiB each, tmpfs to 64 GiB each, and
  CPU quota to 1024 with at most five fractional digits;
- no published ports, caller networks, devices, caller-supplied host namespace
  overrides, runtime sockets, real credential files, or secret-like
  environment keys;
- the only network/file exceptions are the module-generated internal broker
  network, its random runtime-only virtual credential environment file, and
  the agent's exact read-only research socket/client closure;
- only declared state/workspace writable binds and generated Nix-store files
  read-only; all other host paths are rejected, including other agent state;
- digest-pinned images; the mutable-image escape hatch is dev-only;
- optional fail-closed Cosign verification before an OCI unit starts; publisher
  identity is a separate policy decision from digest pinning;
- AppArmor on the host and no administrative Docker-group membership.
- configured private management and SSH authorization capabilities under
  `tentaflake.management`; the supported Tailscale backend retains
  `tag:agent-host` and restrictive operator-maintained remote policy, while
  the public OpenSSH module is rejected. See the
  [management contract](https://docs.tentaflake.dev/operations/tailscale/#host-contract-and-migration).
- normalized mount paths; `.` and `..` escapes are rejected;
- an enabled broker, disposable worker, fixed-size workspace quota, and research relay before
  a secure controller may set `autoStart = true`.

Default per-agent limits are configurable under
`tentaflake.security.resources`: `memory = "2g"`, `memorySwap = "2g"`,
`cpus = "2.0"`, `nofile = 4096`, `tmpfsSize = "256m"`, and
`runTmpfsSize = "64m"`. PID defaults remain runtime-builder specific.

`tentaflake.workspaceQuota.agents` provides a backend-independent hard
workspace ceiling by mounting a fixed-size Btrfs image before the controller.
It is opt-in because first activation formats a new backing file and existing
data requires an explicit migration. See
[persistent workspace quota](https://docs.tentaflake.dev/configuration/workspace-quota/). Runtime-specific mutable
state outside that workspace remains outside this limit.

### User-namespace evidence boundary

Secure policy rejects caller-supplied `--userns`, including `--userns=host`.
That prevents a builder override from weakening a daemon-level mapping, but it
does not itself enable Docker `userns-remap`. Docker remapping is daemon-wide
and changes ownership requirements for every persistent bind mount. The
current root-managed Podman OCI path likewise cannot use the rootless-only
`keep-id` or `nomap` modes. Tentaflake therefore relies on the tested `runsc`
sandbox plus a non-root in-container UID today and does not claim a separate
host UID mapping. Adding one requires an explicit state, seed, Git-helper, and
workspace ownership migration with Docker and Podman runtime tests.

## Image signatures and provenance

Digest pinning makes image bytes reproducible but does not identify or trust a
publisher. When an upstream publishes Cosign signatures, record the exact
signing identity (or reviewed public key) per generated container name:

```nix
tentaflake.imageProvenance = {
  requireForSecureAgents = true;
  agents.hermes-assistant = {
    mode = "keyless";
    certificateIdentity =
      "https://github.com/example/project/"
      + ".github/workflows/release.yml@"
      + "refs/tags/v1.2.3";
    certificateOidcIssuer =
      "https://token.actions.githubusercontent.com";
  };
};
```

The generated `tentaflake-image-verify-*` oneshot runs `cosign verify` against
the same digest-pinned reference. The OCI service requires that unit, so a
missing signature, wrong identity, registry failure, or verification error
prevents the controller from starting. `mode = "key"` instead accepts an
absolute `publicKeyFile`; public verification keys are not secrets. Do not
invent an identity for an unsigned upstream. Leave signature enforcement off,
retain the digest pin, document the reviewed source, and accept the explicit
`TFSEC-024` warning until the publisher supplies verifiable signatures.

## Breaking migration

Old configurations that inject provider keys, publish dashboards/APIs, select
host networking, or use mutable images now fail under the installed default.
Choose one of two explicit paths:

1. Keep the fail-closed balanced capsule, remove direct credentials and ports,
   and either keep it stopped and offline or declare the LLM broker, worker,
   workspace quota, and research relay required for automatic start. See
   [brokered egress](https://docs.tentaflake.dev/configuration/brokered-egress/) and [web research](https://docs.tentaflake.dev/configuration/research/).
2. For a trusted development-only machine, acknowledge the old authority:

   ```nix
   tentaflake.security.profile = "dev";
   ```

The second path restores compatibility; it is not recommended for untrusted
24/7 agents. Legacy `agents.json` entries with direct environment files or
service ports require dev. The schema-v1 generic `agents` array accepts stopped
balanced adapter declarations; see [agent adapters](https://docs.tentaflake.dev/configuration/agent-adapters/).

The former `tentaflake.networking.egress` option is renamed to
`tentaflake.networking.legacyPortEgress`. It is permitted only in `dev`. It
filters host `OUTPUT` by destination port and is explicitly not a destination
allowlist, container `FORWARD` policy, DNS control, or secure egress broker.

## Git auto-push

Every enabled host-side push needs a runtime `tokenEnvFile`, a non-empty list
of canonical full GitHub HTTPS remotes, and at least one branch:

```nix
gitAutoPush = {
  tokenEnvFile = "/run/agenix/github-coding";
  allowedRemotes = [
    "https://github.com/example/coding.git"
  ];
  allowedBranches = [ "main" ];
};
```

The host helper canonicalizes scheme, host, owner, and repository, rejects
userinfo, subdomains, alternate schemes, extra paths, query/fragment syntax,
and double `.git` suffixes, then pushes the exact checked branch. It uses a
repository-scoped `safe.directory`, never `*`, and writes allow/deny/error
records to journald. Use a fine-grained repository token or GitHub App
credential with only contents-write permission.

## Posture check

```text
tentaflake doctor --security
tentaflake doctor --security --json
```

The generated desired-state manifest covers profile agreement, host OpenSSH,
AppArmor, Docker-group membership, broker presence, networking, ports,
privilege, user, capabilities, `no-new-privileges`, root writability, mounts,
resource flags, `runsc`, digest pins, environment files, and disposable-worker
presence. Finding IDs are stable and critical/high findings return a non-zero
exit code. `TFSEC-020` is high severity when a secure agent has no worker;
`TFSEC-021` reports a missing managed workspace ceiling; `TFSEC-022` and
`TFSEC-023` cover declarative Seccomp/AppArmor confinement. `TFSEC-024` warns
when no publisher-signature start gate is configured. `TFSEC-026` reports a
missing backup declaration; `TFSEC-027` reports a missing/stale success stamp.
`TFSEC-028` through `TFSEC-032` cover unknown or active Serve/Funnel state and
root-disk pressure. `TFSEC-033` marks unavailable live OCI evidence and
`TFSEC-034` is a critical live-inspect mismatch. `TFSEC-035` keeps an
unreachable broker unknown, `TFSEC-036` is a high-severity explicit unhealthy
response, and `TFSEC-037` detects inconsistent broker labels/modes/endpoints.
`TFSEC-038` keeps legacy/incomplete management declarations unknown;
`TFSEC-039` reports a disabled declared private SSH policy. The generated
manifest starts with `manifest<TAB>2` and separates host policy from its typed
management record. Legacy enable flags never prove private operator access.

The check combines declarative desired-state evidence with narrow live probes
for Tailscale Serve/Funnel, backup freshness, root-disk pressure, and narrow
Docker/Podman inspect comparisons. Podman uses its documented `OCIRuntime`,
effective/bounding capabilities, AppArmor profile, and Docker-compatible
`HostConfig`; missing schema fields stay explicitly unknown. A blocked or
unparseable probe is a warning, never green. Configured broker endpoints are
also checked for credential, policy, and audit-path readiness through
`/healthz`. The generated manifest records each capsule's exact broker-network
name, bind mounts, tmpfs sizes, absence of ports, and exact resource limits.
Inspection requires `State.Running=true`, the exact network set and internal
broker bridge, no published ports, and matching mounts and limits. Missing or
legacy manifest fields remain unknown. Each OCI subprocess has a five-second
deadline and an 8 MiB combined stdout/stderr ceiling, including inherited
output handles; overflow remains unknown. Broker
health probes run in batches of at most 16. The check still does not prove
complete controller-tool audit coverage, remote policy delivery, or cross-agent
network denial; those require
target-host runtime and VM evidence.

The VM suite supplies part of that runtime evidence with separate Docker,
Podman, and external-attacker nodes plus locally built no-network fixture
images under `runsc`. It checks public-listener denial, direct
DNS/Internet/loopback/private/tailnet/link-local/metadata denial,
an actual scoped LLM-broker response, deterministic bridge interfaces,
non-root/empty capabilities/read-only root, bounded workspace and PID behavior,
Docker live resource-limit inspection and doctor detection of deliberately
changed memory limits, hidden sensitive state/sockets,
changed-remote rejection, worker timeout/approval/cleanup, broker crash
restart, unsafe-doctor exit status, backup/restore, and a controlled VM reboot.
The pinned Podman/runsc fixture reports an empty AppArmor profile and `null`
effective/bounding capability fields. Its live doctor result is therefore
critical for absent AppArmor; the VM verifies that negative result, not full
Podman confinement. Complete-schema Rust fixtures test parsing independently.

It does not emulate a real tailnet policy, upstream provider, production
registry, or a production-host activation/reboot drill.

## Threat model and residual risk

The complete asset, adversary, trust-boundary, control/evidence, non-goal, and
deployment-acceptance analysis is maintained in the
[threat model](https://docs.tentaflake.dev/security/threat-model/). The summary below is intentionally brief.

Assets are the NixOS host, other agents, operator access, persistent state,
provider/Git/backup credentials, and external accounts. Attackers include a
compromised agent runtime, malicious web/document content, dependencies, image
content, and hostile model output. The model assumes the host kernel, NixOS
closure, gVisor package, OCI daemon, and operator identity are trusted.

Phase A narrows filesystem and kernel authority. Phase B prevents direct
egress, substitutes host-held provider credentials, and constrains external
web retrieval. The disposable worker additionally bounds offline untrusted
execution and requires host approval for non-local action classes, but cannot
prove every controller tool actually routed work through it. It does not prove
that an image digest is trustworthy, make
Docker/gVisor immune to vulnerabilities, prevent malicious writes inside the
agent's own workspace, bound runtime-specific state outside the workspace,
prove a host UID remap, enforce tailnet policy, or make untrusted web content
safe. Cosign verification
proves only that the configured signer authorized the pinned artifact; it does
not prove that the software is benign. Those are explicit residual risks and
dependencies, not hidden fallbacks.

---

# Threat model

> Review assets, adversaries, trust assumptions, and mitigations.

Canonical page: https://docs.tentaflake.dev/security/threat-model/

> Source: [`docs/15-threat-model.md`](https://github.com/timfewi/tentaflake/blob/e21be475a3afc6e95c5087003edd0b7f780e4df4/docs/15-threat-model.md) at `e21be475a3af` · docs version **0.4.37**

This document defines the security model for a tentaflake host running
long-lived, potentially compromised AI agents. It describes the intended
boundaries of the generic template. A deployment fork must add a target-host
review, credential inventory, tailnet policy, provider policy, and recovery
evidence.

## Security objective

The `balanced` profile assumes that prompts, retrieved content, dependencies,
and the agent process can become hostile. Compromise of one controller should
not by itself grant access to host secrets, another agent, direct Internet or
LAN egress, a privileged container runtime, or an unrestricted host shell.
Required external access is mediated by narrow host services. Risky local code
execution is delegated to a disposable, offline worker.

The `dev` profile is a compatibility mode, not a security boundary. The
`strict` profile fails evaluation because a tested separate-kernel boundary is
not yet implemented. There is no silent downgrade to either profile.

## Assets

The model protects:

- the host kernel, NixOS configuration, boot state, container engine, and
  system services;
- operator access, tailnet identity, and recovery authority;
- provider, Git, backup, signing, and deployment credentials;
- agent workspaces, broker audit records, backups, and operational logs;
- other agents and their state, credentials, budgets, and broker endpoints;
- external provider accounts, repositories, websites, and cost budgets;
- the integrity and availability of the declarative configuration and pinned
  software artifacts.

## Adversaries and failure modes

The design considers:

- prompt injection or malicious model output controlling an agent;
- hostile documents, websites, repositories, packages, and generated code;
- a compromised or malicious controller image or dependency;
- attempts to escape a container, abuse a Linux capability, exhaust resources,
  scan local networks, reach metadata services, or steal runtime credentials;
- cross-agent access, broker impersonation, SSRF, DNS rebinding, redirect
  abuse, request smuggling through oversized inputs, and cost exhaustion;
- unauthorized Git pushes or external side effects;
- a remote attacker reaching an accidentally published management service;
- operator error, stale backups, policy drift, storage exhaustion, service
  crash loops, and incomplete monitoring;
- supply-chain substitution of a mutable or unauthorized container image.

Denial of service against the whole host through a kernel, storage, container
engine, or hardware failure is reduced but not eliminated.

## Trust assumptions

The following remain trusted computing base or operator assumptions:

- the physical machine, firmware, host kernel, NixOS closure, systemd, Nix
  evaluation, and the operator account;
- the selected Docker or Podman engine, gVisor `runsc`, firewall, filesystem,
  and their correct activated configuration;
- the deployment fork's runtime secret provisioning and file ownership;
- reviewed image-signing identities and keys when provenance enforcement is
  enabled;
- correct provider, repository, backup, tailnet, DNS, and alerting policy owned
  outside this generic template;
- operators review the exact build and target before activation and perform
  real restore and incident-response drills.

An image digest gives reproducible identity, not trust. Cosign verification
shows only that the configured signer authorized that exact artifact. Neither
claim proves that the software is benign.

## Trust boundaries and data flows

### Operator to host

Management enters through the operator account and an explicitly installed
tailnet grants/SSH policy. Tentaflake does not publish its CLI or observability
listeners to a public interface. Tailscale transport alone is not authorization,
and the template cannot verify the remote control-plane policy.

### Host to controller capsule

The host starts a digest-pinned controller with `runsc`, a non-root user, no
added capabilities, a read-only root filesystem, bounded resources, selected
AppArmor/Seccomp policy, and only declared mounts. Secure profiles reject host
networking, published ports, extra devices, unconfined overrides, sensitive
mounts, mutable image tags, and direct provider credential files.

Caller-supplied host user-namespace overrides are rejected. The current
root-managed Docker/Podman paths do not prove daemon-level host UID remapping:
Docker remapping would change every persistent bind-mount owner, and Podman's
`keep-id`/`nomap` modes are rootless-only. `runsc` and the non-root container
UID are enforced boundaries; separate host UID mapping remains an explicit
residual risk until an ownership migration and both runtime paths are tested.
The pinned Podman/runsc VM also reports no AppArmor profile and incomplete
capability metadata; the security doctor treats the absent profile as critical.
Host AppArmor enablement alone is not proof of that capsule's confinement.

The controller workspace is persistent and writable. An optional exact-size
filesystem bounds that workspace, and an optional separate filesystem bounds
private runtime state. The controller may still corrupt or delete its own
data. Other state and aggregate sparse-image allocation need host capacity
policy; individual quotas do not reserve free space for every agent.

### Controller to external services

A balanced controller either has `network=none` or joins one agent-specific
internal bridge. Research-enabled agents use only that agent's host LLM broker
on the bridge; legacy fetch is rejected. Deterministic interface and source-subnet firewall rules reject
forwarding and unrelated host services. The controller receives only a
runtime virtual key, never the upstream credential.

The LLM broker enforces exact routes and models, clamps completion size, and
applies concurrency, rate, token, and cost budgets. It rejects provider-hosted
tools and known web extension fields; operators must select inference-only
models. [Web research](https://docs.tentaflake.dev/configuration/research/) uses a per-agent Unix socket capability
and the public isolated service. Each relay has a distinct host UID even when
container UIDs match. Only its exact socket directory and client closure are
projected read-only; gVisor can open these sockets but cannot create host sockets.
VPN readiness, public-address checks, budgets and worker isolation constrain
research egress. These controls do not make retrieved content safe to execute
or trust, or prevent disclosure of text supplied to an allowed remote service.

Each broker process has
systemd memory/task/file ceilings. Optional aggregate admission checks count
LLM and fetch separately for concurrency and request rate. The credential
setup unit has no capabilities and uses an explicitly read-only broker parent
with one writable agent directory. It preserves its virtual key across unit
restarts. These host controls preserve the existing
provider-credential boundary; they do not add a new external integration.

### Controller to disposable worker

The worker receives a bounded, descriptor-safe workspace snapshot and runs a
short-lived `runsc` capsule with no network or secrets. It returns artifacts
through a controller-specific read-only import. Only `local-reversible`
actions run automatically; other permitted classes wait for a host-side
approval record, and `forbidden` is rejected.

This boundary cannot prove that every tool exposed by every upstream agent
runtime routes shell or file work through the worker. Runtime-specific tool
configuration remains a deployment review item. Approval also does not grant
network access or implement a generic external-action adapter.

### Host to external systems

Git auto-push, backups, provider access, signature verification, remote
tailnet policy, and alert delivery execute with host-held authority. Their
credentials remain runtime files or systemd credentials. Each integration
must be narrowly scoped by the deployment: exact repository and branch,
provider account and models, backup repository, signer identity, tailnet
grants, and alert receiver.

### Telemetry and detection

Systemd journals are the primary local audit trail; there is no custom SQLite
audit database or web console. The optional Alloy/Loki/Prometheus/Grafana
profile is loopback-only and supplies baseline metrics and rules. The separate
Falco profile detects selected runtime behavior but does not prevent it.
Retention, off-host evidence, notification delivery, and sensitive-log review
remain deployment responsibilities.

## Control and evidence map

| Risk | Primary control | Required evidence |
|---|---|---|
| Host or peer access | gVisor capsule, mounts, capabilities, firewall | eval, build, live inspect, negative VM test |
| Direct egress | internal per-agent bridge and broker-only rules | activated firewall plus DNS/LAN/metadata tests |
| Credential theft | host-held credentials and virtual broker key | generated unit and live mount/environment review |
| SSRF and rebinding | fetch allowlist, public-IP checks, pinned DNS | broker tests and target-host broker health |
| Cost exhaustion | rate, concurrency, token, and cost budgets | policy review, audit output, alerts |
| Unsafe code execution | disposable offline worker and approval classes | queue, timeout, cleanup, and no-egress tests |
| Unauthorized Git write | exact remote and branch host helper | denial tests and least-privilege credential review |
| Mutable image substitution | digest pin and optional Cosign start gate | verified policy plus failed-signature start test |
| Resource exhaustion | memory, CPU, PID, tmpfs, log, and workspace limits | live limits and exhaustion tests |
| Lost state | encrypted Restic policy and success freshness | real fresh-host restore drill |
| Runtime anomaly | journald, optional observability and Falco | retention, receiver, and response drill |
| Remote management exposure | loopback listeners and tailnet policy template | live listeners and remote policy audit |

Source inspection proves intended code. Nix evaluation proves option types and
assertions. A successful build proves the closure. VM tests prove only their
fixture. Activation changes a particular host. Live inspection and adversarial
tests are required before calling that host secure. Unavailable AF_UNIX,
container-inspect, tailnet, broker, backup, or alert evidence is unknown, never
green.

## Explicit non-goals and residual risks

Tentaflake does not:

- make Docker, Podman, gVisor, the kernel, firmware, or hardware immune to
  vulnerabilities;
- provide a tested MicroVM or separate-kernel `strict` profile today;
- attest that a signed image, model, dependency, website, or generated artifact
  is safe or correct;
- prevent an agent from damaging its own writable workspace;
- transparently mediate every tool in every supported upstream agent runtime;
- provide generic adapters for arbitrary purchases, messages, deployments, or
  other external side effects;
- configure production tailnet grants, provider accounts, registry trust,
  backup storage, Grafana retention, Alertmanager delivery, or off-host logs;
- guarantee availability under host-wide storage, kernel, engine, network,
  power, or hardware failure;
- replace patch management, key rotation, capacity planning, incident response,
  or human review of requested actions.

Falco and observability improve detection. Backups improve recovery. Neither is
a preventive isolation boundary. A deployment must document accepted residual
risks and the owner of every external control before unattended operation.

## Deployment acceptance checklist

Before treating a host as ready for unattended agents:

1. Keep every untrusted controller on `balanced`; confirm `strict` still fails
   closed and document any deliberate `dev` exception.
2. Review generated units, exact mounts, identities, resource limits, image
   digests, provenance policies, and agent-specific broker policy.
3. Build and run the unit, module-evaluation, and VM suites without activating
   the production host as a side effect.
4. Review and separately approve the exact host activation.
5. Run `tentaflake doctor --security`, live OCI inspection, listener checks,
   and direct-egress, LAN, metadata, peer, and broker negative tests.
6. Audit the deployed tailnet grants/SSH policy and all external credentials
   for least privilege and rotation.
7. Configure log retention, capacity alerts, notification delivery, and an
   operator response path; validate them end to end.
8. Perform a real encrypted backup and fresh-host restore drill.
9. Record remaining unknown evidence and accepted residual risks. Do not count
   a warning or unavailable check as a pass.

---

# Security policy

> Report vulnerabilities and understand supported release policy.

Canonical page: https://docs.tentaflake.dev/security/reporting/

> Source: [`SECURITY.md`](https://github.com/timfewi/tentaflake/blob/e21be475a3afc6e95c5087003edd0b7f780e4df4/SECURITY.md) at `e21be475a3af` · docs version **0.4.37**

## Supported versions

Security updates apply to the latest commit on `main`.

## Report a vulnerability

Do not open a public issue. Use the repository's GitHub Security Advisory
workflow and include the affected component, reproduction, impact, and any
suggested mitigation. The project aims to acknowledge reports within 72 hours.

## Scope

Relevant areas include:

| Area | Examples |
|---|---|
| Secrets | Nix-store, log, environment, or Git leakage |
| Containers | Breakout, mounts, users, capabilities |
| Agent policy | Files, tools, network, repository access |
| Installer | Target resolution and disk-wipe safety |
| Rust CLI | Argument handling and privileged commands |
| Observability | Dashboard exposure and log sensitivity |
| Falco | eBPF privileges, rules, event handling |
| Nix | Supply chain, evaluation, and pinning |

## Deployment practices

- Never commit environment files or literal secret values.
- Use runtime secret files such as agenix outputs.
- Enable only the tools, mounts, and network access an agent needs.
- Treat Docker group membership as root-equivalent.
- Keep Grafana, Prometheus, Loki, and Alloy loopback-only unless an
  authenticated private publishing path is deliberately configured.
- Treat Falco as detection, not prevention or container isolation.
- Keep installed agent hosts on the `balanced` profile. `dev` is a deliberate
  compatibility path, and `strict` is not implemented.
- Install a restrictive tailnet grants/SSH policy for `tag:agent-host` before
  relying on Tailscale; private transport does not replace authorization.
- Do not work around balanced `network=none` by injecting provider credentials.
  Use the per-agent broker path; the agent receives only a virtual key and a
  network with host/FORWARD firewall restrictions.

The detailed security model is split between the
[threat model](https://docs.tentaflake.dev/security/threat-model/) and
[security profiles and broker migration](https://docs.tentaflake.dev/security/profiles/).

## Incident response

1. Stop the exact affected unit after resolving its backend and name:

   ```bash
   tentaflake stop <agent>
   ```

2. Revoke provider and repository credentials. Stopping a container does not
   invalidate an already leaked key.
3. Preserve relevant journald logs, Falco output when enabled, and a read-only
   copy of the agent state directory before cleanup.
4. Inspect the declared mounts, tools, network policy, and recent changes.
5. Rotate runtime secrets and build the repaired configuration.
6. Activate only after reviewing the target and build result.

Tentaflake does not maintain a private SQLite audit database or custom web
console. Host and container-unit evidence is available through journald;
the optional Loki/Alloy profile can retain and query it.

---

# Bundled skills

> Find the bundled operational and Hermes skills.

Canonical page: https://docs.tentaflake.dev/reference/skills/

> Source: [`docs/03-skill-index.md`](https://github.com/timfewi/tentaflake/blob/e21be475a3afc6e95c5087003edd0b7f780e4df4/docs/03-skill-index.md) at `e21be475a3af` · docs version **0.4.37**

Skills are procedural reference files. Tentaflake bundles them under
`.agents/skills/` for development agents and for optional seeding into Hermes
state. They do not grant tool access, credentials, network paths, or operator
authorization.

## Inventory

| Skill | Purpose |
|---|---|
| [handle-the-host](https://github.com/timfewi/tentaflake/blob/e21be475a3afc6e95c5087003edd0b7f780e4df4/.agents/skills/handle-the-host/SKILL.md) | Tailscale SSH, host inspection, maintenance and recovery |
| [tentaflake-change-review](https://github.com/timfewi/tentaflake/blob/e21be475a3afc6e95c5087003edd0b7f780e4df4/.agents/skills/tentaflake-change-review/SKILL.md) | Requirement, focused verification and documentation review |
| [tentaflake-repo-guidance](https://github.com/timfewi/tentaflake/blob/e21be475a3afc6e95c5087003edd0b7f780e4df4/.agents/skills/tentaflake-repo-guidance/SKILL.md) | Modules, adapters, builders, CLI, installer and checks |
| [hermes-config-manager](https://github.com/timfewi/tentaflake/blob/e21be475a3afc6e95c5087003edd0b7f780e4df4/.agents/skills/hermes-config-manager/SKILL.md) | Declarative Hermes settings and read-only configuration |
| [hermes-provider-setup](https://github.com/timfewi/tentaflake/blob/e21be475a3afc6e95c5087003edd0b7f780e4df4/.agents/skills/hermes-provider-setup/SKILL.md) | Provider and model configuration |
| [hermes-memory-personality](https://github.com/timfewi/tentaflake/blob/e21be475a3afc6e95c5087003edd0b7f780e4df4/.agents/skills/hermes-memory-personality/SKILL.md) | Memory, user context and personality files |
| [hermes-tools-config](https://github.com/timfewi/tentaflake/blob/e21be475a3afc6e95c5087003edd0b7f780e4df4/.agents/skills/hermes-tools-config/SKILL.md) | Tool selection, output limits and Research/worker routing |

Each skill is a short procedure with links to the owning guides and source.
The Hermes references follow Tentaflake's declarative configuration and host
policy. Native commands, option names and limits depend on the pinned image;
check matching help/source instead of assuming the latest upstream behavior.
The host's deprecated `hermes` command is a Tentaflake CLI shim; inspect native
help inside the selected container. A skill does not establish vendor startup,
MCP discovery or execution acceptance.

## Seed skills deliberately

The Hermes builder accepts `seedDir`, copied into the agent's state without
overwriting existing files. To seed this repository's skill collection, use a
consumer-owned directory shaped like this:

```text
seed/
  skills/
    handle-the-host/
      SKILL.md
    hermes-config-manager/
      SKILL.md
```

Point `seedDir` at `seed/`, not directly at `.agents/skills/`; the contents are
copied into the state root, which needs a `skills/` child. The path becomes a
Nix input, so include only public, non-secret material. Private context needs
runtime provisioning even in a fork. Deployment-specific content
belongs in the consumer fork, never in the generic template.

Seeding is not automatic synchronization. Changing the seed does not overwrite
an existing skill. Review and update existing state explicitly while preserving
runtime changes. See [agent management](https://docs.tentaflake.dev/operations/agent-management/).

Guide/source links in the bundled files refer to this checkout. Before seeding,
adapt them to the deployment's pinned documentation location; copying skills
alone does not make repository guides available inside the capsule. Preserve
links between skills when selecting only part of the collection.

## Configuration and security limits

Generated Hermes `config.yaml` is mounted read-only when effective `settings`
is non-empty; enabling Research adds settings even when callers supply none.
Use the declarative configuration rather than an interactive save command.
Provider credentials and provider selection instructions in generic Hermes
skills must respect the host profile: balanced capsules receive only a virtual
LLM-broker key. Direct provider credentials and alternate terminal/network
backends are dev-only compatibility instructions.

Balanced web access uses only the pinned `secure-research-tool` stdio relay.
Do not install native web fetch tools, remote MCP servers, provider-hosted web
tools, or alternate browser transports to work around this boundary. See
[web research](https://docs.tentaflake.dev/configuration/research/) and [the threat model](https://docs.tentaflake.dev/security/threat-model/).

Installing additional skills from a community tap requires source review and a
permitted acquisition path. The template does not bundle or verify those taps;
a balanced capsule has no general GitHub download route. Prepare reviewed skill
files outside the capsule and seed them deliberately instead of assuming an
in-container install command can access the Internet.

## Author and maintain a skill

A skill needs a `SKILL.md` with its name, description, and bounded instructions:

```markdown
---
name: example-workflow
description: Inspect one declared agent and report evidence gaps.
---

# Example workflow

1. Resolve the target from the generated inventory.
2. Read the relevant guide and inspect the declared policy.
3. Report what was verified and which runtime evidence is missing.
```

Keep reusable procedures separate from deployment identities and secret values.
Keep the skill to its trigger, actionable steps, boundaries and verification;
link detailed option tables and examples in the owning guide instead of copying
them. Validate local links and frontmatter (`name` matches the directory and
`description` names the task). Verify version-dependent claims against the
pinned runtime, and keep secrets out of diagnostic commands.
When behavior changes, update the relevant skill and its authoritative guide in
the same change. [Documentation maintenance](https://docs.tentaflake.dev/reference/documentation/) describes
source ownership and the website's release boundary.

---

# Fork checklist

> Adapt the generic template safely in a deployment fork.

Canonical page: https://docs.tentaflake.dev/reference/fork-checklist/

> Source: [`docs/05-fork-checklist.md`](https://github.com/timfewi/tentaflake/blob/e21be475a3afc6e95c5087003edd0b7f780e4df4/docs/05-fork-checklist.md) at `e21be475a3af` · docs version **0.4.37**

Change deployment-specific values in a fork, never in the generic template.

| What | Template location |
|---|---|
| Host and admin defaults | `lib/constants.nix` |
| Host modules | `flake.nix` |
| Agent definitions | `my-agents.nix` |
| Runtime secret paths | Host broker, backup and optional profile declarations; direct-agent paths are dev-only |
| SSH keys | `tentaflake.adminAuthorizedKeys` |
| Timezone and locale | `tentaflake.*` options |

The Rust package name and workspace live in `Cargo.toml` and
`crates/tentaflake-cli/Cargo.toml`. If a fork renames the project, update those
manifests, the Nix package wrapper, CLI paths, documentation, and tests together.

## Pinning

The repository tracks `nixos-unstable` at the exact revision in `flake.lock`.
Update deliberately, review the lock diff, and rerun the flake and VM checks.
Do not weaken package security policy merely to retain an old pin.

## Secrets

Create encrypted secret declarations in the fork. Keep the decryption identity
off-host in a protected recovery location. Literal secret values must not enter
Nix expressions, JSON, Git, logs, or build outputs.

## Backend choice

Docker is the default. Under `balanced`, the operator is deliberately not in
its root-equivalent group; narrow CLI actions use sudo. `dev` retains group
access for compatibility. Podman does not add that group, but root-managed
systemd containers and the operator's rootless store are separate. Verify CLI
behavior against the selected backend.

## Security profile

Keep the installed default `balanced` for an untrusted 24/7 workload. It is
fail-closed and has no external connectivity until an exact per-agent broker
policy is declared.
Select `dev` only when a trusted development workflow explicitly needs direct
env credentials, host networking, ports, or mutable local images. `strict`
currently fails evaluation and must not be advertised as deployed.

Enable and migrate to `tentaflake.workspaceQuota.agents.<container>` for every
persistent controller workspace that needs a hard ceiling. The generic module
uses an exact-size Btrfs backing image and refuses to hide existing non-empty
data; runtime-specific state outside the workspace still needs capacity
monitoring.

## Optional profiles

Import observability or Falco only when the deployment needs them. Keep editor,
Hive Research and speech integrations in the consumer flake. Web/research uses
the core [secure research transport](https://docs.tentaflake.dev/configuration/research/). Each optional integration
has a separate package,
credential, network, or privilege boundary; see
[observability and detection](https://docs.tentaflake.dev/operations/observability/).

---

# Build boundaries and contributor checks

> Understand the fast checks, selective VM suites, and build boundaries.

Canonical page: https://docs.tentaflake.dev/reference/builds/

> Source: [`docs/17-builds.md`](https://github.com/timfewi/tentaflake/blob/e21be475a3afc6e95c5087003edd0b7f780e4df4/docs/17-builds.md) at `e21be475a3af` · docs version **0.4.37**

Start with `just fast`. It runs formatting, Nix lint, ShellCheck, Rust checks,
the CI-selection/source-boundary regressions, read-only flake evaluation and
the generated installed flake, including focused adapter evaluation.
`checks.*.agent-adapters` covers compatible mappings, JSON, inventory, lazy
selection, stopped OpenClaw, quota declarations and generated quota-helper refusal
regressions without mounts. Required research closure preparation uses
non-VM builds. `just e2e` still runs the complete gate, both VM suites and the
installer ISO. Run an affected VM suite when changing runtime or security
behavior; use the ISO gate when changing installation.

## Rust cache boundaries

Cargo remains the Rust build definition. `[workspace.package]` in `Cargo.toml`
supplies the shared version; each crate supplies its package description.
Nix packages and the worker image read those manifests. A replaced worker
image supplies its default local reference through `imageName`/`imageTag`;
images without those attributes require an explicit `worker.imageReference`.

Each Rust Nix source contains root Cargo files, all member manifests, and the
selected crate's complete source. Cargo can resolve the workspace while an
unrelated crate source edit leaves the other package derivations unchanged.
Shared manifests, dependencies or toolchain changes still invalidate affected
builds. Declare additional root build inputs explicitly; add shared/path
dependencies to the fileset when introducing them.

A comparison of actual Nix derivation paths with a temporary CLI source edit
invalidated all three packages before this boundary and only the CLI after it.
All three focused-source packages built successfully. This establishes cache
isolation, not a fixed wall-clock speedup. `checks.*.rust-package-sources`
guards manifest availability, source isolation and the worker image version.

## GitHub check selection

The imported policy is `.github/ci-paths.json`; its consumer is
`scripts/ci_changes.py`. One ordered policy selects non-VM checks and the two
VM suites independently. Component paths take precedence over Markdown suffixes.

| Change | Non-VM checks | VM suite |
| --- | --- | --- |
| Docs and listed presentation metadata | Routing regression and whitespace; no Nix setup/cache | None |
| CI routing/workflow/test definition | Routing regression, focused selection package and static checks | None |
| One Rust crate/package | Rust workspace checks, that package and source-boundary check; worker includes its image | Runtime |
| Worker module | Rust workspace/worker package and image; actual worker configuration and policy in the runtime fixture | Runtime |
| Shared Cargo inputs | Rust workspace and all three packages/image; no Research/configuration preparation | Runtime |
| Installer scripts or host VM fixture | Static checks and generated installed flake | Runtime |
| Installer Nix | Configuration evaluation, static checks and generated installed flake | Runtime |
| Research module/client/fixture or research-only input pin | Research/configuration preparation and evaluation, static checks and generated installed flake | Research |
| Dev Container inputs | Dev Container package and static checks | None |
| Generated-flake script or module-evaluation fixture | Its generated-flake or configuration gate and static checks | None |
| Onboarding validator/fixture | Installed preset/generic CLI fixture and static checks; no Research preparation | None |
| Operator onboarding CLI module | Rust/CLI package, source-boundary and installed fixture checks; no Research preparation | None |
| Shared JSON agent parser | Installed fixture and static checks; no Research preparation | Runtime |
| Adapters, shared modules/helpers, other lock updates, unknown paths/history | Full gate | Both |

The selector reads NUL-delimited Git paths and includes deleted/old rename
paths. Invalid lock data requires the full gate. The real-Git regression check
runs on every event directly on the runner; `checks.*.ci-vm-selection` also
packages it for the contributor gate. CI metadata changes are covered by these
regressions rather than building unchanged guest systems.

Documentation does not start Nix installation, cache upload, Research closure
materialization, full flake evaluation or Rust/system builds. A Rust-only change
does not build the separate Research service. Worker-module changes use the
same focused package/runtime gates: the Research fixture enables no workers.
Mixing these changes with shared security paths retains the full gate.
Non-VM package builds run in
groups of at most four. Configuration evaluation still prepares the exact
Research closures because the full flake/module fixtures need them; shared
configuration changes retain their applicable runtime and Research evidence.
Onboarding validation uses the installed fixture rather than full flake checks;
its CLI binary is a small selected package or an explicitly supplied local binary.
The standalone host build runs only for configuration changes affecting the
runtime, not for a Research-only or documentation change. Full VM/ISO/system
workloads remain separate from local fast checks and require their own scope.
New pushes cancel superseded runs for the same PR or branch.

## Nix and Bazel

[Nix + Bazel](https://nix-bazel.build/) and the linked
[rules_nixpkgs guide](https://github.com/tweag/rules_nixpkgs/blob/master/guide.md)
describe providing Nix toolchains/dependencies to Bazel's finer build actions.
That approach can help a large code graph with reusable remote actions.

Tentaflake currently has three independent Rust binaries, plus NixOS modules
and VM tests. Cargo already provides incremental development builds. Narrower
Nix sources retain the other binaries' cached packages, and VM selection avoids
irrelevant guest boots. Bazel would require another target/dependency graph and
cache configuration while the NixOS/VM work remains. No Bazel speedup has been
measured here. Keep Cargo/Nix as the build definition for now; reconsider a
bounded Bazel prototype if measurements show compilation dominates after these
changes or the workspace grows into a larger shared code graph.

---

# Changelog

> Review historical and current user-facing changes across tentaflake releases.

Canonical page: https://docs.tentaflake.dev/reference/changelog/

> Source: [`CHANGELOG.md`](https://github.com/timfewi/tentaflake/blob/e21be475a3afc6e95c5087003edd0b7f780e4df4/CHANGELOG.md) at `e21be475a3af` · docs version **0.4.37**

All notable changes to tentaflake are documented here.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]
### Fixed
- Preserve completed worker results without redispatching stale requests, and
  retain complete inbox copies when legacy private pending state is damaged.
- Identify the exact quota loop backing image inside systemd namespaces;
  `ReadWritePaths` bind mounts must not bypass underlying-data checks or prove
  quota ownership readiness.
- Reject quota mount-source and parent symlinks before privileged permission
  changes; generated-helper regressions preserve unrelated data and permissions.
- Preserve optional bounded `x-opencode-session` conversation-affinity metadata
  on both LLM broker routes for JSON and SSE. Reject malformed values before
  budget reservation or upstream dispatch, omit session IDs from audit logs,
  and keep arbitrary client headers excluded. Clients still own ID propagation.
### Added
- Durable worker pending/running claims, conservative interrupted-job recovery,
  per-agent operator/worker exclusion and aggregate inbox/private queue ceilings.
- Optional fixed-size private agent state through the existing Btrfs quota
  module, with exact adapter path/ownership, mount ordering, startup dependencies
  and selected Restic state/workspace sources. Existing data needs explicit
  offline migration; no automatic conversion or activation is performed.
- Record the product roadmap for general-purpose AI employees, company workflows,
  business connectors and shared runtime configuration, with linked GitHub milestones
  and acceptance criteria. Existing issues are reused and missing scope is tracked
  separately. This documents planned work; website publication remains separate.
- Add a lazy agent adapter registry and `mkAgent`, preserving Hermes/ZeroClaw
  public arguments, container/unit identities, ownership and existing state.
  Explicit metadata drives CLI inventory; versioned generic JSON is additive
  to legacy arrays and rejects unknown fields/adapters and identity collisions.
- Investigate OpenClaw v2026.9.7 and add an explicitly stopped refusal scaffold.
  A reviewed image, broker transport, actual Research discovery and worker tool
  routing remain acceptance prerequisites; this is not operational support.
  Versioned source confirms built-in OpenAI streaming requests; the opt-in
  broker SSE transport still needs acceptance with that runtime. Default
  rejection probes retain the existing budget and credential boundaries.
### Changed
- Pin the published `tentaflake-research` revision 91ce540 (replacing e31fce2):
  independent client/service split, neutral provider defaults, streamed-HTML and
  crawl fixes, typed failure details, constrained package sources, formatted HTML
  text and selectable VPN readiness evidence (WireGuard handshake and peer pins,
  egress-path evidence, planned draining; default off). No fixture or example
  relied on an implicit search provider. The Research policy, module-evaluation and
  adapter checks, the Research integration VM, the flake check, the CI-selection
  checks and the generated installed flake passed; no host was activated and the
  rest of the lockfile is unchanged. Tracked as #110.
### Security
- Automatic-start admission checks the effective OCI setting, closing an
  override path that could start a stopped capsule without its required
  broker, worker, workspace quota and Research declarations.
- Research capability validation uses registered adapter metadata rather than
  trusting runtime-like OCI names. Upstream settings hooks share the existing
  exact read-only closure/socket projection; remote MCP remains rejected.
- Research-enabled controllers use only `secure-research-tool` for web access.
  Native web tools and legacy fetch are disabled, per-agent Unix relays preserve
  distinct job ownership, and model requests stay on the LLM broker. The broker
  rejects provider-hosted tools and web extensions before budget reservation.
- Container mounts reject non-normalized path components, including `..` escapes
  from an approved source or into a sensitive container destination.
- Selectively adopt the archive's exact Docker/Podman posture comparisons and
  bounded OCI subprocess capture: incomplete evidence remains unknown, while
  changed mounts, networks, ports, tmpfs, or resource limits report drift.
  Broker health probes run in bounded batches without serial endpoint delays.
  OCI inspection also caps combined stdout/stderr at 8 MiB and rejects
  overflow as unknown instead of retaining unbounded output until timeout.
- Broker services now have memory/task/file ceilings and optional aggregate
  admission limits. Dual-mode agents count twice for concurrency and rate.
  Credential setup uses only its private preserved runtime directory under
  strict filesystem protection, retaining keys across service restarts.
  VM verification additionally caught the nested runtime directory exposing
  its parent and empty capability lists omitting the directive; explicit
  read-only/writable paths and empty-string capability sets close both gaps.
- Update the broker's locked TLS dependency to `rustls 0.23.45` and its
  required `rustls-webpki 0.103.15`, removing the affected version reported by
  OSV for `RUSTSEC-2026-0285` without changing broker policy or adding code.
- Updated the pinned Dev Containers CLI dependencies `@humanfs/node`,
  `brace-expansion` (all three locked majors), `ip-address`, and `js-yaml` to
  remove current OSV findings. The required HumanFS dependencies and offline
  cache hash move together; the CLI release and prior security backports stay
  unchanged.
### Changed
- Shorten all seven bundled skills into focused, source-linked procedures and
  remove unverified upstream catalogs and credential-dumping instructions.
  Correct read-only Hermes configuration guidance and Agenix recipient/runtime
  separation, host identity setup and environment-file verification.
- Add opt-in bounded SSE forwarding on the existing LLM broker's Chat Completions
  and Responses routes. Streams retain model/tool allowlists and host-only keys,
  cancel on disconnect, enforce byte/time limits and never replay dispatched
  calls. Conservative budget reservations remain charged when usage is missing
  or an exchange aborts; terminal audit admission precedes successful output.
- Refresh documentation for adapter inventory, secure startup prerequisites,
  lifecycle management, skills and credential ownership. Link the public
  website and document authoritative-source and release synchronization rules.
- Link the retained project artwork from the README and an asset overview,
  making logo and wallpaper variants discoverable after removing the old
  ASCII assets.
- Installer and VM fixture import the same disk library, replacing fragile
  extraction of executable code by fixed comment strings.
- GitHub selects host/research VMs separately from an imported path policy;
  contributor-only changes skip them, while unknown/shared changes run both.
- Rust packages include only their own source plus workspace manifests, so
  changes to one binary preserve the other binaries' Nix cache entries.
- Cargo workspace metadata supplies the package/image version and package
  descriptions. Worker image references follow image metadata; shared agent
  UID/GID defaults now import `lib/constants.nix`.
- New installations and fixed-size agent workspaces use Btrfs instead of ext4.
  Workspace images require at least 128 MiB and use read-only offline checks.
- Rust CLI, broker, and worker packages share one builder and a Cargo/crate
  source fileset, removing repeated packaging policy and keeping unrelated
  documentation and host-configuration edits out of their rebuild inputs.
- Secure controllers and brokers now share systemd exponential restart
  backoff from 10 seconds to one minute, avoiding permanent start-limit
  exhaustion after a transient outage. Explicit stops still prevent retries.
### Added
- `just fast` runs the contributor checks, CI-selection regressions and flake
  evaluation without VM suites or the installer ISO.
- Pinned public MIT `tentaflake-research` service, `tentaflake.research.agents`,
  policy checks and a real gVisor stdio-MCP integration fixture. Secure auto-start
  now requires a research relay alongside the broker, worker and workspace quota.
### Fixed
- Keep Tailscale's up-only `--advertise-tags` out of `extraSetFlags`, allowing
  hostname/SSH preferences to apply after manual enrollment. Auth-key enrollment
  retains all three flags through `extraUpFlags`; manual enrollment requests the
  tag explicitly. Regression checks parse the generated flags with the pinned CLI.
- Pin the Dev Containers CLI's transitive `basic-ftp` dependency to 6.2.1
  for `GHSA-c475-qrg2-pj4r`, with the patched Yarn resolution and offline-cache
  hash updated together. The CLI release stays at 0.88.0.
- Installer disk cleanup no longer stops swap or LVM globally. It follows the
  selected device stack, preserves unrelated volumes and refuses cross-disk VGs
  before mutation; encrypted target mappings are identified by device type.
- Pin the verified research client recovery: new MCP calls reconnect after an
  upstream session loss, without replaying interrupted operations.
- `just` recipes load the pinned contributor tools from an ordinary shell,
  avoiding missing Cargo/Statix errors. `just list` now lists available recipes.
- Generated installed flakes now pin and import the research module explicitly,
  avoiding a missing module argument and infinite recursion during installation.
- Broker crash recovery preserves dependent controller processes with systemd
  direct retries. Six consecutive crashes no longer interrupt the controller;
  explicit broker stops still stop it and require a separate controller start.
- Host diagnostics no longer report success when systemd/disk queries fail
  or agent state is unknown. `health` shares `doctor` checks and exit codes;
  intentionally stopped agents remain valid. Text `doctor --hide` now
  redacts host and agent names as advertised, using shared name rendering
  and a single agent-state snapshot instead of duplicated report logic.
- CLI help (`help`, `--help`, `-h`) now works before host configuration is
  installed or while generated inputs are damaged, keeping usage available
  during setup and recovery. Management commands still require those inputs.
- Restic backups now include enabled quota workspaces inside selected state
  paths as separate sources instead of silently skipping their files under
  `--one-file-system`. Required mounts and mount assertions prevent an
  unavailable workspace from producing a misleading successful backup.
- Corrected the Dev Containers CLI offline dependency-cache hash for its
  committed patched lockfile, restoring the package build and full flake
  gate without changing source or dependency versions.
  CI now builds that non-VM check so a stale cache pin cannot pass evaluation
  without exercising the package build.
### Breaking
- Existing ext4 workspace images require a verified backup/restore migration
  before agents can start with the new Btrfs quota module; no automatic conversion.
- Remove the optional Editor, Hive Research, and Piper TTS modules, their flake
  exports, and Piper voice assets. Consumers must remove these imports/options
  and own any replacement integration in their flake.
- Secure controller resource validation now rejects memory/swap above 1 TiB,
  tmpfs above 64 GiB, CPU above 1024, or CPU precision beyond five fractional
  digits, keeping generated limits representable by both OCI backends.

## [0.4.0] - 2026-08-23

### Changed
- GitHub Actions now skips the expensive VM integration build for
  documentation-only changes while retaining the visible `vm-test` job for
  branch-protection compatibility. Every non-Markdown path remains
  conservatively VM-relevant.

### Added
- `just security` now combines a pinned, telemetry-free Semgrep source scan
  with current OSV checks for the Rust and Dev Containers CLI dependency
  lockfiles; GitHub Actions referenced by scanned workflows are commit-pinned.
- Contributor E2E recipes now cover the complete automated gate, a frozen-lock
  Dev Container smoke test, and an interactive installer/installed-system UEFI
  VM backed only by an isolated persistent QCOW2 file.
- A digest-pinned Dev Container can bootstrap Nix and preload the repository's
  lock-file-backed development shell without mounting runtime sockets or
  credentials, giving local editors and Codespaces the same contributor
  toolchain as `nix develop`.

### Fixed
- Contributor E2E now uses a source- and dependency-hash-pinned Dev Containers
  CLI 0.88.0 with the upstream `proxy-from-env` WHATWG URL parsing change
  backported for its CommonJS consumer, removing the legacy `url.parse()` path
  that produced Node.js `DEP0169` security deprecation warnings.
- The installer ISO now embeds its repository below
  `/etc/tentaflake/source`, avoiding a parent-path collision with generated
  runtime manifests such as `/etc/tentaflake/security.tsv`.
- The development shell now uses the canonical `nixfmt` package instead of its
  deprecated `nixfmt-rfc-style` alias, removing the flake evaluation warning.
- The local `just ci` gate no longer stops on existing Statix and Deadnix
  findings; repeated Nix attribute paths are grouped and unused arguments are
  removed without changing their resulting module options.
- Security-manifest generation now records an OCI container with no configured
  user as not non-root instead of passing its `null` user to a string matcher
  and failing evaluation in the `dev` compatibility profile.
- The VM integration test now starts its rebooted node with QEMU reboot support
  enabled and asks systemd to reboot directly instead of relying on a virtual
  key combination. Its headless agent fixtures also omit kmscon, which can
  crash on QEMU's synthetic bochs DRM device, and the stopped-controller
  fixture now declares the broker that its reboot assertion expects to return.
  The controlled-reboot assertion no longer waits indefinitely or checks an
  undeclared boot service.

## [0.4.0-dev] - 2021-08-21
### Breaking
- OpenCode support is removed completely: its builder, image pin, generated
  services, CLI discovery, examples, tests, and current documentation are no
  longer part of the template. Version 0.4 supports Hermes and ZeroClaw;
  OpenClaw is intentionally deferred.
- Phase 0 removes the Go `tentaflake-auditd` package, SQLite event database,
  custom Agent Console, `tentaflake top`, and the interactive agent wizard.
  The operator CLI is now a Rust workspace package verified with Rustfmt,
  Clippy, and Cargo tests.
- The live-agent ISO and its firstboot/persistence scripts are removed. The
  installer ISO is the only bootable image.
- Editor, Hive Research, and Piper TTS leave the core import set and are
  exported as explicit optional modules.
- Installed systems now default to `tentaflake.security.profile = "balanced"`.
  Direct credential env files, host networking, published ports, dashboards,
  extra devices/networks, sensitive mounts, and mutable images fail evaluation
  in a secure profile. Existing trusted development systems must explicitly
  select `dev`; `strict` fails closed until a tested separate-kernel boundary
  exists.
- Automatically started balanced agents now fail evaluation unless their exact
  container key has an enabled broker, disposable worker, and fixed-size
  workspace quota. Deliberately stopped migration scaffolds may remain offline.
- `tentaflake.networking.egress` is renamed to
  `tentaflake.networking.legacyPortEgress` and is dev-only. It was only a host
  OUTPUT port filter, not a destination allowlist or container FORWARD policy.

### Added
- `just security` now combines a pinned, telemetry-free Semgrep source scan
  with current OSV checks for the Rust and Dev Containers CLI dependency
  lockfiles; GitHub Actions referenced by scanned workflows are commit-pinned.
- Contributor E2E recipes now cover the complete automated gate, a frozen-lock
  Dev Container smoke test, and an interactive installer/installed-system UEFI
  VM backed only by an isolated persistent QCOW2 file.
- A digest-pinned Dev Container can bootstrap Nix and preload the repository's
  lock-file-backed development shell without mounting runtime sockets or
  credentials, giving local editors and Codespaces the same contributor
  toolchain as `nix develop`.
- A complete threat model covering assets, adversaries, trust boundaries,
  control/evidence mapping, explicit non-goals, residual risks, and the
  deployment acceptance checklist.
- A per-agent disposable Rust tool-worker: descriptor-safe bounded workspace
  snapshots, Nix-built offline toolchain image, short-lived no-network gVisor
  capsules, read-only result import, cleanup, log/audit limits, action classes,
  and host approval/denial over a private immutable pending request.
- `TFSEC-020` makes a missing disposable worker a high-severity posture finding
  for secure-profile controllers; module and VM tests cover mismatched policy,
  no-egress execution, forbidden/approval-required actions, and cleanup.
- Fixed-size per-agent ext4 workspace volumes with fail-closed empty-workspace
  creation, exact-size drift rejection, checked mount/ownership dependencies,
  VM `ENOSPC` coverage, migration documentation, and posture finding
  `TFSEC-021`.
- Explicit Docker `docker-default` AppArmor selection for secure controllers
  and workers, unconfined Seccomp/AppArmor override rejection, posture findings
  `TFSEC-022`/`TFSEC-023`, and documented Podman evidence limits.
- Optional per-agent Cosign key/keyless verification units that gate OCI
  startup, an all-secure-agents enforcement switch, module tests, and posture
  finding `TFSEC-024` when no signature policy is configured.
- Opt-in systemd hardware-watchdog settings with no assumed device and an
  explicit target-host validation warning.
- Observability textfile metrics and Prometheus rules for broker policy/fetch
  denials, request/token/cost budgets, restart flapping, and root-disk pressure.
- Security-doctor live probes for root-disk pressure, Tailscale Serve/Funnel,
  and Restic freshness, including explicit unknown-evidence warnings and a
  hardened post-success backup timestamp unit.
- Non-interactive live OCI inspection for the core privilege, user, runtime,
  network, capability, root-filesystem, resource, and mount invariants, with
  unavailable evidence kept explicitly unknown.
- Podman live inspection now uses its documented `OCIRuntime`, effective and
  bounding capability sets, AppArmor profile, and compatible HostConfig fields;
  missing version-specific fields remain unknown instead of false-critical.
- The VM integration definition now includes a Podman node that validates its
  generated secure unit, deterministic internal bridge, local image loading,
  `runsc`, non-root/empty capabilities, read-only root, limits, and no egress.
- Security manifests now carry exact configured broker modes/endpoints; the
  doctor distinguishes unreachable evidence from an explicit failed
  credential/policy/audit health response and detects manifest inconsistency.
- VM negative tests now include a separate external attacker node, actual
  `runsc` execution, direct DNS/Internet/private/metadata denials, PID and
  worker-runtime ceilings, sensitive-state invisibility, and a successful
  scoped LLM-broker request from the isolated agent network.
- VM acceptance coverage now also exercises host loopback, the individual
  private/tailnet/link-local/multicast classes, `/usr` immutability, a changed
  Git remote, privileged-request denial, unsafe-doctor exit status, and a
  controlled reboot. Docker live inspection confirms CPU/RAM/swap/PID limits;
  redirect and DNS-rebinding policy uses a deterministic resolver fixture in
  the Rust suite.
- Broker networks now use deterministic bridge interfaces; host INPUT accepts
  broker ports only when interface and source subnet both match, preventing a
  source-address-only allow rule from acting as the authority boundary.

### Fixed
- Contributor E2E now uses a source- and dependency-hash-pinned Dev Containers
  CLI 0.88.0 with the upstream `proxy-from-env` WHATWG URL parsing change
  backported for its CommonJS consumer, removing the legacy `url.parse()` path
  that produced Node.js `DEP0169` security deprecation warnings.
- The installer ISO now embeds its repository below
  `/etc/tentaflake/source`, avoiding a parent-path collision with generated
  runtime manifests such as `/etc/tentaflake/security.tsv`.
- The development shell now uses the canonical `nixfmt` package instead of its
  deprecated `nixfmt-rfc-style` alias, removing the flake evaluation warning.
- The local `just ci` gate no longer stops on existing Statix and Deadnix
  findings; repeated Nix attribute paths are grouped and unused arguments are
  removed without changing their resulting module options.
- Security-manifest generation now records an OCI container with no configured
  user as not non-root instead of passing its `null` user to a string matcher
  and failing evaluation in the `dev` compatibility profile.
- The VM integration test now starts its rebooted node with QEMU reboot support
  enabled and asks systemd to reboot directly instead of relying on a virtual
  key combination. Its headless agent fixtures also omit kmscon, which can
  crash on QEMU's synthetic bochs DRM device, and the stopped-controller
  fixture now declares the broker that its reboot assertion expects to return.
  The controlled-reboot assertion no longer waits indefinitely or checks an
  undeclared boot service.
- The hardened backup-success timestamp service now declares its persistent
  path with systemd `StateDirectory` instead of requiring a nonexistent
  `ReadWritePaths` target before its script could create it. This prevents a
  `226/NAMESPACE` failure after an otherwise successful Restic run, and the VM
  restore and broker-restart probes now report failures within 30 seconds
  instead of waiting for the global 15-minute retry timeout.
- Disposable workers now select backend-compatible bounded log drivers
  (`local` for Docker and `k8s-file` for Podman) instead of sending file
  rotation options to a `journald` default that rejects them. Docker disables
  compression for its single retained local log file, as required by Engine.
- Disposable host snapshots now remain root-owned while source-owner read and
  execute access is mirrored to the declared capsule GID for their read-only
  bind mount. This keeps private ext4 entries such as `lost+found` readable
  without adding `CAP_CHOWN` to the host worker. Capsule copies still discard
  host ownership metadata, and VM worker failures now print service diagnostics
  within 30 seconds instead of waiting for the global 15-minute retry timeout.
  Direct root `approve` and `deny` commands adopt that same declared worker
  group before touching state, so approved jobs do not create root-group
  snapshots that their non-root capsule cannot read.
- Disposable artifact export now happens before the wrapper exits, while its
  bounded workspace tmpfs still exists. Artifacts stream from inside the gVisor
  mount namespace instead of relying on an OCI `cp` view that cannot see that
  tmpfs. A bounded importer rejects unsafe paths, links, and special files; the
  stable completion-marker handshake preserves the real command status without
  adding a writable host bind before releasing and removing the capsule.
- gVisor VM probes now keep a small but startable PID budget. The previous
  limit of 16 charged `runsc` sandbox helpers against the same cgroup and could
  reject sandbox startup before the test workload ran.
- Disposable-worker VM fixtures now request half a CPU, allowing their secure
  capsules to run inside the integration test's single-vCPU guest while the
  production default remains unchanged.
- Path-activated disposable workers no longer use systemd's aggregate service
  start counter, which also counted successful oneshot queue drains and disabled
  a healthy worker after five ordinary jobs. Infrastructure failures retain the
  ten-second restart delay, while per-job resource and timeout limits remain.
- Broker health and authentication now recognize systemd's read-only named-ACL
  access for `LoadCredential` files used with `DynamicUser`. The ACL mask is
  accepted only for exact files below `$CREDENTIALS_DIRECTORY`; ordinary
  group-readable files and all other-readable files remain rejected.
- VM broker-readiness checks now fail after 30 seconds with the raw health
  response, route, listener, firewall chains, unit status, and journal instead
  of waiting for the test driver's 15-minute timeout.
- Disposable worker units now use a statically declared host group matching
  the container GID, preserve tmpfiles' setgid result mode without a blocked
  `chmod`, and inspect NixOS' generated OCI start script when testing secure
  runtime flags.
- VM secure-unit assertions now inspect NixOS' generated OCI start scripts for
  both Docker and Podman instead of expecting runtime flags in wrapper units.
- Disposable workers now retain `openat2` as the primary no-symlink workspace
  opener and use a descriptor-relative, component-by-component `openat`
  fallback only when the kernel reports `ENOSYS`. Relative paths, dot/parent
  traversal, symlinks, and non-directory components remain rejected.
- VM broker readiness assertions now retry the actual health response instead
  of treating systemd's `Type=simple` process start as socket readiness.
- Quota-backed workspaces now mount after ordinary local filesystems without a
  `local-fs.target`/`basic.target` ordering cycle; worker path watches start
  only after mounted ownership and control-directory setup completes.
- Broker health probes no longer append audit events; startup proves audit
  writability once, later readiness checks reject symlinks and sync without
  allowing health polling to churn the bounded audit history. LLM input-token
  budgets now reserve the conservative request-byte upper bound, and bridge
  interface hash collisions fail evaluation.
- Secure controller restart/dependency overrides now merge into the actual
  OCI-generated systemd service attribute instead of a parallel key ending in
  `.service`; broker and workspace-mount ordering is therefore present in the
  rendered container unit.
- Per-agent Phase B brokered egress: dedicated internal OCI networks, disabled
  container DNS/IPv6, subnet-scoped host INPUT/FORWARD firewall rules, and a
  random runtime-only virtual credential per agent.
- A Rust LLM credential broker with exact route/model allowlists, completion
  clamps, concurrency/rate/daily token/cost budgets, provider credential
  substitution through systemd `LoadCredential`, DNS pinning, no redirects or
  environment proxies, and prompt-free JSONL audit.
- A Rust SSRF-safe fetch broker with exact HTTPS host/media allowlists, public
  IP enforcement, redirect revalidation, DNS-rebinding defense, bounded
  responses, mode-0700 quarantine, and explicit untrusted-content envelopes.
- Bounded systemd restart policy and credential/audit readiness endpoints for
  brokers, plus a combined `tentaflake stop` transaction that stops an agent's
  container and all loaded broker endpoints.
- An opt-in encrypted Restic backup module with explicit state/audit paths,
  runtime-only repository/password files, retention, integrity checks, and a
  VM backup/restore drill.
- A loopback-only observability profile with Prometheus, Grafana, Loki, Alloy,
  systemd credential loading, and provisioned data sources.
- A separate Falco modern-eBPF runtime-detection profile. Consumers must supply
  a reviewed pinned Falco package because the pinned nixpkgs has none.
- Direct module-evaluation coverage for core, observability, and Falco profiles.
- One post-merge container security policy shared by Hermes and ZeroClaw:
  explicit non-root user, no privileges/capabilities/ports/devices,
  read-only root, bounded hardened tmpfs, CPU/RAM/swap/PID/ulimit controls,
  sensitive-mount and secret-like-environment rejection, and declarative
  gVisor `runsc` selection without a silent runc fallback.
- `tentaflake doctor --security` with stable machine-readable `TFSEC-*`
  findings and a Nix-generated desired-state manifest.
- Secure profiles require the private Tailscale management path, reject the
  public OpenSSH module, and advertise the documented `tag:agent-host` tag.
- Exact canonical GitHub remote and branch allowlists for host-side
  `gitAutoPush`, repository-scoped `safe.directory`, fail-closed errors, and
  malicious-URL regression tests.

### Previous development notes
- `tentaflake status` now inventories every non-empty physical disk instead of showing only the root filesystem. A mounted disk is reported once using `/` or its first data-like mount (excluding `/boot` and `/nix/store`); an unmounted disk remains visible by device name and raw capacity. Loop, RAM, zram, and zero-byte card-reader devices are excluded. The screenshot-safe `--hide` view keeps this non-identifying capacity telemetry, and `tentaflake-status --selftest` covers root, mounted data, unmounted, and excluded-device cases with synthetic commands.
- `tentaflake health` — host vitals dashboard, the machine-level counterpart to `stats`. Same renderer again (logo, header, bars, colour thresholds), with a **VITALS** section — 24-cell bars for CPU busy (a 0.4 s `/proc/stat` delta, 100% = every core busy), memory, swap, the hottest thermal zone in °C, and every block-device filesystem — and a **CHECKS** section covering failed systemd units, agent states and Tailscale. The header carries a `● healthy` / `▲ degraded` / `✗ critical` verdict, the worst finding on the same thresholds the bars are painted with. `--live` (default 2 s, `--live=5` to slow it) redraws in place on the alternate screen, so Ctrl-C leaves the scrollback untouched and colours survive — unlike `watch`. It always exits 0: `tentaflake doctor` remains the scriptable check with the nonzero exit. `--hide` applies here too and prints a *count* of failed units instead of their names (a unit name carries its container's); `tentaflake-status --selftest` now renders the masked health view alongside the masked wide view and fails if either leaks. `./scripts/banner-test.sh --health` previews and regression-checks it on a dev box.
- `tentaflake stats` — fleet dashboard sharing the login banner's renderer (same logo, header and rows, so the two cannot drift), one density wider: per-agent CPU%, PID count, memory against the container limit, and a fleet total against host RAM. CPU is a 0.4 s cgroup delta on the `docker stats` scale (100% = one busy core); for a live view, `watch -cn2 tentaflake stats`.
- Per-agent memory column in the login banner. Read straight from each container's cgroup (`memory.current - inactive_file`, the exact number `docker stats` reports) instead of `docker stats`' ~2 s sample window, so login stays fast. Known docker and podman cgroup layouts are probed; an agent without a readable cgroup simply gets no cell.
- `--hide` / `-H` on `status` and `stats`: masks host name, tailnet IP and agent names (`agent-1`…`agent-N`, the failed-agents hint included) for screenshot-safe output, with a yellow `· redacted` header marker. Telemetry stays. `tentaflake-status --selftest` renders the masked wide view and exits nonzero if any identifying string leaks.
- DCO enforcement for pull requests: every non-merge commit must include a `Signed-off-by:` line; see `DCO.txt` and `CONTRIBUTING.md`.
- `nix develop` now greets you: the dev shell prints the tentaflake logo with branch/HEAD/working-tree/Nix-version rows and a short `just` cheat sheet, and sets a `(tentaflake)` prompt marker. Same braille art and layout as the host login banner — both read `public/tentaflake-shell-logo.txt`. The banner is skipped when stdout is not a TTY (`nix develop --command …` in CI stays clean) and `TENTAFLAKE_NO_BANNER=1` disables it entirely. The dev shell moved from an inline `pkgs.mkShell` in `flake.nix` to `lib/devshell.nix`; the package set is unchanged, and the banner script is built with `writeShellApplication`, so `nix flake check` shellchecks it.

## [0.3.1] — 2026-07-19

### ⚠ Breaking
- Default hostname and admin user renamed: `agent-host` → **`tentaflake`**, `admin` → **`user`** (`lib/constants.nix`). The built-in host's flake attribute follows the hostname (it is now written `nixosConfigurations.${hostName}`, so the two cannot drift), which means **`nix build .#nixosConfigurations.agent-host` and `nixos-rebuild --flake .#agent-host` no longer resolve** — use `.#tentaflake`. Same for the check attr `checks.<system>.agent-host` → `checks.<system>.tentaflake`. Installed machines are unaffected: the installer generates `nixosConfigurations.${uc.hostName}` from the hostname you typed, so `tentaflake rebuild` keeps working. The installer's suggested defaults also change (`agent-machine` → `tentaflake`, `agent` → `user`); both remain editable in the wizard.
- The fleet-generic infrastructure drops its `hermes-` branding (#52); "Hermes" now only names the Hermes agent runtime. What renames, and what bridges the transition for one release:
  - **systemd units**: `hermes-auditd` → `tentaflake-auditd`, and on the live ISO `hermes-env-detect`/`hermes-data-mount` → `tentaflake-env-detect`/`tentaflake-data-mount`. The old unit names are **gone** — update `systemctl`/`journalctl` scripts. (The console unit was already `tentaflake-console`.)
  - **module options**: `tentaflake.hermes-auditd.*` → `tentaflake.auditd.*`. Old names still evaluate with a deprecation warning via `lib.mkRenamedOptionModule` and will be removed in a future release.
  - **flake package attr**: `packages.hermes-auditd` → `packages.tentaflake-auditd`. The old attr remains as a deprecated alias to the same derivation.
  - **binaries**: `hermes-auditd` → `tentaflake-auditd`, `hermes-top` → `tentaflake-top` (`tentaflake top` now execs the new name). The package installs a deprecated `hermes-top` symlink for one release.
  - **USB labels** (live ISO unattended boot; the interactive `tentaflake agent add` needs no label at all): `TENTAFLAKE_ENV`/`TENTAFLAKE_DATA` are the primary labels; the legacy `HERMES_ENV`/`HERMES_DATA` labels are still accepted, so existing sticks keep working.
  - **live-ISO env dir**: `/run/hermes` → `/run/tentaflake`, with a `/run/hermes` compat symlink for configs that hardcode the old `envFile` path.
  - **Go module** (fork-relevant only): `tentaflake/hermes-auditd` → `tentaflake/tentaflake-auditd`; `internal/hermes` → `internal/event`.
  - **NOT renamed** (deliberately, so existing deployments keep their audit history): the audit DB state dir `/var/lib/hermes-audit`, the default `dbPath` `/var/lib/hermes-audit/events.db`, and the `hermes-audit` user/group. These rename in a future major with a migration.
  - Also: the `hermes-<name>`/`zeroclaw-<name>` container/user/state-dir prefixes, the `HERMES_*` env contract, and `mkHermesAgent` are unchanged — they name the Hermes runtime, not the infrastructure.
- Digest pinning is **enforced**, not merely defaulted (`lib/pinnedImage.nix`): a previously-valid `image = "repo:tag"` is now a hard eval `throw` in **all three** builders (`mkHermesAgent`, `mkZeroClawAgent`, `mkOpenCodeAgent`). Any fork or consumer flake that overrides `image` with a tag **fails `nixos-rebuild` outright**, before a single derivation is built. Repin to `repo@sha256:<digest>` (the error message prints the `skopeo inspect` command that reports it), or set `allowMutableImage = true` on that agent to acknowledge giving up reproducibility. Note that `repo:tag@sha256:...` is rejected too — keep the version in a comment, not in the reference. See Security below for the full rationale.
- The physical console is rendered by **kmscon on TTY1 by default** on installed systems (`tentaflake.modernConsole.enable`, default `true`). On hardware where kmscon cannot grab the framebuffer the operator **loses the physical console** — there is no local login left to fix it from, only Tailscale SSH or a rescue boot. On unfamiliar hardware, check the console on the first rebuild while remote access is still up; `tentaflake.modernConsole.enable = false;` falls back to the legacy VT. Both ISOs are unaffected — they keep the VT deliberately. See Added below.
- The live agent ISO's debug sshd no longer accepts password auth (`PasswordAuthentication = false`). Logging into a booted live ISO with the live user's password stops working — use Tailscale SSH, or the key-only `modules/ssh.nix` with `tentaflake.adminAuthorizedKeys`. See Changed below.
- The host Nix daemon is locked down (`modules/nix-settings.nix`): `allowed-users` is now `root` and `@wheel` **only**, so any non-wheel user or service account that used to run `nix build`/`nix-store` against the host daemon is refused outright; and `sandbox-fallback = false` turns a build that cannot be sandboxed into a hard failure instead of a silent unsandboxed one. See Changed below.

### Security
- `tentaflake-auditd` hardening (#8): the audit DB now sets `secure_delete` and is capped at 10000 pages (~40 MB) so an agent flooding filesystem events cannot fill the host disk; inotify directory watches are capped at 10000 per daemon (with a one-time warning when hit); the console clamps `?limit=` to 1000 and every store read is bounded by a 5 s query timeout. (No `fs.inotify.max_user_watches` sysctl needed — nixpkgs already defaults it to 524288.)
- Default Hermes, ZeroClaw and OpenCode container images, plus the documented Nix build image, are pinned to exact multi-platform OCI manifest digests (`lib/constants.nix`). Identical configurations no longer resolve mutable tags to different image contents across machines or over time. Refresh the pins with `./scripts/update-agent-images.sh` (it prints upstream digests for all three runtimes; the bump stays a manual edit).
- Digest pinning is now **enforced**, not just defaulted (`lib/pinnedImage.nix`). `mkHermesAgent`, `mkZeroClawAgent` and `mkOpenCodeAgent` reject an unpinned `image` at eval time, so a fork cannot silently switch back to a mutable tag; the check runs *after* the `extraContainerConfig` merge, which could otherwise override `image` past it. Locally-built images opt out per-agent with `allowMutableImage = true`. Covered by the new `image-pinning` flake check.
- Image references are also character-whitelisted, **including** under `allowMutableImage`. nixpkgs' `oci-containers` module splices `container.image` into the container unit's shell command without `escapeShellArg` (unlike every neighbouring element), so a reference carrying whitespace or a shell metacharacter could inject extra `docker run` flags — or a command substitution running as root — behind a digest-shaped suffix.
- Pinned references are written `registry/repository@sha256:digest`, never `repository:tag@sha256:digest`. The docker CLI accepts the tag-plus-digest form but podman and skopeo reject it outright ("Docker references with both a tag and digest are currently not supported"), which would have broken every `tentaflake.containerBackend = "podman"` host.
- CI workflow hardening (`.github/workflows/check.yml`): a top-level `permissions: contents: read`, and all five third-party actions pinned to full 40-character commit SHAs with a trailing version comment. `GITHUB_TOKEN` is otherwise write-capable on pushes while every step in this workflow only reads the repo, and a tag like `@v4` is re-pointable by whoever owns the action — an unpinned action is arbitrary code execution in a job that holds the repo token. `golangci-lint-action` moves v6 → v9.3.0 in the same pass rather than being pinned where it was: v6 only drives golangci-lint v1, and this repo's `.golangci.yaml` is the v2 format, so pinning v6 would have failed outright. The linter itself is pinned to `v2.12.2` instead of `latest`, so an upstream lint release can no longer break CI on an unrelated PR. (v9 additionally validates `.golangci.yaml` against its JSONSchema, which v6 skipped — worth watching on the first run.)

### Added
- `mkOpenCodeAgent` — a third agent runtime builder ([OpenCode](https://opencode.ai)) alongside `mkHermesAgent`/`mkZeroClawAgent`. Runs `opencode serve` (headless HTTP, documented OpenAPI 3.1) in the same isolation contract (own container/uid/`0700` state dir + `/workspace`, secrets via `--env-file`), forwarding a loopback `hostPort` to the gateway with an optional `tailscale serve` on `servePort`. Ideal for external orchestrators (n8n, CI): `POST /session` then `POST /session/<id>/message`. It joins the other two runtimes' safety contract: the image defaults to a digest pin in `lib/constants.nix` and goes through the same enforced `pinnedImage` check (mutable tags are an eval error; `allowMutableImage = true` opts a locally-built image out). Setting `servePort` now **requires** an `envFile` or `agenixFile` — the container is started with `--hostname 0.0.0.0` so the gateway is reachable across the container boundary, and `opencode serve` only turns on HTTP basic auth when `OPENCODE_SERVER_PASSWORD` is set (username defaults to `opencode`), so publishing without one put an unauthenticated agent on the tailnet; the assertion checks the env file is wired up, and `allowUnauthenticatedServe = true` is the explicit acknowledgement. Because Nix cannot read a runtime/agenix file at eval time, the serve unit re-checks at start: an `ExecStartPre` greps the env file(s) for a non-empty `OPENCODE_SERVER_PASSWORD` and refuses to publish without one — so the documented proxy-key pattern (an env file carrying only an LLM key) fails closed instead of satisfying the assertion and publishing anyway. It also runs `tailscale serve --https=<servePort> off` on that path, because `tailscale serve --bg` state is persisted by tailscaled and restored on boot: dropping the password de-publishes the agent rather than leaving a stale mapping in front of a now-unauthenticated session API. The password must live in `envFile`/`agenixFile` and **not** `extraEnvironment` — that attrset is rendered into the container unit's start script in the world-readable Nix store. The `tailscale serve` unit also gained an `ExecStop` (`tailscale serve --https=<servePort> off`), so removing or stopping an agent tears its tailnet mapping down instead of leaving a dangling public endpoint pointed at a dead port. Credentials via a local LLM proxy (`settings` base_url + env-file key, host-portable) or an opt-in read-only `authFile`. Exported from `lib.<system>.mkOpenCodeAgent` and passed to `my-agents.nix` (older files taking only the previous builders keep working via `intersectAttrs`). Wired into the same multi-runtime plumbing as the other runtimes: the `tentaflake` CLI/status banner discovery, the `tentaflake-auditd` state-dir watcher (Go `agentNameFromPath` + NixOS auto-discovery), and the installer's env-placeholder/USB-persistence loops all recognize the `opencode-` prefix; the VM integration test now declares an OpenCode agent and asserts its unit + `0700` state dir. See `docs/08-opencode.md` and `my-agents.nix.example`.
- Modern physical console (`tentaflake.modernConsole.enable`, on by default): the installed system renders TTY1 with **kmscon** (KMS/DRI + pango) instead of the legacy Linux VT, so the console gets real TTF fonts and full Unicode. A Linux VT maps at most 512 glyphs and the kernel's built-in font ships 256 CP437-era ones, so on the machine's own screen the braille logo in `tentaflake-status` and `btop`'s graphs rendered as boxes — no `console.font` could fix that, the ceiling is the VT. Default font is Cascadia Mono (one of the few modern monospace fonts covering U+2800–28FF; JetBrains Mono, Hack and Fira Mono do not), size via `tentaflake.modernConsole.fontSize` (default 14). The keymap follows `tentaflake.consoleKeyMap` through xkb, since kmscon replaces getty. Disable it on hardware where kmscon cannot grab the framebuffer — the legacy VT then uses the new `tentaflake.consoleFont` (Terminus `ter-v16n` by default, still far better than the kernel font for box drawing). `consoleFont` applies *only* when kmscon is off, and `null` skips the boot-time `setfont` entirely; both ISOs use `null`, because their `dialog` TUIs draw in ASCII anyway and the fbcon reconfiguration flickers on some Intel panels (`pipe A FIFO underrun`). Both **ISOs** deliberately keep the legacy VT: kmscon hands the login a pty and owns the VT in graphics mode, which would defeat the `[ "$(tty)" = /dev/tty1 ]` guard that auto-launches `installer.sh` and the live ISO's firstboot wizard (the latter also writes straight to `/dev/tty1`). See `docs/06-shell.md#physical-console-kmscon`.
- `tentaflake agent add|list|set-model|remove` — an interactive CLI wizard for declaring Hermes and ZeroClaw agents without writing Nix. Config lands in a flat, non-secret, git-tracked `agents.json` (schema v1: name/provider/model/base_url/ports/envFile path) read by `lib/agentsFromData.nix`, additive to a hand-written `my-agents.nix`. API keys are read with `read -rs` (or imported from a USB stick, see below) and written only to a root-owned `0600` file at `/var/lib/tentaflake/secrets/<runtime>-<name>.env` — never into `agents.json` or the Nix store. See `docs/08-agent-cli.md`.
- **USB key import in `tentaflake agent add`** — nobody wants to type a 73-character OpenRouter key. The wizard searches already-mounted media (`/run/media/*/*`, `/media/*`, `/mnt/*`) and, only if that finds no key, unmounted *removable* partitions which it mounts itself `ro,nosuid,nodev,noexec`; it looks two directories deep for `.env` files carrying the provider's `KEY=` line and for `.txt`/`.key` files holding a bare token. **No filesystem label and no filename convention are required** — the `TENTAFLAKE_ENV` label is unchanged and undeprecated, it stays the deterministic marker for the live ISO's unattended boot and here merely ranks a volume first. Findings are presented with a masked preview (first 6 + last 4 + length) and their source volume/file, never silently consumed; "type or paste it myself" is always available and the wizard works with no stick at all. Files on removable media are parsed with a regex, **never `source`d, `eval`d or executed**, and every mount is released on abort, error or Ctrl-C. Typed keys now also get a masked echo-back and a retry, so a truncated paste is visible.
- **Pinned logo during `tentaflake agent add`** — the logo stays at the top of the terminal for the whole wizard using the terminal's own scroll region (DECSTBM `CSI top;bottom r`), no curses and no new dependency; `tput` is unusable here because the installed console is kmscon and `TERM=kmscon` has no terminfo entry, so geometry comes from `stty size`. Restored via an `EXIT`/`INT`/`TERM`/`HUP` trap. Degrades to plain sequential output when not on a tty, under `NO_COLOR`, or on `TERM=dumb`; a window under 24×60 (or a legacy Linux VT, which has no braille glyphs) gets a compact two-row header instead, down to 12 rows × 40 columns. Colour is now gated globally on the tty / `NO_COLOR` / `TERM=dumb` conditions, independent of window size, so all `tentaflake` output is clean when piped.
- NixOS VM integration test (`checks.x86_64-linux.vm-integration`, `tests/integration.nix`): boots a VM built from `nixosModules.default` and asserts the runtime path — the `tentaflake` CLI runs (`help`, `ps`), the status banner renders and names the host, `tentaflake-auditd` comes up and creates its SQLite event DB, and each declared agent produces its systemd unit and `0700` state dir (the Hermes agent also its system user; the OpenCode agent runs as an anonymous uid, so it has none, and gets its `/workspace` checked instead). The daemon subtest waits for `/var/lib/hermes-audit/events.db` rather than a port, because `tentaflake-auditd` **opens no socket at all** — it writes events to SQLite and the separate `tentaflake-console` service is the HTTP surface — and `Type=simple` marks the unit active before the binary has run, so `wait_for_unit` alone would race the DB creation. Agents are declared `autoStart = false` so the sandboxed, offline VM never tries to pull an image: the test proves each unit is *defined*, not that a container runs. The VM node pins its own `hostName`/`adminUser` (`agent-host`/`admin`) instead of inheriting the template defaults, so the banner assertion is unaffected by the `agent-host` → `tentaflake` rename above; it also sets `nixSettings.enable = false`, since that module defines `nixpkgs.config.allowUnfree` and the test harness declares `nixpkgs.config` read-only. Runs as part of `nix flake check` and CI. Previously CI only proved the config *built*; nothing exercised runtime behavior.
- CI now runs `golangci-lint` on `pkgs/tentaflake-auditd` (`.github/workflows/check.yml`), enforcing the lint standard the contributor docs already required.
- `tentaflake` CLI host-management subcommands (#51): `rebuild` (nixos-rebuild switch on the system flake — same command as the `rebuild` alias), `update` (`nix flake update` on `/etc/nixos`, shows the `flake.lock` diff, asks y/N, then rebuilds), `doctor` (deep health check — failed systemd units, root disk ≥90%, Tailscale, `tentaflake-auditd`/`tentaflake-console` service state when enabled, per-agent unit state — every problem paired with its exact fix command; nonzero exit when problems found), `console` (Agent Console URL + the `tailscale serve` publish one-liner, or how to enable the console when it's off), and `backup <name>` (one-shot `sudo tar` snapshot of the agent's state dir to `./tentaflake-<name>-<UTC timestamp>.tar.gz`, with an active-agent consistency warning and the matching restore one-liner). Agent records now carry each agent's state dir, derived from the container's first volume mount (so custom `stateDir`s are honored).
- Opt-in egress filtering (`tentaflake.networking.egress.enable`, **default off**): an nftables output chain (table `tentaflake-egress`, policy drop) allowing loopback, established/related, ICMP/ICMPv6, and configurable outbound ports (`allowedTCPPorts` default `[443]`, `allowedUDPPorts` default `[53 67 123 547 41641]` — DNS, DHCP, NTP, DHCPv6, tailscale). Because agent containers use host networking, the host rules cover the whole fleet. (#9, [docs](https://github.com/timfewi/tentaflake/blob/v0.3.1/docs/07-operations.md#egress-filtering-opt-in))
- `modules/ssh.nix` — opt-in hardened OpenSSH server (`tentaflake.ssh.enable`, default off): key-only auth (no passwords, no keyboard-interactive, no root login, max 3 auth tries), fail2ban, and TCP 22 opened in the otherwise deny-all firewall. Tailscale SSH remains the primary access path; admin keys come from `tentaflake.adminAuthorizedKeys`.
- CI security workflows: CodeQL analysis of the Go code (`.github/workflows/codeql.yml`, push/PR to main + weekly), gitleaks full-history secret scanning (`.github/workflows/gitleaks.yml`), and a weekly `flake.lock` update PR via DeterminateSystems/update-flake-lock (`.github/workflows/update-flake-lock.yml`).
- `.pre-commit-config.yaml` — optional local hooks mirroring the CI gates (gitleaks, shellcheck, gofmt, `go vet`, `nix fmt`); setup documented in CONTRIBUTING.md.
- `.agents/skills/tentaflake-change-review/` — a bundled skill carrying this repo's change-review discipline, alongside `tentaflake-repo-guidance`. It borrows one idea from the heavyweight AI-engineering frameworks — traceability: every change ties back to a stated reason (issue/requirement) and forward to a verification step, with doc sync before it counts as done — and strips the ceremony. Three gates, no lifecycle, no review matrices; sized for a ~30-file NixOS template maintained by Conventional Commits and PRs, and explicitly skipped for formatting, typo and comment-only edits. Its target is orphaned changes (code with no stated reason), silent assumptions and doc drift.
- `scripts/banner-test.sh` (also `just banner`) — renders the `tentaflake-status` banner with a stubbed `systemctl` and a fake mixed-runtime fleet (active/inactive/failed) so the banner can be previewed and regression-checked on any dev machine; self-checks cover fleet counters, duration formatting, logo loading, and logo/info-column alignment. The wizard's own logic is covered by the hidden `tentaflake agent __selftest` (env/bare key parsing including a "must not execute" fixture, masking, header geometry and degradation, scroll-region teardown).

### Changed
- Live agent ISO: the debug sshd no longer accepts password auth (`PasswordAuthentication = false`) — Tailscale SSH covers remote debugging on the live ISO.
- `pidsLimit` parameter on `mkHermesAgent` and `mkZeroClawAgent` — renders `--pids-limit=<n>` on the agent container (default `512`, generous because agents compile code; `null` skips the flag). Hermes containers additionally gain `--security-opt=no-new-privileges:true` by default (ZeroClaw containers already had it). Opt-in `--cap-drop=ALL` recipe and the docker-group-is-root-equivalent tradeoff (podman as the rootless alternative) documented in `docs/02-agent-tips.md` and `docs/07-operations.md`. (#3)
- Kernel hardening (#4): `hardening.nix` gains sysctls (kexec/sysrq off, BPF JIT hardening, TCP RFC 1337, ICMP broadcast/bogus-error ignore, loose rp_filter — Docker-compatible, martian logging, ARP ignore/announce) , `boot.kernelParams` (`slab_nomerge`, `init_on_alloc/free`, `pti=on`, `vsyscall=none`, `debugfs=off`, `randomize_kstack_offset=on`), and an explicit LSM order via `security.lsm` (`landlock,yama,apparmor,bpf`); `boot.nix` disables the systemd-boot menu editor (blocks `init=/bin/sh`).
- Git hygiene: `.gitignore` now covers `age.key` and `*.log`; `CONTRIBUTING.md` documents SSH commit/tag signing and signed release tags (`git tag -s`). (#10)
- `docs/04-agenix-secrets.md`: "Key Rotation & Recovery" section — rotating secret values, rekeying out a compromised recipient (old ciphertext in Git history stays exposed), recovering from a lost host SSH key via any surviving recipient, and an example rotation cadence for forks. (#6)
- `SECURITY.md`: operator **incident response** runbook — isolate the container, revoke provider keys, inspect the `tentaflake-auditd` audit trail (UTC timestamps), preserve the state dir, rotate secrets and rebuild. (#11)
- `docs/07-operations.md`: **Backup & restore** (state dirs, audit DB via `sqlite3 .backup` — never `cp` a live WAL db — off-host age identity, restic example for forks) and **Log forwarding** (`services.journald.upload` snippet) sections. (#11)
- `modules/piper-tts-server.nix`: systemd unit hardened to parity with `tentaflake-auditd`/`hive-research` (`ProtectSystem=strict`, `ProtectHome`, `PrivateDevices`, `RestrictNamespaces`, `LockPersonality`, empty capability bounding set, `RestrictAddressFamilies`, `SystemCallFilter=@system-service`) plus resource limits: `TasksMax=64` and a new `services.piper-tts-server.memoryMax` option (default `2G` — raise for larger voice models). (#12)
- `modules/nix-settings.nix`: Nix daemon hardening — `allowed-users` restricted to `root` and `@wheel` (agents live in containers and never talk to the host daemon), explicit `sandbox = true` with `sandbox-fallback = false` (no silent downgrade to unsandboxed builds), and `min-free`/`max-free` (2 GiB / 8 GiB) so builds cannot fill the disk. (#5)
- Live-ISO firstboot env detection genericized (#52): agent env-file placeholders are now created for `docker-`/`podman-` units of **both** runtimes (`hermes-*` and `zeroclaw-*`, previously `hermes-*` only), and the USB data-mount persists `/var/lib/zeroclaw-*` state dirs too. Hermes env files keep their bare names (`coding.env`); other runtimes keep the runtime prefix (`zeroclaw-scout.env`).
- `.golangci.yaml` migrated to the golangci-lint v2 config format — the v1 file no longer loaded under golangci-lint 2.x, so lint was silently broken; the errcheck findings that had accumulated unseen are fixed.
- `tentaflake-status` login banner redesigned: braille-art octopus-snowflake logo in cyan (embedded at build time from `public/tentaflake-shell-logo.txt`, the single source of truth) with the header and host facts rendered as a column to its right, and the container backend in the tagline; `AGENTS` header now cyan with a fleet count (`total · active · inactive`, plus `failed` in red when present); each runtime gets its own color (hermes yellow, zeroclaw blue, other magenta) on dot/runtime/status; agents sorted by name; active agents show their uptime (`active 2d 4h`); inactive agents render dimmed; a failed agent adds a red `⚠ failed: <name> — tentaflake logs <name>` hint. Also: memory/disk lines gain usage-colored percentages (green/yellow ≥75%/red ≥90%), host uptime is read from `/proc/uptime` (fixes the duplicated load average in the old fallback), and a separator rule divides host facts from the agent list.

### Fixed
- Installer: adding the **first agent** on an installed machine broke `tentaflake rebuild` with `error: infinite recursion encountered`. The flake `installer.sh` generates for the target passed only `mkHermesAgent` and `constants` in `specialArgs`, but `configuration.nix` also takes `mkZeroClawAgent` and `agentsFromData` — and consumes them inside `imports`. A missing helper there does not fail loudly: Nix falls back to `config._module.args`, which needs `config`, which needs `imports`, so the rebuild dies in recursion instead of naming the absent argument. It stayed hidden on a fresh install because `lib.optionals (builtins.pathExists ./agents.json)` leaves `agentsFromData` unforced until an `agents.json` exists — so the very first `tentaflake agent add` triggered it. The generated flake now splats the whole `lib/default.nix` helper set, so it cannot drift from what `configuration.nix` asks for again. Covered by the new `scripts/generated-flake-test.sh`. **Machines already installed need a one-line manual patch** — see [docs/08-agent-cli.md](https://github.com/timfewi/tentaflake/blob/v0.3.1/docs/08-agent-cli.md#infinite-recursion-encountered-on-the-first-rebuild-after-agent-add).
- Installer: `nixos-install` failed evaluating the target flake with a `readFile` error — the redesigned `tentaflake-status` banner reads `public/tentaflake-shell-logo.txt` from `modules/shell.nix`, but the installer never copied `public/` into `/mnt/etc/nixos`. The installer now copies the logo file (just that file, not the multi-MB imagery in `public/`).
- USB data persistence (`tentaflake-data-mount`) and the env-file placeholder loop (`tentaflake-env-detect`) never ran at all. Neither unit declared a `path`, so both got only the default service PATH (coreutils, findutils, gnugrep, gnused, systemd) — no `blkid`, `mount`/`umount` or `awk`. `blkid` exiting 127 was swallowed by the surrounding `|| true`, leaving `USB_DEV` empty and tripping the `[ -z "$USB_DEV" ] && exit 0` guard on every boot, so a `TENTAFLAKE_DATA` stick was silently ignored and agents never got their placeholder `.env` files (which is what made `docker run --env-file` fail on a cold boot). Both units now declare `path = [ pkgs.util-linux pkgs.gawk ]`.
- USB data persistence could **destroy agent state** (`installer/firstboot.nix`). `tentaflake-data-mount` used to `mkdir -p` the target on the USB stick, then `rm -rf` the real `/var/lib/<runtime>-<name>` and symlink it — with nothing verifying that the copy had landed or that the target filesystem could even hold Unix ownership. The target was created root-owned `0755`, so an agent running as its own uid could no longer write its own `0700` state dir; on the vfat/exfat stick most people hand-make, "state dir is 0700 and owned by the container uid" quietly evaporated altogether. Any non-empty target was treated as authoritative, so resuming after a partially-copied stick replaced a complete local state dir with a truncated one and reported success. And on a full or read-only stick the `rm -rf` ran regardless, so the agent's state was simply gone. Persistence now goes through a `stage_to_usb()` helper that mirrors the original uid/gid/mode onto the USB directory and re-`stat`s to confirm it stuck (a filesystem that cannot hold Unix ownership is detected and skipped), then branches on the state that is actually present:
  - **USB target empty** — seed it from the local dir, verify by comparing entry counts, and *only then* remove the original and symlink.
  - **Local dir empty** — the USB copy wins and is symlinked in. This is the ordinary reboot path: tmpfiles recreated an empty state dir and the real state is on the stick, so there is nothing to lose.
  - **Both sides hold state** — **refuse**. Copying either way discards the other and nothing can know which the operator wants, so it logs `… both hold state — refusing to discard either; resolve by hand` and touches neither side. Resolving a conflict is never done by discarding.

  A failure leaves the original directory fully intact and logs `WARNING: could not persist … — keeping local state, no USB persistence`. The one path that cannot honestly claim that says so instead: if the `rm -rf` succeeded but the symlink did not, the state is on the stick, and the message is `ERROR: … removed but symlink … failed — state is on the USB, relink by hand`. The helper's exit status is captured rather than left to `errexit`, so one unpersistable agent no longer aborts the whole loop, and it sits inside the existing loop, so all three runtimes (`hermes-*`, `zeroclaw-*`, `opencode-*`) are covered at once. Covered by `scripts/data-mount-test.sh`.
- `modules/piper-tts-server.nix`: every `POST /v1/audio/speech` returned `500 piper binary not found` while `/health` and `/v1/models` kept answering, so the service looked healthy and only synthesis was dead. The server shells out to `piper` via `subprocess.run`, but `/run/current-system/sw/bin` — where `environment.systemPackages` lands — is not on a systemd unit's default PATH. The unit now declares `path = [ pkgs.piper-tts ]`. Pre-existing since the module landed in 0.2.0, not caused by this release's sandbox hardening; found while reviewing it.
- `tentaflake` re-enabled terminal echo on exit even when it had never disabled it. `tf_cleanup` runs from an `EXIT` trap on every invocation — including `tentaflake help` — and called `stty echo` unconditionally, corrupting any caller on a `-echo` terminal (serial console, scripted pty, the NixOS test driver's backdoor console, where it desynced the driver outright). Echo is now restored only when a masked `read -rs` actually turned it off.

## [0.2.0] — 2026-07-11

### ⚠ Breaking
- The host operator CLI is now **`tentaflake`**. The `hermes` host command remains in this release as a deprecated shim (prints a warning to stderr, execs `tentaflake`) and **will be removed in a future release** — update scripts and habits. (The `hermes` CLI *inside* Hermes agent containers is unrelated and unchanged.)
- `tentaflake.shell.hermesCli.enable` → `tentaflake.shell.tentaflakeCli.enable`. The old option still evaluates via `lib.mkRenamedOptionModule` with a deprecation warning and will be removed together with the shim.
- `my-agents.nix` is now expected to accept `{ mkHermesAgent, mkZeroClawAgent }`. Inside this template, `configuration.nix` only passes the arguments your file declares, so old `{ mkHermesAgent }:`-only files keep evaluating — but flake-input consumers that call `import ./my-agents.nix { inherit mkHermesAgent mkZeroClawAgent; }` directly must update old files to the new signature (or add `...`).
- `tentaflake ps` behaves differently from the old `hermes ps`: it lists agent containers of **all runtimes including stopped ones** (`--all` + anchored name filters) instead of `--filter name=hermes-` on running containers only.
- Agent Console / `tentaflake top` label non-Hermes agents with a runtime prefix (`zeroclaw-<name>`). Hermes agents keep their bare labels, so existing audit DBs and dashboards are unaffected.

### Added
- `lib/mkZeroClawAgent.nix` — second agent runtime alongside Hermes: OCI container (`ghcr.io/zeroclaw-labs/zeroclaw`), TOML `settings` attrset generated into a read-only `config.toml`, a `tailscale serve` unit for tailnet HTTPS access, and an optional `seedDir` copied into the state dir on first boot. State dir `/var/lib/zeroclaw-<name>`, secrets via `agenixFile` (`--env-file`).
- `zeroclaw.env.example` — mirrors `hermes.env.example`; documents the `ZEROCLAW_<section>__<sub>__<key>` env-var convention ZeroClaw uses for config overrides, with an OpenRouter `api_key` example.
- `my-agents.nix.example` now takes `{ mkHermesAgent, mkZeroClawAgent }` and defines both a `hermesAgents` list and a `zeroclawAgents` list (`map mkHermesAgent hermesAgents ++ map mkZeroClawAgent zeroclawAgents`), with a fully-commented generic ZeroClaw reference agent (schema_version, model, agentic runtime profile, supervised risk profile).
- `my-agents.nix.example`: rewritten around a per-runtime agent list (now `hermesAgents`, see above) with a fully-commented generic reference agent documenting every `settings` option (model + fallbacks, auxiliary routing, compression, memory, toolsets, approvals, web/terminal backends, provider timeouts, TTS/STT) plus seed/volume/env/container fields. Active agent renamed `default` → `coding` to match the docs. Quickstart and agent-tips docs updated to the new structure.
- `modules/shell.nix`: optional **zsh** (`tentaflake.shell.zsh.enable`) — Oh My Zsh + autosuggestions + syntax-highlighting + fzf-tab, with Starship as the prompt; becomes the admin login shell when enabled. Plus `tentaflake.shell.zoxide.enable` (smart-cd, cross-shell, default on) and `tentaflake.shell.lazygit.enable` (+ `lg` alias). Aliases moved to `environment.shellAliases` so they apply to bash and zsh.
- `modules/shell.nix`: **tmux** (`tentaflake.shell.tmux.enable`, configured). Convenience aliases `rebuild` (→ `nixos-rebuild switch --flake /etc/nixos#<host>`), `reload` (`exec $SHELL`) and `cls`. tmux moved out of the `tools` package set into its own toggle.
- `modules/editor.nix` + `nixosModules.editor`: optional **Neovim via nvf** (`tentaflake.editor.nvf.enable`) — LSP, treesitter, telescope, gitsigns, blink-cmp; lean language set (nix/bash/lua/markdown/yaml). Added the `nvf` flake input (kept out of `nixosModules.default` so external consumers aren't forced to have it).
- Installer: a **feature checklist** (zsh / zoxide / nvf / lazygit / modern tools) whose selections are written into the generated `/etc/nixos/flake.nix` as `tentaflake.*` toggles (nvf also injects the rev-pinned input + editor module import).
- `hermes-top` — live TUI dashboard (bubbletea) of agent filesystem activity, launched via `tentaflake top`. Reads the `hermes-auditd` SQLite DB directly (no network surface — runs over Tailscale SSH). Implements the read side the daemon previously discarded (the `internal/store` already had the query methods).
- `hermes-auditd` is now wired up: enabled by default on `agent-host`, `watchDirs` auto-derives from the agents defined in `my-agents.nix`, runs as an unprivileged `hermes-audit` user with only `CAP_DAC_READ_SEARCH`, and stores its DB group-readable so the admin can run `tentaflake top` without sudo.
- `internal/store`: `Since(afterID, limit)` (incremental tail) and `AgentRows(window)` (per-agent activity summary) read helpers.
- `modules/shell.nix` — operator shell experience for SSH/console: the agent-management CLI (`tentaflake`, see Changed), dynamic `tentaflake-status` login banner, Starship/bash prompt, completion, and a curated modern CLI tool set. Toggle via `tentaflake.shell.*`. See [`docs/06-shell.md`](https://docs.tentaflake.dev/operations/cli-shell/).
- `modules/piper-tts-server.nix` — local TTS via Piper (OpenAI-compatible `/v1/audio/speech`)
- `modules/hermes-firstboot.nix` — USB env detection + first-boot TUI wizard
- `nixosConfigurations.live-agent` + `nix build .#live-agent-iso` — bootable ISO with Hermes + TTS

### Changed
- Host operator CLI renamed `hermes` → `tentaflake` (same subcommands: `status|logs|restart|start|stop|shell|exec|ps|top|help`), now multi-runtime aware — `tentaflake status`/`ps` and the login banner list agents across all runtimes with a runtime column. A deprecated `hermes` shim still works: it prints a deprecation note to stderr and execs `tentaflake`.
- `tentaflake.shell.hermesCli.enable` renamed to `tentaflake.shell.tentaflakeCli.enable` (old name still accepted via `lib.mkRenamedOptionModule`).
- `modules/hermes-auditd.nix`: `watchDirs` auto-discovery widened from `hermes-*` containers to all declarative agent containers (`hermes-*`/`zeroclaw-*` prefixes), so ZeroClaw state dirs are watched too — unrelated oci-containers are still excluded. Auto-derived console roots keep bare names for Hermes (`coding`) and runtime-prefixed names otherwise (`zeroclaw-assistant`).
- `hermes-auditd` 0.1.3: event attribution (`agentNameFromPath`) and `hermes-top` path shortening now recognize `/var/lib/zeroclaw-<name>` state dirs — ZeroClaw activity is labelled `zeroclaw-<name>` instead of lumped into `unknown`; Hermes labels stay bare for existing DBs.
- `flake.nix` now exports `mkZeroClawAgent` alongside `mkHermesAgent` (`lib.${system}`, `specialArgs`); `configuration.nix` passes each builder to `my-agents.nix` only if the file's function asks for it (`lib.functionArgs` intersection), so old `{ mkHermesAgent }`-only files keep working unmodified.

### Fixed
- `store.go`: fixed event time-window comparisons (`Stats`, `Prune`) — stored RFC3339 timestamps were string-compared against SQLite's `datetime('now', …)` (space-separated) form, which only agreed when the date differed, so same-day events outside the window were mis-counted and never pruned. Now normalized via `datetime(timestamp)`.
- `cmd/hermes-auditd/main.go`: removed the misleading "HTTP/WebSocket server not implemented — notify channel discarded" warning; the notify channel is now drained intentionally (data is read back via `hermes-top`).
- `watcher.go`: added `Close()` method to fix fsnotify file descriptor leak
- `flake.nix`: fixed `mkHermesAgent` import (was attrset, now unwrapped function)
- `watcher.go`: fixed `FlushAll` timer race with `entry.timer = nil` guard after `Stop()`
- CI: added `go test` step to GitHub Actions workflow
- `installer.sh`: removed dead `spinner()` function
- `store.go`: removed unused `done` channel
- Docs: fixed nftables attribution (`hardening.nix` → `networking.nix`)

## [0.1.0] — 2026-06-19

### Added
- Initial release: NixOS flake template for multi-agent Hermes orchestration
- `lib/mkHermesAgent.nix` — declarative Docker-based Hermes agent creation
- `lib/constants.nix` — template-level defaults (stateVersion, locale, hostname)
- `modules/` — reusable NixOS modules (boot, locale, networking, users, hardening, tailscale, etc.)
- `pkgs/hermes-auditd/` — Go daemon for filesystem event auditing with SQLite
- `installer/` — interactive TUI installer ISO (`nix build .#installer-iso`)
- `docs/` — quickstart guide, agent tips, skill index, and 4 bundled Hermes skills
- GitHub Actions CI: `nix flake check` on PR and push to main

[Unreleased]: https://github.com/timfewi/tentaflake/compare/v0.4.0...main
[0.4.0]: https://github.com/timfewi/tentaflake/compare/v0.3.9...v0.4.0
[0.3.1]: https://github.com/timfewi/tentaflake/compare/v0.2.0...v0.3.1
[0.2.0]: https://github.com/timfewi/tentaflake/compare/v0.1.0...v0.2.0
[0.1.0]: https://github.com/timfewi/tentaflake/releases/tag/v0.1.0

---

# Documentation ownership and release synchronization

> Understand source ownership, release synchronization and documentation evidence.

Canonical page: https://docs.tentaflake.dev/reference/documentation/

> Source: [`docs/18-documentation.md`](https://github.com/timfewi/tentaflake/blob/e21be475a3afc6e95c5087003edd0b7f780e4df4/docs/18-documentation.md) at `e21be475a3af` · docs version **0.4.37**

The public Tentaflake repository owns product behavior, configuration examples,
support status, and operational guides. The public website and documentation
site present a pinned release of that material. Website maintenance can remain
separate without making contributors maintain two copies of every guide.

## Source ownership

| Content | Authoritative source | Maintenance |
|---|---|---|
| Scope and entry points | `README.md` | Update with behavior and support changes |
| Product direction and planned milestones | `docs/roadmap.md` | Keep plans separate from implemented and released support |
| Preset, capability and acceptance facts | `adapters/catalog.json` | Generate the adapter table; embed the same catalog in CLI discovery |
| Configuration and operational detail | `docs/` | Keep examples consistent with modules and builders |
| Development workflow | `CONTRIBUTING.md`, `AGENTS.md` / `CLAUDE.md` | Keep commands and verification boundaries accurate |
| Bundled operational skills | `.agents/skills/` | Update the skill and its related guide together |
| Release history | `CHANGELOG.md` | Preserve historical entries; describe new changes separately |
| Navigation, search, summaries and presentation | Website | Generate guides from the public source; review summaries |

Private deployment details and website implementation belong outside this
generic repository. Public contributor workflows must not depend on access to
a separately maintained repository or its credentials.

## For public contributors

1. Change the implementation and its authoritative guide together. Update the
   README, agent instructions, examples and affected skills where needed.
2. Check option names, defaults, CLI examples, local links, and support claims
   against source. Keep configured policy, fixture evidence and actual runtime
   acceptance distinct.
3. Run `just fast` and any explicitly selected checks required for the change;
   record what passed and what remains unverified.
4. Identify added, renamed or removed guides in the PR so the website maintainer
   can update navigation. Public contributors need only the public repository.

## For the website maintainer

Use one exact release tag and its resolved commit for every generated guide,
source link, version label, Markdown endpoint and agent-readable index. Preserve
that pin until a newer release is ready; a dirty checkout is not a release.

After a release:

1. Compare the pinned source with the new tag, including guide and skill changes.
2. Map every new guide to a stable route or record why it is deliberately excluded.
   Remove obsolete routes when upstream removes a guide.
3. Regenerate mirrored guides and indexes from that commit. Fix generated
   content upstream; avoid hand edits to synchronized copies.
4. Review landing copy, FAQ, handwritten summaries and the published skill
   against the same source diff. Generation cannot validate those claims by itself.
5. Verify source coverage, generation drift, links and the affected pages before
   publishing. Deployment remains a separate maintainer action.

A scheduled check can report a newer release without public-repository write
access or a cross-repository credential. Keep release following and website
deployment under the website maintainer's control. Local status should separately
report unreleased commits and working-tree changes instead of calling them synced.

## Reading the right version

The [website](https://tentaflake.dev/) and
[documentation site](https://docs.tentaflake.dev/) can lag a new release until
the maintainer updates them. Compare a page's source commit with the release
used by your deployment. Repository `main`, an unreleased checkout and a
released website are different evidence boundaries.

The [archive comparison](https://github.com/timfewi/tentaflake/blob/e21be475a3afc6e95c5087003edd0b7f780e4df4/docs/archive-comparison.md) and older changelog entries are
historical records. The current [adapter guide](https://docs.tentaflake.dev/configuration/agent-adapters/) defines
runtime support; a stopped scaffold does not imply operational support.

## Review checkpoint — 2026-10-04

The seven bundled skills and current guides were reviewed against adapter,
CLI, host-policy and example source. Skills now use short procedures and
source/guide links instead of copied upstream catalogs. Corrections cover
read-only Hermes/ZeroClaw configuration, Agenix recipient/runtime separation,
host identities, environment-file verification and complete agent removal.
The adapter status reflects implemented opt-in SSE while retaining the actual
vendor acceptance gaps. Historical changelog links use their matching release.

Local Markdown paths/anchors, skill YAML frontmatter and the Agenix example's
Nix syntax/metadata/formatting checks passed. Fast-gate checks passed, including all
94 Rust tests, lint/formatting, policy/adapter evaluation, read-only flake
validation and the generated installer flake. No VM/ISO build or host activation
was run. Live website, provider behavior, secret decryption and actual vendor
workloads were not exercised; publication and deployment acceptance remain
separate operations.

---

# Documentation source

> How docs.tentaflake.dev stays aligned with the public tentaflake repository.

Canonical page: https://docs.tentaflake.dev/reference/website-source/

The detailed guides on this site are synchronized from the public [`timfewi/tentaflake`](https://github.com/timfewi/tentaflake) repository. That repository stays authoritative; this site adds navigation, search, theming, stable URLs, and agent-readable formats.

## How the pin works

The site is built from one exact release tag and commit of that repository. Every synchronized page starts with a source line that names the upstream file and that commit, so you can always open the text the page was built from.

- The synchronized pages, the [`llms.txt`](https://docs.tentaflake.dev/llms.txt) index, and the list of bundled skills are all generated from the pinned commit.
- The site's build fails when a published page differs from its pinned source.
- A deployment is blocked unless the pinned commit is the one the release tag resolves to.
- A new guide must have a documentation route or an explicit exclusion before
  synchronization passes. Removed sources and duplicate routes also fail validation.

## Following new releases

tentaflake tags each merge to `main`. Release drift can be checked independently
of publishing. The maintainer moves the pin to a newer tag, regenerates the guides,
and reviews the summaries before publishing. Between a release and that update,
this site can be behind the repository; the source line on each page identifies
the exact commit you are reading. Unreleased checkout changes are outside that pin.

Pages written for this site, such as [Overview](https://docs.tentaflake.dev/getting-started/overview/), [Security boundaries](https://docs.tentaflake.dev/concepts/security-boundaries/), and [Use with AI agents](https://docs.tentaflake.dev/getting-started/ai-agents/), are reviewed by hand in the same update.

## If a page looks wrong

A synchronized page is only as current as its pin. Compare it with the linked source file on GitHub, and treat the repository as correct when the two disagree.