Skip to the content
docs

Testing

Cloudfield was named Aether until November 2026. The commands, paths and hostnames in this guide still use that name. View this guide on GitHub.

Browse guides / Testing

Which tests to run for a change, how to run each suite, what it needs, and how to read a failure. Build the tree first (Build from source); the checks every pull request sessions are in Test your change.

Which suite to run

You changed Run Needs
Any Go code make test Go
Server code that touches containers, git, SSH or storage make test-integration Docker, git; root to match CI
Dashboard components in web/src cd web && bun run typecheck && bun run test, then make public-audit Bun, Node 22+
Something a person clicks through in the dashboard make test-e2e Docker, git, Bun, Node 22+, Playwright's Chromium
scripts/, the installers, desktop/ make test-scripts sh, Node, OpenSSL 3, jq, Go
.github/workflows/ make lint-workflows ShellCheck
How an agent CLI is launched (internal/harness) Real-agent smoke tests Docker
How a session drives, reads or wakes its agent Real agents against a mock model Docker, git
The assistant's instructions or tools (internal/assistant/instructions.md, internal/ops) Assistant eval Claude Code signed in, spends tokens
Native hook adapters (the .mjs files in internal/coordhooks and internal/agentstatus) make test-native-hooks Node 22.13+
cmd/aether-edge, internal/edge go test -race ./internal/edge/edgetest/ Go
images/edge/Dockerfile The edge image Docker, curl
The edge store or scripts/edge-update.sh make test-edge-upgrade root, Docker with Compose and buildx, jq, curl
images/browser/ make browser-image && make browser-smoke Docker
android/ make android Docker
internal/vmimage go test ./internal/vmimage, then make test-kvm e2fsprogs, qemu-utils; root for the owner test
images/kernel/ The guest kernel, then make test-kvm Docker; a KVM host to boot it
internal/guest (the VM helper) go test ./internal/guest/..., then make test-kvm Go; a KVM host for the VM test
internal/runtime/vm (the VM engine) go test ./internal/runtime/vm, then make test-kvm Go; a KVM host for the VM test
Only docs/ or root *.md make public-audit sh

Unit tests

make test runs go test -race over every package, with -d=checkptr=0 passed through -gcflags to modernc.org/sqlite/..., modernc.org/libc/..., modernc.org/memory and modernc.org/mathutil: checkptr, which -race turns on, is off for the SQLite library's packages and stays on for every other package. make test TEST_RACE= drops the race detector and those -gcflags; the SQLite driver is transpiled C, which the detector slows ten to twenty times (internal/store takes 26 s without it and over five minutes with it). CI runs pull requests without -race, except internal/store, and the merge queue and main with it.

make test-store runs internal/store alone under -race with the same -gcflags. CI runs it on every event as the store-tests job and leaves internal/store out of build-and-test's make test.

CODE
make test                                   # everything
make test TEST_PKGS=./internal/store        # only these packages
make test TEST_SKIP=./internal/ptyhost      # everything but these
go test -race ./internal/scheduler -run '^TestName$' -v

When you write a test:

  • Open a database outside internal/store with storetest.Open (internal/store/storetest). It copies a database migrated once per test binary; migrating a fresh one costs seconds under the race detector.
  • Call testhome.Isolate(t) (internal/testhome) in any test that reads or writes the CLI's link config through cli.Load, cli.Save or a gateway handler. It sets AETHER_CONFIG_DIR and the platform home and config variables, and clears SSH_AUTH_SOCK. Without it the test uses your real config.
  • Never wait for real time to pass. Give a cache or deadline an injectable clock and advance it by hand. Windows' clock is coarse enough that two reads can return the same instant.
  • Give behavior that differs by platform one test per platform. CI runs the client packages on Windows, so a test whose subject refuses on Windows goes in a //go:build !windows file with a _windows_test.go counterpart that asserts the refusal.
  • Start a bug fix with a test that reproduces it, on the real user path when you can. Do not prove the same behavior again at another layer.

Integration tests

make test-integration runs the tests behind the integration build tag, against real Docker and real git, in the packages that have them. It fails when Docker is unreachable; it does not skip.

Variable Effect
INTEGRATION_PKGS Packages to run (default: every package with integration tests)
INTEGRATION_SKIP Packages to leave out
INTEGRATION_RUN Passed to go test -run
INTEGRATION_SKIP_PATTERN Passed to go test -skip

CI runs the suite as root. Run it the same way to match:

CODE
sudo env "PATH=$PATH" "HOME=$HOME" make test-integration INTEGRATION_PKGS=./internal/scheduler

Without sudo, session containers use your uid, because the scheduler hands the checkout to the container user and an unprivileged process can only hand files to itself.

make test-integration and make test-agents run with -race. A non-empty INTEGRATION_NORACE turns it off. CI runs the Docker suites in the merge queue and on main only, with -race.

CI splits the suite into two jobs. To repeat one that failed:

CI job Variables
integration (server-rest, browser) INTEGRATION_PKGS=./internal/server INTEGRATION_SKIP_PATTERN='^TestIntegration(Chaos|Coordination|EndToEnd|MultiMember|ServerUpdate)' with the browser image variables, after make browser-smoke
integration (server-flows, scheduler, rest, agents, smoke) INTEGRATION_PKGS=./internal/server INTEGRATION_RUN='^TestIntegration(Chaos|Coordination)', then the same with '^TestIntegration(EndToEnd|MultiMember|ServerUpdate)', then INTEGRATION_SKIP="./internal/harness ./internal/server" INTEGRATION_SKIP_PATTERN='^TestDockerBrowser', then the smoke tests and make test-agents

The merge queue and main run the Docker suites on amd64 and also run the arm64 job: the standard-image and browser-image builds, the smoke tests, make browser-smoke and the browser runtime tests on a native arm64 runner.

CI sets GOFLAGS=-v, so each test's duration is in the job log.

Some tests take an image from the environment:

Variable Used by When unset
AETHER_BROWSER_TEST_IMAGE TestDockerBrowser* and the server's browser scenarios aether/browser:test, which make browser-image builds (The browser companion)
AETHER_BROWSER_IMAGE The companion image a server under test starts The build's default companion image
AETHER_NESTED_DOCKER_TEST_IMAGE TestDockerNestedDocker (inside a session) The test skips
AETHER_ACP_IMAGE Enhanced-mode adapter tests (Enhanced mode), TestIntegrationAssistantIgnoresPlantedGatewayDocker and TestIntegrationAssistantIssuerMatchesTheBundledCLIs, which install the assistant's adapters from the npm registry The test skips, or uses busybox only

The resource tests check real cgroup limits, PID exhaustion and OOM containment. The capacity test starts its own dockerd and containerd in private namespaces and never reconfigures the host's daemon. It needs root, permission to unshare namespaces, dockerd and containerd on PATH, and Docker API 1.46 or newer:

CODE
sudo env "PATH=$PATH" "HOME=$HOME" go test -race -tags=integration \
  ./internal/runtime \
  -run '^TestDocker(CapacityContainerdIsolated|ResourceLimits|PIDLimitEnforced|OOMContained)$' -v

Tests in internal/runtime, internal/scheduler and internal/harness label their containers aether.test=<test name>. After an interrupted session, docker ps -a --filter label=aether.test lists what was left behind.

The E2E scenario suite

The *_integration_test.go files in internal/server start the whole server and drive it over real SSH and git transport, with real containers when Docker is reachable. Without Docker most scenarios fall back to an in-process runtime. The chaos scenarios, which run aether-server as a child process and kill it with SIGKILL, and the in-container coordination scenario skip instead.

The agents are scripted. The fake harness runs the command in AETHER_FAKE_AGENT (typically sh /workspace/agent.sh {task}, with the script committed to the test repository).

Test Covers
TestIntegrationEndToEnd One member: push, launch, attach, detach, steer, finish, pull
TestIntegrationMultiMember Joining, roles, handoff, budget caps, an agent crash
TestIntegrationChaos* The server killed mid-session, disk pressure, a stalled agent
TestIntegrationCoordination* Messages between sessions
TestIntegrationServerUpdate* aether-server update: preflight, rollback, downgrade guards
TestIntegrationRestart* Enhanced sessions through server restarts and updates (Restart acceptance)
TestIntegrationServerGateway The dashboard the server hosts on the tailnet
TestIntegrationConnectorFlows A ChatGPT connector, over OAuth and MCP, launching, messaging, answering and stopping enhanced sessions
CODE
sudo env "PATH=$PATH" "HOME=$HOME" make test-integration \
  INTEGRATION_PKGS=./internal/server INTEGRATION_RUN='^TestIntegrationEndToEnd$'

Restart acceptance

internal/server/restart_integration_test.go checks that Enhanced sessions keep working through a server restart or update, in three sizes:

  • TestIntegrationRestartSmoke runs in CI, in about 90 seconds. Two Enhanced sessions, A streaming a numbered line every 100 ms and B holding a permission request, each run a probe process started with docker exec (not part of the agent). The probes call the server through the staged CLI and MCP bridge and exchange mail, keeping an inbox --wait 20 long poll open. Just before each of two SIGTERMs and one SIGKILL, each probe fires keyed sends through both the CLI and the bridge, and one unkeyed dev.terminal.input.
  • TestIntegrationRestartCoordination measures the coordination socket alone with one external probe.
  • TestIntegrationRestartScaling repeats two SIGTERM restarts with 2, 4 and 7 live Enhanced sessions and prints the stream delay of each; it skips unless AETHER_RESTART_FULL=1.
  • TestIntegrationRestartAcceptance is the full matrix for a release or an owner check; it skips unless AETHER_RESTART_FULL=1.

The full matrix builds aether-server twice (the second reports v0.0.0-soak-b), runs one as a child process against real Docker with a temporary data directory, and launches three Enhanced sessions of a scripted agent (internal/acphost/acpmock):

  • Sessions A and B each start a 120-second shell command that appends a line a second to a marker file, then for 105 seconds call the server through the staged aether-internal CLI and the staged aether-server mcp bridge (internal/server/testdata/soakagent), about one request a second in all. A sends mail to B with the CLI and reads with the bridge; B does the reverse. Half of A's CLI sends omit the idempotency key; the bridge requires one. Each side keeps one inbox long poll in flight. The agent streams a numbered line every 100 ms as its answer.
  • Session C holds a permission request open until the end.

The test then restarts the server five ways, 20 seconds apart: SIGTERM and start the same binary twice, replace the binary with the second build, put the first build back (the rollback case), and SIGKILL. Every restart lands while a call, an inbox long poll and a send are in flight. The swap and rollback stop the server with SIGTERM and replace the file, as aether-server update does; the exec step itself is covered by TestIntegrationServerUpdate*.

It fails on any of:

  • a turn interrupted or a run parked with turn interrupted;
  • a failed CLI or bridge call the agent saw;
  • a send that failed but was delivered, unless its error says it "may already have applied", or a message delivered twice;
  • an unkeyed terminal input applied more than once, or its echo not read from the output cursor held before the restart without missing_cursor or truncated;
  • no terminal input accepted after the first restart with the control lease the probe acquired before it;
  • a sent message that was never received, or was received twice;
  • a streamed line missing or repeated;
  • the marker file not holding lines 1 to 120, or the container's ID or start time changing;
  • the permission request lost, or its answer not reaching the agent.

It prints, per restart, the milliseconds between the last success before the signal and the first success after it for each channel (CLI and bridge calls on both runs and the aether-acp stream). Status calls run every 2 seconds (4 in the smoke test), because the coordination socket allows one request a second per run, so a gap under about one interval is the probe interval and means no outage.

CODE
sudo env "PATH=$PATH" "HOME=$HOME" AETHER_RESTART_FULL=1 make test-integration \
  INTEGRATION_PKGS=./internal/server INTEGRATION_RUN='^TestIntegrationRestart'

Both use busybox:1.36, which must be pulled; the full matrix takes about four minutes.

Real-agent smoke tests

internal/harness/smoke_integration_test.go starts the vendors' real agent CLIs to check that the command lines Cloudfield builds are still accepted. A vendor renaming a flag breaks every session of that agent, and nothing else catches it.

TestSmokeHeadlessNoLogin runs in CI. It launches each headless template with no credentials and passes when the session fails for want of a login, never for an unparseable command line. CI's integration (server-flows, scheduler, rest, agents, smoke) job builds the standard image, adds the agent CLIs at their current versions with images/smoke/Dockerfile, and sessions:

CODE
docker build -f images/standard/Dockerfile -t aether-standard:local .
docker build -f images/smoke/Dockerfile --build-arg BASE=aether-standard:local \
  -t aether-smoke:local .
AETHER_SMOKE_IMAGE_CLAUDE_NOLOGIN=aether-smoke:local \
AETHER_SMOKE_IMAGE_CODEX_NOLOGIN=aether-smoke:local \
AETHER_SMOKE_IMAGE_OPENCODE_NOLOGIN=aether-smoke:local \
  go test -tags integration -run TestSmokeHeadlessNoLogin ./internal/harness/

A _NOLOGIN image must carry no credentials. TestSmokeClaude, TestSmokeOpencode and TestSmokeCodexFlags drive the agent far enough to produce output, so they need an image that carries that agent's login, named by AETHER_SMOKE_IMAGE_<NAME> (for example AETHER_SMOKE_IMAGE_CLAUDE). CI has no such credentials, so they are a manual check. Each test skips when its variable is unset. On failure a test prints the agent's output up to that point.

pi and omp have no smoke test in that image.

Enhanced mode

Enhanced sessions drive the agent through an adapter that speaks the Agent Client Protocol (ACP). Most tests use internal/acphost/acpmock, which replays conversations recorded from the real adapters and answers prompts by their text (ask permission, ask form, wait, hit the usage limit, demo, demo edit):

CODE
make test-integration INTEGRATION_PKGS='./internal/scheduler ./internal/server' \
  INTEGRATION_RUN='TestIntegrationEnhancedRun'

TestACPAdapterInstall installs the pinned Claude Code and Codex adapters into the standard image and completes the ACP handshake with each. It needs the npm registry. With ACP_LIVE=1 it also opens a session, which needs no login (a logged-out Codex skips):

CODE
AETHER_ACP_IMAGE=aether-standard:local \
  go test -tags integration -run TestACPAdapterInstall -v ./internal/harness/

Two tests run the agents and adapters on your PATH with your own logins. TestLiveSwitch switches a session between Standard and Enhanced both ways; ACP_LIVE_OMP_MODEL picks omp's model when its default cannot answer. TestLiveEnhancedRunsWithoutPermissionPrompts fails if an Enhanced session asks for permission to run a shell command:

CODE
ACP_LIVE=1 go test -tags integration -run TestLiveSwitch -v ./internal/harness/
ACP_LIVE=1 go test -tags integration -run TestLiveEnhancedRunsWithoutPermissionPrompts -v ./internal/harness/

Real agents against a mock model

TestIntegrationRealAgents (internal/server/realagents_integration_test.go) launches real Claude Code and Codex sessions, Standard, Enhanced and Background, on real Docker, and checks that each report an agent makes, or fails to make, leaves the session in the state the dashboard shows. A mock model in the test process (internal/server/mockmodel_integration_test.go) serves streaming Anthropic /v1/messages and OpenAI /v1/responses. Each prompt the test sends carries a tag, aether-script:<turn>;, naming the script the mock plays for that turn: one tool call per step, then a text answer. The mock answers each report reminder Cloudfield sends with aether-internal report.

One Standard and one Enhanced session per agent play these turns in order. The launch starts turn, mail starts wake, and run.inject starts the rest:

Turn The agent The session must show
turn runs a command and ends the turn without a report answered after one reminder; Standard Codex, which Cloudfield cannot remind, waiting
wake gets mail while idle a new turn, where Cloudfield can wake the session
waiting reports waiting --for ci waiting-ci with the summary
answered reports answered answered, not Needs you
late-report reports waiting --for process from a detached process five seconds later the report landing after the session parked, then waiting-tools
compaction claims more tokens than its context holds, compacts, then reports the reporting rules still in the next request, then answered
subagent starts a subagent that runs for eight seconds not Needs you at any point while it runs, the subagent never reminded, then the turn ends like turn
needs-input reports needs-input needs-you, Needs you

Each check also reads the session's cause, its report entries on the timeline, and whether the report's output reached the model. A failure names the broken link: the reporting rules missing from a request (instruction missing), no report output reaching the model (report never made), a report missing from the timeline or its output missing from the model's requests (report lost), or a report recorded but shown as the wrong state (misclassified). The first turn and every Background session also require the launch to record where its instructions went.

The wake turn runs where Cloudfield can wake the session: every Enhanced session, and a Standard session whose agent sets IdleWake in its capabilities (internal/harness/capabilities.go); the others skip it. A session must never need you while its subagent works. It should show working, or waiting-agents when Standard Claude Code ends the turn with the subagent still running in the background. Cloudfield gets no signal for a Codex subagent, and Standard Claude Code's Stop hook can arrive without the subagent among its background tasks, so those sessions can read as idle; the test logs their idle states as a known gap and fails any other.

Two Background sessions per agent check that a Background session is never reminded: one reports success and finishes awaiting review, the only Background state that needs you; the other has its needs-input report refused and finishes without a report.

TestIntegrationSilentModelParksAsPossibleStall (internal/server/liveness_integration_test.go) launches an Enhanced Claude Code run against a model that accepts every request and never answers. It requires the session to park as a possible stall awaiting its first event, and never to read disconnected.

TestIntegrationClaudeEnhancedSleepsAndWakes (internal/server/sleep_integration_test.go) runs a server with --run-sleep-after 5s. An Enhanced Claude Code session answers one turn and falls asleep; the test requires its container to be stopped, not removed, and the session still live. A prompt then wakes it, and the model's next request must carry the turn from before the sleep. An agent gets Capabilities.Sleep only with a test like this one.

Sessions reach the mock the way they reach a model gateway: ANTHROPIC_BASE_URL for Claude Code, openai_base_url in ~/.codex/config.toml for Codex. The test seeds the member home with what a member's first sessions leave there: onboarding, an approved API key and a trusted checkout. The Codex config also sets model_auto_compact_token_limit, so the compaction turn's claimed usage makes Codex compact.

CI's integration (server-flows, scheduler, rest, agents, smoke) job runs both as root. Run them the same way, with Docker and git:

CODE
make agents-image   # images/agents/Dockerfile on the published standard image
sudo env "PATH=$PATH" "HOME=$HOME" make test-agents

AGENTS_BASE=aether-standard:local builds on a standard image you built. A failure prints the broken link, the session's report entries, the agent's screen (Standard) or ACP history (Enhanced), and the final item of each request the mock received.

Bumping the pinned versions

images/agents/Dockerfile pins the CLIs in CLAUDE_CODE_VERSION and CODEX_VERSION; the ACP adapters follow internal/harness/acpregistry.json, the versions members install. The smoke image takes each vendor's latest release; this one is pinned so a vendor release cannot fail an unrelated pull request.

scripts/bump-agent-pins.sh moves every pin to the newest release and prints each one it moved: the adapters in acpregistry.json and internal/harness/acp.go to the versions the ACP registry lists, each adapter's CLIVersion to the CLI that release installs, the two CLIs above to npm's latest, and the oh-my-pi release session VMs' auth broker runs, in internal/serversetup/omp.go, to the newest GitHub release with its SHA-256 and size. It needs curl, jq and npm. Run the two commands above on the result, and fix what the new releases broke in the same pull request.

The Agent pins workflow (.github/workflows/agent-pins.yml) runs the script every Monday at 06:00 UTC and on gh workflow run agent-pins.yml, and opens or updates one pull request from bot/agent-pins. Pull request CI runs the unit tests; the merge queue runs the real-agent suite on the new versions. The workflow never replaces a pull request that carries a commit anyone else authored or committed, so push a fix for a failing bump to that branch.

The workflow pushes as a GitHub App, because a pull request opened with the workflow's own token starts no CI. To set it up once, an organization owner:

  1. Creates a GitHub App with no webhook and the repository permissions Contents, Pull requests and Workflows set to Read and write. GitHub refuses an App's push that changes a workflow file without Workflows, and the branch rebuilt on a newer main changes every workflow main changed since the last run.
  2. Installs it on this repository only.
  3. Saves its App ID as the repository variable AGENT_PINS_APP_ID and a private key as the repository secret AGENT_PINS_APP_KEY.

Without them, each run fails with set the AGENT_PINS_APP_ID variable and the AGENT_PINS_APP_KEY secret.

The assistant installs the adapters at the versions in acpregistry.json, and Codex at the CLIVersion next to its pin (internal/harness/acp.go). It refreshes its sign-ins with the endpoints and client ids in internal/scheduler/assistant_container.go. CI's integration job checks them with the test below; after moving any of them, run it yourself:

CODE
AETHER_ACP_IMAGE=aether-standard:local \
  go test -tags integration -run TestIntegrationAssistantIssuerMatchesTheBundledCLIs -v ./internal/scheduler/

Assistant eval

internal/assistant/eval measures what the built-in assistant does with internal/assistant/instructions.md. It runs the real Claude Code CLI (claude -p) in a directory whose CLAUDE.md is those instructions. The agent's only tools are the assistant's own, ops.RegisterAssistant, served over HTTP MCP by a fake control surface that answers from synthetic scenario state. The fake records every tool call. A confirmed-tier call (launch, message, merge) is recorded and answered with a made-up success, so nothing is ever started, sent or merged; the eval scores the parameters the agent proposed.

It spends model tokens, so it is behind the assistanteval build tag and no CI job runs it. Claude Code must be on PATH and signed in:

CODE
AETHER_ASSISTANT_EVAL_LABEL=main \
  go test -count=1 -tags assistanteval -run TestAssistantEval -timeout 40m ./internal/assistant/eval
Variable Default Meaning
AETHER_ASSISTANT_EVAL_LABEL unlabeled Names the output files; use the commit or branch under test
AETHER_ASSISTANT_EVAL_RUNS 3 Sessions per scenario
AETHER_ASSISTANT_EVAL_MODEL Claude Code's default Passed to claude --model; the report names the model the harness reports
AETHER_ASSISTANT_EVAL_ONLY all Comma-separated scenario ids
AETHER_ASSISTANT_EVAL_PARALLEL 4 Concurrent agent sessions
AETHER_ASSISTANT_EVAL_OUT $TMPDIR/assistant-eval Where <label>.md and <label>.json go

The scenarios and their expected checks are in internal/assistant/eval/testdata/scenarios.json: a rough voice request, a question, a research-only request, a prohibition, hostile pasted text, a saved preference against a one-off override, two members with different defaults, a launch with no dashboard view, a shared account, an unavailable saved target, merged against released, a failing PR behind a success report, and a reopened session. The checks are keyword and tool-call tests, so they are deterministic but coarse: text matches are whole words (so bo does not match both), an invented-scope keyword does not count when the brief negates it ("no refactors"), and the same holds for words a brief must omit ("do not delete staging" does not copy the hostile instruction), and a question counts only when it is a sentence addressed to the member, not a ?run= link or inline code. A question after a completed launch that offers to show or edit the brief is reported in its own column, not as unnecessary. The Markdown report ends with a manual-review list per scenario, with each launch brief and reply.

The report has a row per scenario: sessions passed, unnecessary questions, brief offers, launches that should not have happened, dropped prohibitions, invented-scope keywords, tool calls and latency per session, and cost. Cost is the figure claude reports, and "not measured" when it reports none. Results are rewritten after every session, and each session's timeout is capped by the test's -timeout, so a session that hits the limit still leaves a report.

A scenario marked pending: <tool> needs a tool the registry does not offer yet, such as the saved launch choices. The eval serves an eval-only stub of the tool from the scenario's contract so the baseline is honest about what the agent does without it. Its rate is not product behavior. When the registry offers the tool, the real one is served instead, and the scenario's answers need an entry for each RPC method it calls; a missing one shows up as "is not available in this synthetic scenario" in the reply.

Compare a change with main by running the command on both commits with different labels. Model output varies between sessions, so a few sessions per scenario shows a direction, not a proof. Scripted tests of the scorer itself, go test ./internal/assistant/eval, run in make test.

Native hook tests

The pi, omp and opencode adapters are JavaScript that Go never runs. make test-native-hooks runs them under Node 22.13 or newer against lifecycle fixtures shaped like each agent's SDK:

CODE
make test-native-hooks
node --experimental-vm-modules --test --test-name-pattern='omp:' \
  internal/coordhooks/native_pi_omp_lifecycle_test.mjs

internal/agentstatus also runs status.ts under bun from go test, and skips those tests where bun is not installed.

The fixtures do not prove a real model turn. To check one, start an isolated coordinated Standard session with the exact CLI, server and extension versions under test, end a turn idle, send it one message with aether-internal send --to <session> --body <nonce>, and confirm exactly one new turn reads and acknowledges it. Never edit a live hook, credentials or trust settings to get there.

The edge suite

internal/edge/edgetest runs edge remote access end to end in one process, under plain make test. It needs no Docker and no network. Each test starts a real edge against a fake GitHub, real servers under each access policy, the real client and the local gateway. A proxy in front of the edge holds the edge's signing key and plays a compromised edge: it forges, replays, alters and drops control messages, and tests read each server's store to see what an attack changed.

CODE
go test -race ./internal/edge/edgetest/
go test -race ./internal/edge/edgetest/ -run '^TestClaim$' -v

Every request reaches the edge from 127.0.0.1, so each test moves its edge's clock forward to refill the rate limits. Tests run in parallel; the client calls that read AETHER_CONFIG_DIR take turns.

Behind nginx

scripts/edge-nginx-test.sh, part of make test-scripts, runs a real nginx from packaging/nginx/aether-edge.conf.example in front of the edge, and skips when nginx is not on PATH. It checks sign-in, a claim, a relayed connection held silent past nginx's 60-second read timeout, and that a request with a forged X-Forwarded-For is recorded under the address nginx saw. It takes about 80 seconds:

CODE
sh scripts/edge-nginx-test.sh

The edge image

scripts/edge-image-smoke.sh runs a built images/edge/Dockerfile image with fake GitHub credentials. It checks that the image runs as a non-root user with no shell, fails to start without configuration, passes its healthcheck, stops cleanly on SIGTERM without logging the secret, and keeps its key on its volume. It needs a container runtime, so it is not part of make test-scripts. The second argument names the runtime (default docker):

CODE
make edge-image
sh scripts/edge-image-smoke.sh aether/edge:test

With podman, build with podman build --format docker -f images/edge/Dockerfile .. The default OCI format drops HEALTHCHECK and the smoke test fails.

The image builds with its own golang base image, so scripts/edge-go-version-test.sh, in make test-scripts, fails when that image and go.mod's toolchain line name different Go versions:

CODE
edge-go-version-test: images/edge/Dockerfile builds with golang:1.26.8, but go.mod pins go1.26.9; change the golang tag and its @sha256 digest in images/edge/Dockerfile to 1.26.9 (or go.mod's toolchain line to go1.26.8)

Edge upgrade qualification

TestEdgeUpgrade updates real edge containers with scripts/edge-update.sh, from the releases edge hosts run to builds of this tree. It needs root (the data directory belongs to uid 65532), Docker with Compose and buildx, jq, curl, Go, network access to github.com, and git history with tags:

CODE
sudo env "PATH=$PATH" "HOME=$HOME" make test-edge-upgrade

It starts from each release the signed stable.json and beta.json pointers name, and with EDGE_QUALIFY_CHANNEL=nightly, which release-build.yml sets for a nightly, also from the one nightly.json names. While no pointer exists it starts from the previous tag (git describe --tags HEAD^). Each starting release is built in a git worktree into a local registry, runs from packaging/edge/compose.yaml behind nginx, has real servers enroll and be claimed, and is updated to this tree as v99.0.1. From the first starting point the test goes on through three more builds of this tree with an extra migration: one that fails, one that raises the compatibility floor and then fails, and one that succeeds. It checks that:

  1. aether-edge-update refuses an image it cannot read a version from.
  2. A failing migration is rolled back without a restore and skipped next time.
  3. A failure after a raised floor restores the pre-update backup.
  4. A good migration applies, with a backup from the edge and the updater.
  5. A server of the starting release still enrolls and relays.
  6. aether-edge-update --rollback returns to the old binary.

After each step the edge must keep its key and its accounts, and each owner must reach their server. Stub cosign and docker commands from internal/edge/edgetest/testdata/upgrade/ answer the promotion checks, and the release-check of a starting release, whose binary trusts only the release keys built into it. This tree's builds are stable releases at build 9000001 and up that also trust a test key. Their own release-check verifies the manifests the test signs with it, which the fake GitHub serves. From step 3 on, the fake answers 404 for the running release's manifest, so the updater orders by the copy it kept.

It refuses to start while a container named aether-edge exists, removes everything it creates, and takes about two and a half minutes with warm caches, plus about a minute for each further starting point; this tree's builds are made once for all of them. CI runs it in edge-image.yml in the merge queue and on main, and in release-build.yml before a release publishes.

Dashboard component tests

Component tests live beside their components in web/src/ and run in vitest with jsdom:

CODE
cd web
bun run test                               # all of them
bun run test src/routes/files/home.test.tsx

jsdom has no layout, so web/src/test/setup.ts answers every media query with false and a component renders its widest layout. atViewport (web/src/test/viewport.ts) puts one test on one screen size. Real layout, hit testing and focus outlines belong to the dashboard end-to-end suite.

The dashboard end-to-end suite

web/e2e/ drives Chromium against the dashboard embedded in dist/aether, through a real aether gui gateway and a real aether-server over SSH. Playwright is the runner.

CODE
(cd web && bunx playwright install chromium)        # once, from the repo root
make test-e2e                                        # builds dist/ first
make test-e2e E2E_ARGS='--shard=1/2'                 # one of CI's two shards
make test-e2e E2E_ARGS='--project=chromium run-files.spec.ts'
cd web && bunx playwright test --project=mobile      # reuse the binaries already in dist/

make test-e2e rebuilds the dashboard and binaries because the CLI serves the dashboard it embeds: a stale binary tests a stale dashboard.

There are two projects. chromium runs every spec except *.mobile.spec.ts on a desktop viewport. mobile runs only those, as a Pixel 7 (touch, 412x839). Mobile specs tap instead of clicking and attach a full-page screenshot to every test. WebKit is not installed, so iOS Safari is not covered.

Each test gets its own stack:

  • One aether-server on its own loopback port, with a temporary data directory, the busybox:1.36 image and AETHER_FAKE_AGENT. The first identity to connect becomes the admin.
  • One aether gui per member, each with its own HOME and AETHER_CONFIG_DIR, so nothing touches your own keys or links.
  • Real git repositories on disk.

aether.server.restart() kills the server with SIGKILL and starts it again on the same port and data directory, for specs about what survives a crash.

Specs that open a real session need Docker and skip without it. onboarding-github and remote-development-git/native.spec.ts fail without Docker instead. Some specs need more:

Spec Needs
onboarding-github, and native, git-card and github in remote-development-git/ The real standard image: docker build -f images/standard/Dockerfile -t aether-standard:ci ., or another build named by AETHER_E2E_STANDARD_IMAGE
development-browser/* The browser companion image (below) and node:22.14.0-alpine3.21
remote-development-git/import.spec.ts Network access to AETHER_E2E_GIT_SOURCE_URL (default https://github.com/3xDevOps/cloudfield.git)
remote-development-git/github.spec.ts Opt-in: AETHER_E2E_GITHUB_PUBLISH=1, AETHER_E2E_GITHUB_TOKEN, AETHER_E2E_GITHUB_BASE_REPOSITORY (a public owner/name), AETHER_E2E_GITHUB_HEAD_REPOSITORY (your writable fork); AETHER_E2E_GITHUB_BASE_BRANCH defaults to main

github.spec.ts opens, comments on and closes a real pull request upstream. Pass the token through the environment only, never in a file or a log.

To run as root like an installed server:

CODE
sudo env "PATH=$PATH" "HOME=$HOME" \
  "PLAYWRIGHT_BROWSERS_PATH=$HOME/.cache/ms-playwright" \
  AETHER_E2E_STANDARD_IMAGE=aether-standard:ci \
  AETHER_BROWSER_TEST_IMAGE=aether/browser:test \
  AETHER_BROWSER_IMAGE=aether/browser:test make test-e2e

Reading a failure

  • web/playwright-report/ is the HTML report; open it with cd web && bunx playwright show-report. A failed test has the server's output attached.
  • web/test-results/ holds traces and screenshots.
  • A failed test keeps its scratch directory, /tmp/aether-e2e-*. After a session as root it holds root-owned files; remove it with sudo rm -rf.
  • In CI each shard uploads playwright-report-<shard> and playwright-results-<shard> for seven days, on success and failure: gh run download <session-id> -n playwright-report-1.

A new onboarding wizard step is one class in web/e2e/pages/wizard.ts: give it the aria-label of the step's <section>, add its label to stepNames in header order, and hang it off OnboardingWizard.

The browser companion

A session's Browser tab is a Chromium in its own container, the companion, built from images/browser/. It needs no display, host browser or host browser libraries, but it does need a working kernel user-namespace and AppArmor policy on Ubuntu. A sandbox failure must fail the check: do not retry with --no-sandbox or relax the host's security policy.

CODE
docker build -f images/standard/Dockerfile -t aether-standard:ci .
make browser-image                 # aether/browser:test; BROWSER_IMAGE=<ref> for another tag
make browser-smoke                 # runs the image's own sandbox and behavior checks

sudo env "PATH=$PATH" "HOME=$HOME" \
  AETHER_BROWSER_TEST_IMAGE=aether/browser:test \
  go test -race -timeout=10m -tags=integration ./internal/runtime \
    -run '^TestDockerBrowser' -v

AETHER_BROWSER_TEST_IMAGE=aether/browser:test \
AETHER_BROWSER_IMAGE=aether/browser:test make test-integration

Run the runtime test as root, like the installed server. Do not make its control directory world-writable to get a pass.

Committing selected paths from the Changes view uses native git and gh. Their suite runs against a local TLS stand-in for GitHub:

CODE
go test ./internal/runrepo

Android

make android runs the app's JUnit tests, then builds the APK and the app bundle, all in pinned containers; make android-debug builds a debug APK without the tests. Building, signing and installing are in Android shell. scripts/android-version-code-test.sh and scripts/android-verify-signature-test.sh, both in make test-scripts, cover the version code and the signature check with the SDK tools stubbed.

The guest kernel

make kernel builds dist/aether-vmlinux-x86_64, the kernel session VMs boot, from the version and checksum in images/kernel/build.sh and the options in images/kernel/config-x86_64. It runs in a Debian container pinned by digest, with packages from a dated snapshot, so two builds give the same SHA-256; compare them after changing either file. The build fails, naming the options, when a line of the config does not reach the kernel's .config because something it depends on is off.

On a KVM host, TestKVMStandardImageRunsDocker (VM tests) checks that a change still boots an image and runs Docker in it: it builds the image's disks with internal/vmimage, boots them with the kernel, starts dockerd, runs a container and a -p port mapping, and refuses to save the root disk while the VM runs.

Docker's contrib/check-config.sh from the moby repository, pointed at the config (/proc/config.gz in the guest), must report every Generally Necessary option enabled.

go test ./internal/vmimage covers layer extraction and the image store with real mkfs.ext4, debugfs, qemu-img and qemu-io; it skips the tests whose tools are missing. Run it once as root too, as CI's build-and-test does: only root can keep other owners, set-user-ID bits and file capabilities, as the server does.

The VM helper

go test ./internal/guest/... runs the real helper service against the real client over a Unix socket that answers Cloud Hypervisor's vsock handshake: processes on pipes and terminals, reattaching from a new client, lossless output, spawn failures, signals, orphan reaping and Boot. What only PID 1 in a VM does (switching root, mounts, the network, socket forwarding, the process limit, an out-of-memory kill) needs a KVM host. TestKVMHelper (VM tests) boots an image from internal/vmimage disks with the test binary as the helper and drives it through the client.

VM tests

Tests that boot real Cloud Hypervisor VMs carry the kvm build tag, so make test and make test-integration never build them. make test-kvm runs every package that has them:

Package Test Boots
internal/vmimage TestKVMStandardImageRunsDocker The standard image from store disks, with Docker inside (The guest kernel)
internal/guest/helper TestKVMHelper The helper as PID 1, driven through its client (The VM helper)
internal/runtime/vm TestKVM... in each kvm_*_test.go Sessions through the engine, with real systemd units and the server binary as monitor and helper: lifecycle, save and restore, a guest reboot, the home, the checkout, the network, Save environment

It needs root on a Linux x86-64 host with /dev/kvm, cloud-hypervisor, qemu-img, mkfs.ext4, virtiofsd at /usr/libexec/virtiofsd, systemd, iptables and network access to pull images. sudo aether-server vm install --vm-kernel dist/aether-vmlinux-x86_64 installs them the way a server does, as CI's kvm job does. make test-kvm fails, naming what is missing, rather than skip. It runs only the tests named TestKVM...: the unit tests already run in make test, and the helper's reap every child of the test binary, the VM monitor's too. Build the kernel first:

CODE
make kernel
sudo env "PATH=$PATH" "HOME=$HOME" make test-kvm
sudo env "PATH=$PATH" "HOME=$HOME" make test-kvm KVM_RUN='^TestKVMHelper$' KVM_SKIP='./internal/vmimage ./internal/runtime/vm'
Variable Default Meaning
KVM_KERNEL dist/aether-vmlinux-x86_64 The guest kernel
KVM_IMAGE ghcr.io/3xdevops/aether-standard:latest The image the store and helper tests boot; the engine tests boot ubuntu:24.04
KVM_RUN ^TestKVM Passed to go test -run
KVM_SKIP none Packages to leave out

The target builds the test binaries and aether-server with CGO_ENABLED=0: the binaries also run inside the VMs, where there are no libraries. It builds them, and keeps the disks, in a new private directory under $TMPDIR (default /tmp) that it removes at the end: root runs that server, so no other user may be able to replace it. A test's VM gets at most 2 GiB; on a CI runner the tests take about three minutes.

On a KVM host without Go, build on your machine, copy the directory over, and run each test binary there with the variables make test-kvm sets:

CODE
CGO_ENABLED=0 go test -c -tags kvm -o kvm/ ./internal/vmimage ./internal/guest/helper ./internal/runtime/vm
CGO_ENABLED=0 go build -o kvm/aether-server ./cmd/aether-server
cp dist/aether-vmlinux-x86_64 kvm/
scp -r kvm <host>:

# on the host, in kvm/; then the same for helper.test and vmimage.test
sudo env AETHER_TEST_KERNEL="$PWD/aether-vmlinux-x86_64" \
  AETHER_TEST_IMAGE=ghcr.io/3xdevops/aether-standard:latest \
  AETHER_VM_TEST_KERNEL="$PWD/aether-vmlinux-x86_64" AETHER_VM_TEST_DIR="$(sudo mktemp -d)" \
  AETHER_VM_TEST_SERVER="$PWD/aether-server" ./vm.test -test.run TestKVM -test.v

A test that is interrupted can leave a VM behind: systemctl list-units 'aether-vm-*' lists the engine's, and pgrep -a cloud-hypervisor the others.

Windows

CI's windows job builds, vets and tests the client packages on Windows. The windows-install workflow runs scripts/install-test.ps1 under Windows PowerShell 5.1 and PowerShell 7. You can run it on your own Windows machine with Go installed; it restores your PATH and environment when it ends:

CODE
powershell -NoProfile -File scripts/install-test.ps1
pwsh -NoProfile -File scripts/install-test.ps1

scripts/install-smoke.ps1 and scripts/update-smoke.ps1 install, update and launch the real app under Defender. They change Defender settings, the Start Menu and the registry, so they refuse to run unless GITHUB_ACTIONS=true and RUNNER_ENVIRONMENT=github-hosted. When one fails, scripts/defender-diagnostics.ps1 prints Defender's detections in the job log. The screenshots are uploaded as windows-desktop-* artifacts.

When update-smoke.ps1 cannot move aether.exe because another process holds it open, it prints the processes Windows Restart Manager names and waits up to 30 s for them to close it. It fails if Restart Manager names no process, or if the move still fails after the wait.

Before they run, scripts/windows-defender.ps1 -Mode Prepare removes the runner image's Defender exclusions and adds one for GitHub's own agent, provjobd.exe* in the runner user's Temp. On windows-11-arm, Defender flags that agent as Trojan:Win32/Gracing.I and removes it; the job then hangs until GitHub fails it with "The hosted runner lost communication with the server". Nothing Cloudfield builds or installs is excluded.

How CI runs the suites

The CI workflow (.github/workflows/ci.yml) runs on every pull request, every merge queue entry, every push to main or a release/X.Y hotfix branch and nightly on main at 08:00 UTC. A pull request, draft or ready, runs changes, audit, lint, build-and-test, store-tests and dashboard. Every other job lives in .github/workflows/ci-suite.yml, which ci.yml calls as the suite job, so a run that skips them shows one skipped suite check; in a run that calls it each job shows as suite / <job>. The merge queue runs every job, so the full suite runs once per change, on the merge result, before main moves; a failure there removes the pull request from the queue. The one exception is kvm, which boots VMs: the queue runs it only when the change can affect the VM tests (the vm mode below), and the nightly run always does.

The queue moves main to the commit it tested, so a push to main reuses that verdict: scripts/ci-queue-verdict.sh finds the merge queue run on the same commit that completed with conclusion success and whose CI gate passed, and the push runs only changes and CI gate, whose summary links that run. The push runs every job when no such run exists, when a lookup fails, when the push touches 300 files or more (GitHub's compare limit), or when it changes a file a cache key hashes (ci-classify-areas.sh caches). Caches save only from runs on main, because a merge queue run's caches are scoped to its own queue branch; the nightly run refreshes them, so between nightly runs a session restores compiler caches up to a day old. It also tests main against the newest agent CLIs and the latest published release; its result gates nothing, so read a red nightly CI gate on the main tip yourself. A merge queue run re-run after its push reused it does not change that push's verdict: re-run the push session too. A push to a release/X.Y branch never reuses a verdict and runs every job, the VM tests included; a hotfix release reads that run.

CI gate is the one required check: it passes when every other job passed or was skipped by its own rule, and fails when any failed or was cancelled. In the merge queue no rule skips a job, so there it passes only when every job passed: a cancellation can leave a job skipped. A new job in ci.yml must be added to its needs list to gate main; a job in ci-suite.yml gates it through suite.

main takes changes through a merge queue. gh pr merge <n> --squash adds a ready pull request whose CI gate passed; the queue batches up to five entries, runs the full suite on each batch's merge result, and moves main only when that run's CI gate passes. Each queue run's changes job first cancels the runs of groups the queue has discarded, so they stop holding runners. In the Docker-heavy jobs, the runner's Docker daemon and the buildx builders pull Docker Hub images through mirror.gcr.io and fall back to Docker Hub, so a Docker Hub outage alone does not fail them.

Job Sessions Locally
audit scripts/public-audit.sh make public-audit
lint Formatting, workflow lint, make lint (golangci-lint, which includes go vet and shadow), then the script and native hook tests; make vulncheck as advice on every session that runs the full suite make fmt-check lint-workflows lint test-scripts test-native-hooks
build-and-test Unit tests except internal/store (-race on the merge queue and main only), then make build make test TEST_SKIP=./internal/store build
store-tests internal/store's unit tests, always with -race make test-store
integration (server-rest, browser), integration (server-flows, scheduler, rest, agents, smoke) Integration tests, the smoke tests, real agents against a mock model and the browser companion, on amd64 See the table under Integration tests
arm64 Standard and browser image builds, smoke tests and browser runtime tests on arm64 Same commands on an arm64 host
dashboard bun run typecheck and bun run test Same, from web/
dashboard-e2e End-to-end suite in two shards make test-e2e E2E_ARGS='--shard=N/2'
windows, windows-install Windows; windows also scans release-build's Windows binaries with Defender, and the windows-install install (zip) and update legs package and launch the Windows desktop zip on amd64 and arm64
release-build (binaries, desktop, android) make release-binaries, aether gui build on Linux with and without Node on PATH, make android make release-binaries, Desktop shell, make android
edge-image The edge image and qualification
kvm VM tests on a hosted runner, which has /dev/kvm (.github/workflows/kvm.yml) make kernel, then make test-kvm as root

A pull request that changes only docs/** or root-level *.md files runs audit and nothing else. A Markdown file anywhere else counts as code. scripts/ci-classify-areas.sh narrows the rest by area:

Mode Jobs True for
go lint, build-and-test, store-tests Everything except web/ (other than web/embed.go), android/ and docs/
web dashboard web/, cmd/aether/, internal/localgw/, internal/webgate/, go.mod, Makefile, .github/actions/, ci.yml
caches The full suite on a push to main go.mod, go.sum, .golangci.yml, .github/, images/, the fonts the browser image copies from web/public/fonts/, the web and desktop package manifests and lockfiles, web/.bun-version, the Gradle files
vm kvm in the merge queue and on a push to main internal/runtime/vm/, internal/vmimage/, the packages the VM code runs (internal/guest/, internal/acpkeeper/, internal/version/, internal/devexec/, internal/disk/, internal/egress/, internal/gitengine/, internal/rootfs/), images/kernel/, the kvm*_test.go files, the files of internal/runtime/ the VM engine shares (session.go, stream.go, managed_exec.go, docker_image.go, docker_cgroup.go), files named vm*.go or *_vm.go (and their tests) anywhere under internal/, go.mod, go.sum, vm.mk, kvm.yml and cmd/aether-server/vm.go

VM code outside the VM packages goes in files named for the vm mode, so the merge queue still runs kvm for it. make test-scripts fails when the KVM tests compile a package that is neither in the list nor in the exclusions in scripts/ci-classify-areas-test.sh, each with the reason it needs no VM run, or when a test file with the kvm tag is outside the list. make lint builds the VM tests on every pull request. The merge queue classifies the files GitHub's compare lists between main and the group, because a group also carries the entries ahead of it; a push compares with the commit before it; past its 300-file limit, or when the lookup fails, kvm runs. The guest kernel takes up to six minutes to build, so kvm restores it from a cache keyed by a hash of images/kernel/ and the make kernel recipe, and builds it only on a miss; runs on main save it.

The classifier runs from the pull request's (or merge group's) base revision, so a pull request cannot change its own classification, and a renamed file counts by its old path as well as its new one. A failed classification runs both. A skipped job reports as skipped, not failed. A newer push cancels a pull request's run (cancel-in-progress for pull_request only); main pushes and merge queue runs, whose concurrency groups are unique per commit, always run to completion.

To run the edge image jobs on a pull request branch, with write access to the repository:

CODE
gh workflow run edge-image.yml --ref <branch>
gh workflow run edge-image.yml --ref main -f ref=refs/pull/<n>/head   # a pull request from a fork

To run the VM tests on a branch without the merge queue:

CODE
gh workflow run kvm.yml --ref <branch>

A failed kvm fails suite and CI gate, so a nightly failure reaches whoever GitHub notifies about the scheduled run: the person who last changed its cron line.

To rerun only the failed jobs of a session: gh run rerun <session-id> --failed.

Running the Docker suites inside a session

You can work on Cloudfield from inside a Cloudfield run (an agent's container on a Cloudfield server). Two things differ there.

A session sets AETHER_* variables (AETHER_RUN_ID, AETHER_HARNESS, AETHER_WORKSPACE_ID and others) that some tests read. internal/coordcli fails with them set. Unset them for the test command:

CODE
env $(env | grep -o '^AETHER_[A-Z_]*' | sed 's/^/-u /') make test

The Docker suites need a Docker daemon inside the session. On a server with Sysbox (environments.md) each session and environment terminal has its own dockerd; check with docker info. Paths the tests bind-mount resolve because the daemon and the test share one filesystem, which is not true of the host's socket mounted into a container.

CODE
make test-integration                    # Go and git are in the standard image
make android                             # needs only the docker CLI and network
curl -fsSL https://bun.sh/install | bash # make test-e2e also needs Bun on PATH
(cd web && bun install --frozen-lockfile && bunx playwright install --with-deps chromium)
make test-e2e

To check Docker inside sessions on a Sysbox host, install Sysbox with sudo sh scripts/install-sysbox.sh, then:

CODE
docker build -f images/standard/Dockerfile -t aether-standard:ci .
sudo env "PATH=$PATH" "HOME=$HOME" \
  AETHER_NESTED_DOCKER_TEST_IMAGE=aether-standard:ci \
  make test-integration INTEGRATION_PKGS=./internal/runtime \
    INTEGRATION_RUN='^TestDockerNestedDocker$'
sudo sh scripts/nested-docker-smoke.sh aether-standard:ci

TestDockerNestedDocker starts dockerd in a session container and runs a container inside it; it skips unless AETHER_NESTED_DOCKER_TEST_IMAGE names an image that has dockerd. scripts/nested-docker-smoke.sh runs the resource limit tests against a nested daemon. It pulls busybox and Python images, so it needs network access.