Which tests to run for a change, how to run each suite, what it needs, and how to read a failure. Build the tree first (Build from source); the checks every pull request sessions are in Test your change.
Which suite to run
| You changed | Run | Needs |
|---|---|---|
| Any Go code | make test |
Go |
| Server code that touches containers, git, SSH or storage | make test-integration |
Docker, git; root to match CI |
Dashboard components in web/src |
cd web && bun run typecheck && bun run test, then make public-audit |
Bun, Node 22+ |
| Something a person clicks through in the dashboard | make test-e2e |
Docker, git, Bun, Node 22+, Playwright's Chromium |
scripts/, the installers, desktop/ |
make test-scripts |
sh, Node, OpenSSL 3, jq, Go |
.github/workflows/ |
make lint-workflows |
ShellCheck |
How an agent CLI is launched (internal/harness) |
Real-agent smoke tests | Docker |
| How a session drives, reads or wakes its agent | Real agents against a mock model | Docker, git |
The assistant's instructions or tools (internal/assistant/instructions.md, internal/ops) |
Assistant eval | Claude Code signed in, spends tokens |
Native hook adapters (the .mjs files in internal/coordhooks and internal/agentstatus) |
make test-native-hooks |
Node 22.13+ |
cmd/aether-edge, internal/edge |
go test -race ./internal/edge/edgetest/ |
Go |
images/edge/Dockerfile |
The edge image | Docker, curl |
The edge store or scripts/edge-update.sh |
make test-edge-upgrade |
root, Docker with Compose and buildx, jq, curl |
images/browser/ |
make browser-image && make browser-smoke |
Docker |
android/ |
make android |
Docker |
internal/vmimage |
go test ./internal/vmimage, then make test-kvm |
e2fsprogs, qemu-utils; root for the owner test |
images/kernel/ |
The guest kernel, then make test-kvm |
Docker; a KVM host to boot it |
internal/guest (the VM helper) |
go test ./internal/guest/..., then make test-kvm |
Go; a KVM host for the VM test |
internal/runtime/vm (the VM engine) |
go test ./internal/runtime/vm, then make test-kvm |
Go; a KVM host for the VM test |
Only docs/ or root *.md |
make public-audit |
sh |
Unit tests
make test runs go test -race over every package, with -d=checkptr=0
passed through -gcflags to modernc.org/sqlite/...,
modernc.org/libc/..., modernc.org/memory and modernc.org/mathutil:
checkptr, which -race turns on, is off for the SQLite library's packages
and stays on for every other package. make test TEST_RACE= drops the race
detector and those -gcflags; the SQLite driver is transpiled C, which the
detector slows ten to twenty times (internal/store takes 26 s without it and
over five minutes with it). CI runs pull requests without -race, except
internal/store, and the merge queue and main with it.
make test-store runs internal/store alone under -race with the same
-gcflags. CI runs it on every event as the store-tests job and leaves
internal/store out of build-and-test's make test.
make test # everything
make test TEST_PKGS=./internal/store # only these packages
make test TEST_SKIP=./internal/ptyhost # everything but these
go test -race ./internal/scheduler -run '^TestName$' -vWhen you write a test:
- Open a database outside
internal/storewithstoretest.Open(internal/store/storetest). It copies a database migrated once per test binary; migrating a fresh one costs seconds under the race detector. - Call
testhome.Isolate(t)(internal/testhome) in any test that reads or writes the CLI's link config throughcli.Load,cli.Saveor a gateway handler. It setsAETHER_CONFIG_DIRand the platform home and config variables, and clearsSSH_AUTH_SOCK. Without it the test uses your real config. - Never wait for real time to pass. Give a cache or deadline an injectable clock and advance it by hand. Windows' clock is coarse enough that two reads can return the same instant.
- Give behavior that differs by platform one test per platform. CI runs the
client packages on Windows, so a test whose subject refuses on Windows goes
in a
//go:build !windowsfile with a_windows_test.gocounterpart that asserts the refusal. - Start a bug fix with a test that reproduces it, on the real user path when you can. Do not prove the same behavior again at another layer.
Integration tests
make test-integration runs the tests behind the integration build tag,
against real Docker and real git, in the packages that have them. It fails
when Docker is unreachable; it does not skip.
| Variable | Effect |
|---|---|
INTEGRATION_PKGS |
Packages to run (default: every package with integration tests) |
INTEGRATION_SKIP |
Packages to leave out |
INTEGRATION_RUN |
Passed to go test -run |
INTEGRATION_SKIP_PATTERN |
Passed to go test -skip |
CI runs the suite as root. Run it the same way to match:
sudo env "PATH=$PATH" "HOME=$HOME" make test-integration INTEGRATION_PKGS=./internal/schedulerWithout sudo, session containers use your uid, because the scheduler hands the
checkout to the container user and an unprivileged process can only hand
files to itself.
make test-integration and make test-agents run with -race. A non-empty
INTEGRATION_NORACE turns it off. CI runs the Docker suites in the merge
queue and on main only, with -race.
CI splits the suite into two jobs. To repeat one that failed:
| CI job | Variables |
|---|---|
integration (server-rest, browser) |
INTEGRATION_PKGS=./internal/server INTEGRATION_SKIP_PATTERN='^TestIntegration(Chaos|Coordination|EndToEnd|MultiMember|ServerUpdate)' with the browser image variables, after make browser-smoke |
integration (server-flows, scheduler, rest, agents, smoke) |
INTEGRATION_PKGS=./internal/server INTEGRATION_RUN='^TestIntegration(Chaos|Coordination)', then the same with '^TestIntegration(EndToEnd|MultiMember|ServerUpdate)', then INTEGRATION_SKIP="./internal/harness ./internal/server" INTEGRATION_SKIP_PATTERN='^TestDockerBrowser', then the smoke tests and make test-agents |
The merge queue and main run the Docker suites on amd64 and also run the
arm64 job: the standard-image and browser-image builds, the smoke tests,
make browser-smoke and the browser runtime tests on a native arm64 runner.
CI sets GOFLAGS=-v, so each test's duration is in the job log.
Some tests take an image from the environment:
| Variable | Used by | When unset |
|---|---|---|
AETHER_BROWSER_TEST_IMAGE |
TestDockerBrowser* and the server's browser scenarios |
aether/browser:test, which make browser-image builds (The browser companion) |
AETHER_BROWSER_IMAGE |
The companion image a server under test starts | The build's default companion image |
AETHER_NESTED_DOCKER_TEST_IMAGE |
TestDockerNestedDocker (inside a session) |
The test skips |
AETHER_ACP_IMAGE |
Enhanced-mode adapter tests (Enhanced mode), TestIntegrationAssistantIgnoresPlantedGatewayDocker and TestIntegrationAssistantIssuerMatchesTheBundledCLIs, which install the assistant's adapters from the npm registry |
The test skips, or uses busybox only |
The resource tests check real cgroup limits, PID exhaustion and OOM
containment. The capacity test starts its own dockerd and containerd in
private namespaces and never reconfigures the host's daemon. It needs root,
permission to unshare namespaces, dockerd and containerd on PATH, and
Docker API 1.46 or newer:
sudo env "PATH=$PATH" "HOME=$HOME" go test -race -tags=integration \
./internal/runtime \
-run '^TestDocker(CapacityContainerdIsolated|ResourceLimits|PIDLimitEnforced|OOMContained)$' -vTests in internal/runtime, internal/scheduler and internal/harness
label their containers aether.test=<test name>. After an interrupted session,
docker ps -a --filter label=aether.test lists what was left behind.
The E2E scenario suite
The *_integration_test.go files in internal/server start the whole
server and drive it over real SSH and git transport, with real containers
when Docker is reachable. Without Docker most scenarios fall back to an
in-process runtime. The chaos scenarios, which run aether-server as a child
process and kill it with SIGKILL, and the in-container coordination
scenario skip instead.
The agents are scripted. The fake harness runs the command in
AETHER_FAKE_AGENT (typically sh /workspace/agent.sh {task}, with the
script committed to the test repository).
| Test | Covers |
|---|---|
TestIntegrationEndToEnd |
One member: push, launch, attach, detach, steer, finish, pull |
TestIntegrationMultiMember |
Joining, roles, handoff, budget caps, an agent crash |
TestIntegrationChaos* |
The server killed mid-session, disk pressure, a stalled agent |
TestIntegrationCoordination* |
Messages between sessions |
TestIntegrationServerUpdate* |
aether-server update: preflight, rollback, downgrade guards |
TestIntegrationRestart* |
Enhanced sessions through server restarts and updates (Restart acceptance) |
TestIntegrationServerGateway |
The dashboard the server hosts on the tailnet |
TestIntegrationConnectorFlows |
A ChatGPT connector, over OAuth and MCP, launching, messaging, answering and stopping enhanced sessions |
sudo env "PATH=$PATH" "HOME=$HOME" make test-integration \
INTEGRATION_PKGS=./internal/server INTEGRATION_RUN='^TestIntegrationEndToEnd$'Restart acceptance
internal/server/restart_integration_test.go checks that Enhanced sessions keep
working through a server restart or update, in three sizes:
TestIntegrationRestartSmokeruns in CI, in about 90 seconds. Two Enhanced sessions, A streaming a numbered line every 100 ms and B holding a permission request, each run a probe process started withdocker exec(not part of the agent). The probes call the server through the staged CLI and MCP bridge and exchange mail, keeping aninbox --wait 20long poll open. Just before each of twoSIGTERMs and oneSIGKILL, each probe fires keyedsends through both the CLI and the bridge, and one unkeyeddev.terminal.input.TestIntegrationRestartCoordinationmeasures the coordination socket alone with one external probe.TestIntegrationRestartScalingrepeats twoSIGTERMrestarts with 2, 4 and 7 live Enhanced sessions and prints the stream delay of each; it skips unlessAETHER_RESTART_FULL=1.TestIntegrationRestartAcceptanceis the full matrix for a release or an owner check; it skips unlessAETHER_RESTART_FULL=1.
The full matrix builds aether-server twice (the second reports v0.0.0-soak-b), runs one as a
child process against real Docker with a temporary data directory, and launches
three Enhanced sessions of a scripted agent (internal/acphost/acpmock):
- Sessions A and B each start a 120-second shell command that appends a line a
second to a marker file, then for 105 seconds call the server through the
staged
aether-internalCLI and the stagedaether-server mcpbridge (internal/server/testdata/soakagent), about one request a second in all. A sends mail to B with the CLI and reads with the bridge; B does the reverse. Half of A's CLI sends omit the idempotency key; the bridge requires one. Each side keeps oneinboxlong poll in flight. The agent streams a numbered line every 100 ms as its answer. - Session C holds a permission request open until the end.
The test then restarts the server five ways, 20 seconds apart: SIGTERM and
start the same binary twice, replace the binary with the second build, put the
first build back (the rollback case), and SIGKILL. Every restart lands while
a call, an inbox long poll and a send are in flight. The swap and rollback stop
the server with SIGTERM and replace the file, as aether-server update does;
the exec step itself is covered by TestIntegrationServerUpdate*.
It fails on any of:
- a turn interrupted or a run parked with
turn interrupted; - a failed CLI or bridge call the agent saw;
- a send that failed but was delivered, unless its error says it "may already have applied", or a message delivered twice;
- an unkeyed terminal input applied more than once, or its echo not read
from the output cursor held before the restart without
missing_cursorortruncated; - no terminal input accepted after the first restart with the control lease the probe acquired before it;
- a sent message that was never received, or was received twice;
- a streamed line missing or repeated;
- the marker file not holding lines 1 to 120, or the container's ID or start time changing;
- the permission request lost, or its answer not reaching the agent.
It prints, per restart, the milliseconds between the last success before the
signal and the first success after it for each channel (CLI and bridge calls
on both runs and the aether-acp stream). Status calls run every
2 seconds (4 in the smoke test), because the coordination socket allows one request a second per
run, so a gap under about one interval is the probe interval and means no outage.
sudo env "PATH=$PATH" "HOME=$HOME" AETHER_RESTART_FULL=1 make test-integration \
INTEGRATION_PKGS=./internal/server INTEGRATION_RUN='^TestIntegrationRestart'Both use busybox:1.36, which must be pulled; the full matrix takes about four minutes.
Real-agent smoke tests
internal/harness/smoke_integration_test.go starts the vendors' real agent
CLIs to check that the command lines Cloudfield builds are still accepted. A
vendor renaming a flag breaks every session of that agent, and nothing else
catches it.
TestSmokeHeadlessNoLogin runs in CI. It launches each headless template
with no credentials and passes when the session fails for want of a login, never
for an unparseable command line. CI's integration (server-flows, scheduler, rest, agents, smoke)
job builds the standard image, adds the agent CLIs at their current versions with
images/smoke/Dockerfile,
and sessions:
docker build -f images/standard/Dockerfile -t aether-standard:local .
docker build -f images/smoke/Dockerfile --build-arg BASE=aether-standard:local \
-t aether-smoke:local .
AETHER_SMOKE_IMAGE_CLAUDE_NOLOGIN=aether-smoke:local \
AETHER_SMOKE_IMAGE_CODEX_NOLOGIN=aether-smoke:local \
AETHER_SMOKE_IMAGE_OPENCODE_NOLOGIN=aether-smoke:local \
go test -tags integration -run TestSmokeHeadlessNoLogin ./internal/harness/A _NOLOGIN image must carry no credentials. TestSmokeClaude,
TestSmokeOpencode and TestSmokeCodexFlags drive the agent far enough to
produce output, so they need an image that carries that agent's login, named
by AETHER_SMOKE_IMAGE_<NAME> (for example AETHER_SMOKE_IMAGE_CLAUDE). CI
has no such credentials, so they are a manual check. Each test skips when its
variable is unset. On failure a test prints the agent's output up to that
point.
pi and omp have no smoke test in that image.
Enhanced mode
Enhanced sessions drive the agent through an adapter that speaks the Agent
Client Protocol (ACP). Most tests use internal/acphost/acpmock, which replays
conversations recorded from the real adapters and answers prompts by their
text (ask permission, ask form, wait, hit the usage limit, demo,
demo edit):
make test-integration INTEGRATION_PKGS='./internal/scheduler ./internal/server' \
INTEGRATION_RUN='TestIntegrationEnhancedRun'TestACPAdapterInstall installs the pinned Claude Code and Codex adapters
into the standard image and completes the ACP handshake with each. It needs
the npm registry. With ACP_LIVE=1 it also opens a session, which needs no
login (a logged-out Codex skips):
AETHER_ACP_IMAGE=aether-standard:local \
go test -tags integration -run TestACPAdapterInstall -v ./internal/harness/Two tests run the agents and adapters on your PATH with your own logins.
TestLiveSwitch switches a session between Standard and Enhanced both ways;
ACP_LIVE_OMP_MODEL picks omp's model when its default cannot answer.
TestLiveEnhancedRunsWithoutPermissionPrompts fails if an Enhanced session
asks for permission to run a shell command:
ACP_LIVE=1 go test -tags integration -run TestLiveSwitch -v ./internal/harness/
ACP_LIVE=1 go test -tags integration -run TestLiveEnhancedRunsWithoutPermissionPrompts -v ./internal/harness/Real agents against a mock model
TestIntegrationRealAgents (internal/server/realagents_integration_test.go)
launches real Claude Code and Codex sessions, Standard, Enhanced and Background,
on real Docker, and checks that each report an agent makes, or fails to make,
leaves the session in the state the dashboard shows. A mock model in the test
process (internal/server/mockmodel_integration_test.go) serves streaming
Anthropic /v1/messages and OpenAI /v1/responses. Each prompt the test
sends carries a tag, aether-script:<turn>;, naming the script the mock
plays for that turn: one tool call per step, then a text answer. The mock
answers each report reminder Cloudfield sends with aether-internal report.
One Standard and one Enhanced session per agent play these turns in order. The
launch starts turn, mail starts wake, and run.inject starts the rest:
| Turn | The agent | The session must show |
|---|---|---|
turn |
runs a command and ends the turn without a report | answered after one reminder; Standard Codex, which Cloudfield cannot remind, waiting |
wake |
gets mail while idle | a new turn, where Cloudfield can wake the session |
waiting |
reports waiting --for ci |
waiting-ci with the summary |
answered |
reports answered |
answered, not Needs you |
late-report |
reports waiting --for process from a detached process five seconds later |
the report landing after the session parked, then waiting-tools |
compaction |
claims more tokens than its context holds, compacts, then reports | the reporting rules still in the next request, then answered |
subagent |
starts a subagent that runs for eight seconds | not Needs you at any point while it runs, the subagent never reminded, then the turn ends like turn |
needs-input |
reports needs-input |
needs-you, Needs you |
Each check also reads the session's cause, its report entries on the timeline,
and whether the report's output reached the model. A failure names the broken
link: the reporting rules missing from a request (instruction missing), no
report output reaching the model (report never made), a report missing from
the timeline or its output missing from the model's requests (report lost),
or a report recorded but shown as the wrong state (misclassified). The
first turn and every Background session also require the launch to record where
its instructions went.
The wake turn runs where Cloudfield can wake the session: every Enhanced session, and a
Standard session whose agent sets IdleWake in its capabilities
(internal/harness/capabilities.go); the others skip it. A session must never
need you while its subagent works. It should show working, or
waiting-agents when Standard Claude Code ends the turn with the subagent
still running in the background. Cloudfield gets no
signal for a Codex subagent, and Standard Claude Code's Stop hook can arrive
without the subagent among its background tasks, so those sessions can read as
idle; the test logs their idle states as a known gap and fails any other.
Two Background sessions per agent check that a Background session is never reminded:
one reports success and finishes awaiting review, the only Background state
that needs you; the other has its needs-input report refused and finishes
without a report.
TestIntegrationSilentModelParksAsPossibleStall
(internal/server/liveness_integration_test.go) launches an Enhanced Claude
Code run against a model that accepts every request and never answers. It
requires the session to park as a possible stall awaiting its first event, and
never to read disconnected.
TestIntegrationClaudeEnhancedSleepsAndWakes
(internal/server/sleep_integration_test.go) runs a server with
--run-sleep-after 5s. An Enhanced Claude Code session answers one turn and
falls asleep; the test requires its container to be stopped, not removed, and
the session still live. A prompt then wakes it, and the model's next request
must carry the turn from before the sleep. An agent gets
Capabilities.Sleep only with a test like this one.
Sessions reach the mock the way they reach a model
gateway: ANTHROPIC_BASE_URL for Claude Code,
openai_base_url in ~/.codex/config.toml for Codex. The test seeds the
member home with what a member's first sessions leave there: onboarding, an
approved API key and a trusted checkout. The Codex config also sets
model_auto_compact_token_limit, so the compaction turn's claimed usage
makes Codex compact.
CI's integration (server-flows, scheduler, rest, agents, smoke) job runs both as root. Run them the same way, with Docker and
git:
make agents-image # images/agents/Dockerfile on the published standard image
sudo env "PATH=$PATH" "HOME=$HOME" make test-agentsAGENTS_BASE=aether-standard:local builds on a standard image you built. A
failure prints the broken link, the session's report entries, the agent's screen
(Standard) or ACP history (Enhanced), and the final item of each request the
mock received.
Bumping the pinned versions
images/agents/Dockerfile pins the CLIs in CLAUDE_CODE_VERSION and
CODEX_VERSION; the ACP adapters follow internal/harness/acpregistry.json,
the versions members install. The smoke image takes each vendor's latest
release; this one is pinned so a vendor release cannot fail an unrelated pull
request.
scripts/bump-agent-pins.sh moves every pin to the newest release and prints
each one it moved: the adapters in acpregistry.json and
internal/harness/acp.go to the versions the
ACP registry
lists, each adapter's CLIVersion to the CLI that release installs, the
two CLIs above to npm's latest, and the oh-my-pi release session VMs' auth
broker runs, in internal/serversetup/omp.go, to the newest GitHub release
with its SHA-256 and size. It needs curl, jq and npm. Run the two commands
above on the result, and fix what the new releases broke in the same pull
request.
The Agent pins workflow (.github/workflows/agent-pins.yml) runs the
script every Monday at 06:00 UTC and on gh workflow run agent-pins.yml, and
opens or updates one pull request from bot/agent-pins. Pull request CI runs
the unit tests; the merge queue runs the real-agent suite on the new versions.
The workflow never replaces a pull request that carries a commit anyone else
authored or committed, so push a fix for a failing bump to that branch.
The workflow pushes as a GitHub App, because a pull request opened with the workflow's own token starts no CI. To set it up once, an organization owner:
- Creates a GitHub App with no webhook and the repository permissions
Contents, Pull requests and Workflows set to Read and write.
GitHub refuses an App's push that changes a workflow file without
Workflows, and the branch rebuilt on a newer
mainchanges every workflowmainchanged since the last run. - Installs it on this repository only.
- Saves its App ID as the repository variable
AGENT_PINS_APP_IDand a private key as the repository secretAGENT_PINS_APP_KEY.
Without them, each run fails with set the AGENT_PINS_APP_ID variable and the AGENT_PINS_APP_KEY secret.
The assistant installs the adapters at the versions in acpregistry.json, and
Codex at the CLIVersion next to its pin (internal/harness/acp.go). It
refreshes its sign-ins with the endpoints and client ids in
internal/scheduler/assistant_container.go. CI's integration job checks them
with the test below; after moving any of them, run it yourself:
AETHER_ACP_IMAGE=aether-standard:local \
go test -tags integration -run TestIntegrationAssistantIssuerMatchesTheBundledCLIs -v ./internal/scheduler/Assistant eval
internal/assistant/eval measures what the built-in assistant does with
internal/assistant/instructions.md. It runs the real Claude Code CLI (claude -p)
in a directory whose CLAUDE.md is those instructions. The agent's only tools
are the assistant's own, ops.RegisterAssistant, served over HTTP MCP by a fake
control surface that answers from synthetic scenario state. The fake records
every tool call. A confirmed-tier call (launch, message, merge) is recorded and
answered with a made-up success, so nothing is ever started, sent or merged;
the eval scores the parameters the agent proposed.
It spends model tokens, so it is behind the assistanteval build tag and no CI
job runs it. Claude Code must be on PATH and signed in:
AETHER_ASSISTANT_EVAL_LABEL=main \
go test -count=1 -tags assistanteval -run TestAssistantEval -timeout 40m ./internal/assistant/eval| Variable | Default | Meaning |
|---|---|---|
AETHER_ASSISTANT_EVAL_LABEL |
unlabeled |
Names the output files; use the commit or branch under test |
AETHER_ASSISTANT_EVAL_RUNS |
3 |
Sessions per scenario |
AETHER_ASSISTANT_EVAL_MODEL |
Claude Code's default | Passed to claude --model; the report names the model the harness reports |
AETHER_ASSISTANT_EVAL_ONLY |
all | Comma-separated scenario ids |
AETHER_ASSISTANT_EVAL_PARALLEL |
4 |
Concurrent agent sessions |
AETHER_ASSISTANT_EVAL_OUT |
$TMPDIR/assistant-eval |
Where <label>.md and <label>.json go |
The scenarios and their expected checks are in
internal/assistant/eval/testdata/scenarios.json: a rough voice request, a
question, a research-only request, a prohibition, hostile pasted text, a saved
preference against a one-off override, two members with different defaults, a
launch with no dashboard view, a shared account, an unavailable saved target,
merged against released, a failing PR behind a success report, and a reopened session. The checks are keyword and
tool-call tests, so they are deterministic but coarse: text matches are whole
words (so bo does not match both), an invented-scope keyword does not count
when the brief negates it ("no refactors"), and the same holds for words a brief must omit ("do not delete staging" does not copy the hostile instruction), and a question counts only when it
is a sentence addressed to the member, not a ?run= link or inline code. A
question after a completed launch that offers to show or edit the brief is
reported in its own column, not as unnecessary. The Markdown report ends
with a manual-review list per scenario, with each launch brief and reply.
The report has a row per scenario: sessions passed, unnecessary questions, brief offers, launches
that should not have happened, dropped prohibitions, invented-scope keywords,
tool calls and latency per session, and cost. Cost is the figure claude reports,
and "not measured" when it reports none. Results are rewritten after every
session, and each session's timeout is capped by the test's -timeout, so a session that
hits the limit still leaves a report.
A scenario marked pending: <tool> needs a tool the registry does not offer
yet, such as the saved launch choices. The eval serves an eval-only stub of the
tool from the scenario's contract so the baseline is honest about what the
agent does without it. Its rate is not product behavior. When the registry
offers the tool, the real one is served instead, and the scenario's answers
need an entry for each RPC method it calls; a missing one shows up as
"is not available in this synthetic scenario" in the reply.
Compare a change with main by running the command on both commits with
different labels. Model output varies between sessions, so a few sessions per scenario
shows a direction, not a proof. Scripted tests of the scorer itself,
go test ./internal/assistant/eval, run in make test.
Native hook tests
The pi, omp and opencode adapters are JavaScript that Go never runs.
make test-native-hooks runs them under Node 22.13 or newer against
lifecycle fixtures shaped like each agent's SDK:
make test-native-hooks
node --experimental-vm-modules --test --test-name-pattern='omp:' \
internal/coordhooks/native_pi_omp_lifecycle_test.mjsinternal/agentstatus also runs status.ts under bun from go test, and
skips those tests where bun is not installed.
The fixtures do not prove a real model turn. To check one, start an isolated
coordinated Standard session with the exact CLI, server and extension versions under
test, end a turn idle, send it one message with
aether-internal send --to <session> --body <nonce>, and confirm exactly one new
turn reads and acknowledges it. Never edit a live hook, credentials or trust
settings to get there.
The edge suite
internal/edge/edgetest runs edge remote access end to end in one process,
under plain make test. It needs no Docker and no network. Each test starts a
real edge against a fake GitHub, real servers under each access policy, the
real client and the local gateway. A proxy in front of the edge holds the
edge's signing key and plays a compromised edge: it forges, replays, alters
and drops control messages, and tests read each server's store to see what an
attack changed.
go test -race ./internal/edge/edgetest/
go test -race ./internal/edge/edgetest/ -run '^TestClaim$' -vEvery request reaches the edge from 127.0.0.1, so each test moves its edge's
clock forward to refill the rate limits. Tests run in parallel; the client
calls that read AETHER_CONFIG_DIR take turns.
Behind nginx
scripts/edge-nginx-test.sh, part of make test-scripts, runs a real nginx
from
packaging/nginx/aether-edge.conf.example
in front of the edge, and skips when nginx is not on PATH. It checks
sign-in, a claim, a relayed connection held silent past nginx's 60-second
read timeout, and that a request with a forged X-Forwarded-For is recorded
under the address nginx saw. It takes about 80 seconds:
sh scripts/edge-nginx-test.shThe edge image
scripts/edge-image-smoke.sh runs a built
images/edge/Dockerfile image with fake GitHub
credentials. It checks that the image runs as a non-root user with no shell,
fails to start without configuration, passes its healthcheck, stops cleanly
on SIGTERM without logging the secret, and keeps its key on its volume. It
needs a container runtime, so it is not part of make test-scripts. The
second argument names the runtime (default docker):
make edge-image
sh scripts/edge-image-smoke.sh aether/edge:testWith podman, build with podman build --format docker -f images/edge/Dockerfile .. The default OCI format drops HEALTHCHECK and the
smoke test fails.
The image builds with its own golang base image, so
scripts/edge-go-version-test.sh, in make test-scripts, fails when that
image and go.mod's toolchain line name different Go versions:
edge-go-version-test: images/edge/Dockerfile builds with golang:1.26.8, but go.mod pins go1.26.9; change the golang tag and its @sha256 digest in images/edge/Dockerfile to 1.26.9 (or go.mod's toolchain line to go1.26.8)Edge upgrade qualification
TestEdgeUpgrade updates real edge containers with
scripts/edge-update.sh, from the releases
edge hosts run to builds of this tree. It needs root (the data directory
belongs to uid 65532), Docker with Compose and buildx, jq, curl, Go,
network access to github.com, and git history with tags:
sudo env "PATH=$PATH" "HOME=$HOME" make test-edge-upgradeIt starts from each release the signed stable.json and beta.json
pointers name, and with EDGE_QUALIFY_CHANNEL=nightly, which
release-build.yml sets for a nightly, also from the one nightly.json
names. While no pointer exists it starts from the previous tag
(git describe --tags HEAD^). Each starting release is built in a
git worktree into a local registry, runs from
packaging/edge/compose.yaml behind nginx, has real servers enroll and be
claimed, and is updated to this tree as v99.0.1. From the first starting
point the test goes on through three more builds of this tree with an extra
migration: one that fails, one that raises the compatibility floor and then
fails, and one that succeeds. It checks that:
aether-edge-updaterefuses an image it cannot read a version from.- A failing migration is rolled back without a restore and skipped next time.
- A failure after a raised floor restores the pre-update backup.
- A good migration applies, with a backup from the edge and the updater.
- A server of the starting release still enrolls and relays.
aether-edge-update --rollbackreturns to the old binary.
After each step the edge must keep its key and its accounts, and each owner
must reach their server. Stub cosign and docker commands from
internal/edge/edgetest/testdata/upgrade/ answer the promotion checks, and
the release-check of a starting release, whose binary trusts only the
release keys built into it. This tree's builds are stable releases at
build 9000001 and up that also trust a test key. Their own release-check
verifies the manifests the test signs with it, which the fake GitHub
serves. From step 3 on, the fake answers 404 for the running release's
manifest, so the updater orders by the copy it kept.
It refuses to start while a container named aether-edge exists, removes
everything it creates, and takes about two and a half minutes with warm
caches, plus about a minute for each further starting point; this tree's
builds are made once for all of them. CI runs
it in edge-image.yml in the merge queue and on main, and in
release-build.yml before a release publishes.
Dashboard component tests
Component tests live beside their components in web/src/ and run in vitest
with jsdom:
cd web
bun run test # all of them
bun run test src/routes/files/home.test.tsxjsdom has no layout, so web/src/test/setup.ts answers every media query
with false and a component renders its widest layout. atViewport
(web/src/test/viewport.ts) puts one test on one screen size. Real layout,
hit testing and focus outlines belong to the
dashboard end-to-end suite.
The dashboard end-to-end suite
web/e2e/ drives Chromium against the dashboard embedded in dist/aether,
through a real aether gui gateway and a real aether-server over SSH.
Playwright is the runner.
(cd web && bunx playwright install chromium) # once, from the repo root
make test-e2e # builds dist/ first
make test-e2e E2E_ARGS='--shard=1/2' # one of CI's two shards
make test-e2e E2E_ARGS='--project=chromium run-files.spec.ts'
cd web && bunx playwright test --project=mobile # reuse the binaries already in dist/make test-e2e rebuilds the dashboard and binaries because the CLI serves
the dashboard it embeds: a stale binary tests a stale dashboard.
There are two projects. chromium runs every spec except
*.mobile.spec.ts on a desktop viewport. mobile runs only those, as a
Pixel 7 (touch, 412x839). Mobile specs tap instead of clicking and attach a
full-page screenshot to every test. WebKit is not installed, so iOS Safari is
not covered.
Each test gets its own stack:
- One
aether-serveron its own loopback port, with a temporary data directory, thebusybox:1.36image andAETHER_FAKE_AGENT. The first identity to connect becomes the admin. - One
aether guiper member, each with its ownHOMEandAETHER_CONFIG_DIR, so nothing touches your own keys or links. - Real git repositories on disk.
aether.server.restart() kills the server with SIGKILL and starts it
again on the same port and data directory, for specs about what survives a
crash.
Specs that open a real session need Docker and skip without it.
onboarding-github and remote-development-git/native.spec.ts fail
without Docker instead. Some specs need more:
| Spec | Needs |
|---|---|
onboarding-github, and native, git-card and github in remote-development-git/ |
The real standard image: docker build -f images/standard/Dockerfile -t aether-standard:ci ., or another build named by AETHER_E2E_STANDARD_IMAGE |
development-browser/* |
The browser companion image (below) and node:22.14.0-alpine3.21 |
remote-development-git/import.spec.ts |
Network access to AETHER_E2E_GIT_SOURCE_URL (default https://github.com/3xDevOps/cloudfield.git) |
remote-development-git/github.spec.ts |
Opt-in: AETHER_E2E_GITHUB_PUBLISH=1, AETHER_E2E_GITHUB_TOKEN, AETHER_E2E_GITHUB_BASE_REPOSITORY (a public owner/name), AETHER_E2E_GITHUB_HEAD_REPOSITORY (your writable fork); AETHER_E2E_GITHUB_BASE_BRANCH defaults to main |
github.spec.ts opens, comments on and closes a real pull request upstream.
Pass the token through the environment only, never in a file or a log.
To run as root like an installed server:
sudo env "PATH=$PATH" "HOME=$HOME" \
"PLAYWRIGHT_BROWSERS_PATH=$HOME/.cache/ms-playwright" \
AETHER_E2E_STANDARD_IMAGE=aether-standard:ci \
AETHER_BROWSER_TEST_IMAGE=aether/browser:test \
AETHER_BROWSER_IMAGE=aether/browser:test make test-e2eReading a failure
web/playwright-report/is the HTML report; open it withcd web && bunx playwright show-report. A failed test has the server's output attached.web/test-results/holds traces and screenshots.- A failed test keeps its scratch directory,
/tmp/aether-e2e-*. After a session as root it holds root-owned files; remove it withsudo rm -rf. - In CI each shard uploads
playwright-report-<shard>andplaywright-results-<shard>for seven days, on success and failure:gh run download <session-id> -n playwright-report-1.
A new onboarding wizard step is one class in web/e2e/pages/wizard.ts: give
it the aria-label of the step's <section>, add its label to stepNames
in header order, and hang it off OnboardingWizard.
The browser companion
A session's Browser tab is a Chromium in its own container, the companion,
built from images/browser/. It needs no display, host browser or host
browser libraries, but it does need a working kernel user-namespace and
AppArmor policy on Ubuntu. A sandbox failure must fail the check: do not
retry with --no-sandbox or relax the host's security policy.
docker build -f images/standard/Dockerfile -t aether-standard:ci .
make browser-image # aether/browser:test; BROWSER_IMAGE=<ref> for another tag
make browser-smoke # runs the image's own sandbox and behavior checks
sudo env "PATH=$PATH" "HOME=$HOME" \
AETHER_BROWSER_TEST_IMAGE=aether/browser:test \
go test -race -timeout=10m -tags=integration ./internal/runtime \
-run '^TestDockerBrowser' -v
AETHER_BROWSER_TEST_IMAGE=aether/browser:test \
AETHER_BROWSER_IMAGE=aether/browser:test make test-integrationRun the runtime test as root, like the installed server. Do not make its control directory world-writable to get a pass.
Committing selected paths from the Changes view uses native git and gh.
Their suite runs against a local TLS stand-in for GitHub:
go test ./internal/runrepoAndroid
make android runs the app's JUnit tests, then builds the APK and the app
bundle, all in pinned containers; make android-debug builds a debug APK
without the tests. Building, signing and installing are in
Android shell.
scripts/android-version-code-test.sh and
scripts/android-verify-signature-test.sh, both in make test-scripts,
cover the version code and the signature check with the SDK tools stubbed.
The guest kernel
make kernel builds dist/aether-vmlinux-x86_64, the kernel session VMs
boot, from the version and checksum in images/kernel/build.sh and the
options in images/kernel/config-x86_64. It runs in a Debian container pinned
by digest, with packages from a dated snapshot, so two builds give the same
SHA-256; compare them after changing either file. The build fails, naming
the options, when a line of the config does not reach the kernel's
.config because something it depends on is off.
On a KVM host, TestKVMStandardImageRunsDocker (VM tests)
checks that a change still boots an image and runs Docker in it: it builds
the image's disks with internal/vmimage, boots them with the kernel, starts
dockerd, runs a container and a -p port mapping, and refuses to save the
root disk while the VM runs.
Docker's contrib/check-config.sh from the moby repository, pointed at the
config (/proc/config.gz in the guest), must report every
Generally Necessary option enabled.
go test ./internal/vmimage covers layer extraction and the image store
with real mkfs.ext4, debugfs, qemu-img and qemu-io; it skips the tests whose
tools are missing. Run it once as root too, as CI's build-and-test does:
only root can keep other owners, set-user-ID bits and file capabilities, as
the server does.
The VM helper
go test ./internal/guest/... runs the real helper service against the real
client over a Unix socket that answers Cloud Hypervisor's vsock handshake:
processes on pipes and terminals, reattaching from a new client, lossless
output, spawn failures, signals, orphan reaping and Boot. What only PID 1
in a VM does (switching root, mounts, the network, socket forwarding, the
process limit, an out-of-memory kill) needs a KVM host. TestKVMHelper
(VM tests) boots an image from internal/vmimage disks with the
test binary as the helper and drives it through the client.
VM tests
Tests that boot real Cloud Hypervisor VMs carry the kvm build tag, so
make test and make test-integration never build them. make test-kvm
runs every package that has them:
| Package | Test | Boots |
|---|---|---|
internal/vmimage |
TestKVMStandardImageRunsDocker |
The standard image from store disks, with Docker inside (The guest kernel) |
internal/guest/helper |
TestKVMHelper |
The helper as PID 1, driven through its client (The VM helper) |
internal/runtime/vm |
TestKVM... in each kvm_*_test.go |
Sessions through the engine, with real systemd units and the server binary as monitor and helper: lifecycle, save and restore, a guest reboot, the home, the checkout, the network, Save environment |
It needs root on a Linux x86-64 host with /dev/kvm, cloud-hypervisor,
qemu-img, mkfs.ext4, virtiofsd at /usr/libexec/virtiofsd, systemd,
iptables and network access to pull images. sudo aether-server vm install --vm-kernel dist/aether-vmlinux-x86_64 installs them the way a server does,
as CI's kvm job does. make test-kvm fails, naming what is missing,
rather than skip. It runs only the tests named TestKVM...: the
unit tests already run in make test, and the helper's reap every child of
the test binary, the VM monitor's too. Build the kernel first:
make kernel
sudo env "PATH=$PATH" "HOME=$HOME" make test-kvm
sudo env "PATH=$PATH" "HOME=$HOME" make test-kvm KVM_RUN='^TestKVMHelper$' KVM_SKIP='./internal/vmimage ./internal/runtime/vm'| Variable | Default | Meaning |
|---|---|---|
KVM_KERNEL |
dist/aether-vmlinux-x86_64 |
The guest kernel |
KVM_IMAGE |
ghcr.io/3xdevops/aether-standard:latest |
The image the store and helper tests boot; the engine tests boot ubuntu:24.04 |
KVM_RUN |
^TestKVM |
Passed to go test -run |
KVM_SKIP |
none | Packages to leave out |
The target builds the test binaries and aether-server with
CGO_ENABLED=0: the binaries also run inside the VMs, where there are no
libraries. It builds them, and keeps the disks, in a new private directory
under $TMPDIR (default /tmp) that it removes at the end: root runs that
server, so no other user may be able to replace it. A test's VM gets at most 2 GiB; on a CI runner the tests take
about three minutes.
On a KVM host without Go, build on your machine, copy the directory over,
and run each test binary there with the variables make test-kvm sets:
CGO_ENABLED=0 go test -c -tags kvm -o kvm/ ./internal/vmimage ./internal/guest/helper ./internal/runtime/vm
CGO_ENABLED=0 go build -o kvm/aether-server ./cmd/aether-server
cp dist/aether-vmlinux-x86_64 kvm/
scp -r kvm <host>:
# on the host, in kvm/; then the same for helper.test and vmimage.test
sudo env AETHER_TEST_KERNEL="$PWD/aether-vmlinux-x86_64" \
AETHER_TEST_IMAGE=ghcr.io/3xdevops/aether-standard:latest \
AETHER_VM_TEST_KERNEL="$PWD/aether-vmlinux-x86_64" AETHER_VM_TEST_DIR="$(sudo mktemp -d)" \
AETHER_VM_TEST_SERVER="$PWD/aether-server" ./vm.test -test.run TestKVM -test.vA test that is interrupted can leave a VM behind: systemctl list-units 'aether-vm-*' lists the engine's, and pgrep -a cloud-hypervisor the
others.
Windows
CI's windows job builds, vets and tests the client packages on Windows.
The windows-install workflow runs scripts/install-test.ps1 under Windows
PowerShell 5.1 and PowerShell 7. You can run it on your own Windows machine
with Go installed; it restores your PATH and environment when it ends:
powershell -NoProfile -File scripts/install-test.ps1
pwsh -NoProfile -File scripts/install-test.ps1scripts/install-smoke.ps1 and scripts/update-smoke.ps1 install, update
and launch the real app under Defender. They change Defender settings, the
Start Menu and the registry, so they refuse to run unless
GITHUB_ACTIONS=true and RUNNER_ENVIRONMENT=github-hosted. When one
fails, scripts/defender-diagnostics.ps1 prints Defender's detections in the
job log. The screenshots are uploaded as windows-desktop-* artifacts.
When update-smoke.ps1 cannot move aether.exe because another process
holds it open, it prints the processes Windows Restart Manager names and
waits up to 30 s for them to close it. It fails if Restart Manager names no
process, or if the move still fails after the wait.
Before they run, scripts/windows-defender.ps1 -Mode Prepare removes the
runner image's Defender exclusions and adds one for GitHub's own agent,
provjobd.exe* in the runner user's Temp. On windows-11-arm, Defender
flags that agent as Trojan:Win32/Gracing.I and removes it; the job then
hangs until GitHub fails it with "The hosted runner lost communication with
the server". Nothing Cloudfield builds or installs is excluded.
How CI runs the suites
The CI workflow (.github/workflows/ci.yml) runs on every pull request,
every merge queue entry, every push to main or a release/X.Y hotfix
branch and nightly on main at 08:00 UTC. A pull request, draft or ready, runs changes, audit, lint,
build-and-test, store-tests and dashboard. Every other job lives in
.github/workflows/ci-suite.yml, which ci.yml calls as the suite job, so
a run that skips them shows one skipped suite check; in a run that calls it
each job shows as suite / <job>. The merge queue runs every job, so the full
suite runs once per change, on the merge result, before main moves; a
failure there removes the pull request from the queue. The one exception is
kvm, which boots VMs: the queue runs it only when the change can affect
the VM tests (the vm mode below), and the nightly run always does.
The queue moves main to the commit it tested, so a push to main reuses
that verdict: scripts/ci-queue-verdict.sh finds the merge queue run on the
same commit that completed with conclusion success and whose CI gate
passed, and the push runs only changes and CI gate, whose summary links
that run. The push runs every job when no such run exists, when a lookup
fails, when the push touches 300 files or more (GitHub's compare limit), or
when it changes a file a cache key hashes (ci-classify-areas.sh caches).
Caches save only from runs on main, because a merge queue run's caches are
scoped to its own queue branch; the nightly run refreshes them, so between
nightly runs a session restores compiler caches up to a day old. It also
tests main against the newest agent CLIs and the latest published release;
its result gates nothing, so read a red nightly CI gate on the main tip
yourself. A merge queue run re-run after its push reused it does not change
that push's verdict: re-run the push session too. A push to a release/X.Y
branch never reuses a verdict and runs every job, the VM tests included; a
hotfix release reads that run.
CI gate is the one required check: it passes when
every other job passed or was skipped by its own rule, and fails when any
failed or was cancelled. In the merge queue no rule skips a job, so there it
passes only when every job passed: a cancellation can leave a job skipped. A
new job in ci.yml must be added to its needs list to gate main; a job
in ci-suite.yml gates it through suite.
main takes changes through a merge queue. gh pr merge <n> --squash adds a
ready pull request whose CI gate passed; the queue batches up to five
entries, runs the full suite on each batch's merge result, and moves main
only when that run's CI gate passes. Each queue run's changes job first
cancels the runs of groups the queue has discarded, so they stop holding
runners. In the Docker-heavy jobs, the runner's Docker daemon and
the buildx builders pull Docker Hub images through mirror.gcr.io and fall
back to Docker Hub, so a Docker Hub outage alone does not fail them.
| Job | Sessions | Locally |
|---|---|---|
audit |
scripts/public-audit.sh |
make public-audit |
lint |
Formatting, workflow lint, make lint (golangci-lint, which includes go vet and shadow), then the script and native hook tests; make vulncheck as advice on every session that runs the full suite |
make fmt-check lint-workflows lint test-scripts test-native-hooks |
build-and-test |
Unit tests except internal/store (-race on the merge queue and main only), then make build |
make test TEST_SKIP=./internal/store build |
store-tests |
internal/store's unit tests, always with -race |
make test-store |
integration (server-rest, browser), integration (server-flows, scheduler, rest, agents, smoke) |
Integration tests, the smoke tests, real agents against a mock model and the browser companion, on amd64 | See the table under Integration tests |
arm64 |
Standard and browser image builds, smoke tests and browser runtime tests on arm64 | Same commands on an arm64 host |
dashboard |
bun run typecheck and bun run test |
Same, from web/ |
dashboard-e2e |
End-to-end suite in two shards | make test-e2e E2E_ARGS='--shard=N/2' |
windows, windows-install |
Windows; windows also scans release-build's Windows binaries with Defender, and the windows-install install (zip) and update legs package and launch the Windows desktop zip on amd64 and arm64 |
|
release-build (binaries, desktop, android) |
make release-binaries, aether gui build on Linux with and without Node on PATH, make android |
make release-binaries, Desktop shell, make android |
edge-image |
The edge image and qualification | |
kvm |
VM tests on a hosted runner, which has /dev/kvm (.github/workflows/kvm.yml) |
make kernel, then make test-kvm as root |
A pull request that changes only docs/** or root-level *.md files runs
audit and nothing else. A Markdown file anywhere else counts as code.
scripts/ci-classify-areas.sh narrows the rest by area:
| Mode | Jobs | True for |
|---|---|---|
go |
lint, build-and-test, store-tests |
Everything except web/ (other than web/embed.go), android/ and docs/ |
web |
dashboard |
web/, cmd/aether/, internal/localgw/, internal/webgate/, go.mod, Makefile, .github/actions/, ci.yml |
caches |
The full suite on a push to main |
go.mod, go.sum, .golangci.yml, .github/, images/, the fonts the browser image copies from web/public/fonts/, the web and desktop package manifests and lockfiles, web/.bun-version, the Gradle files |
vm |
kvm in the merge queue and on a push to main |
internal/runtime/vm/, internal/vmimage/, the packages the VM code runs (internal/guest/, internal/acpkeeper/, internal/version/, internal/devexec/, internal/disk/, internal/egress/, internal/gitengine/, internal/rootfs/), images/kernel/, the kvm*_test.go files, the files of internal/runtime/ the VM engine shares (session.go, stream.go, managed_exec.go, docker_image.go, docker_cgroup.go), files named vm*.go or *_vm.go (and their tests) anywhere under internal/, go.mod, go.sum, vm.mk, kvm.yml and cmd/aether-server/vm.go |
VM code outside the VM packages goes in files named for the vm mode, so the
merge queue still runs kvm for it. make test-scripts fails when the KVM
tests compile a package that is neither in the list nor in the exclusions in
scripts/ci-classify-areas-test.sh, each with the reason it needs no VM run,
or when a test file with the kvm tag is outside the list. make lint builds
the VM tests on every pull request. The merge queue classifies the files
GitHub's compare lists between main and the group, because a group also
carries the entries ahead of it; a push compares with the commit before it;
past its 300-file limit, or when the lookup fails, kvm runs. The guest
kernel takes up to six minutes to build, so kvm restores it from a cache
keyed by a hash of images/kernel/ and the make kernel recipe, and builds
it only on a miss; runs on main save it.
The classifier runs from the pull request's (or merge group's) base revision,
so a pull request cannot change its own classification, and a renamed file
counts by its old path as well as its new one. A failed classification runs
both. A skipped job reports as skipped, not failed. A newer push cancels a
pull request's run (cancel-in-progress for pull_request only); main
pushes and merge queue runs, whose concurrency groups are unique per commit,
always run to completion.
To run the edge image jobs on a pull request branch, with write access to the repository:
gh workflow run edge-image.yml --ref <branch>
gh workflow run edge-image.yml --ref main -f ref=refs/pull/<n>/head # a pull request from a forkTo run the VM tests on a branch without the merge queue:
gh workflow run kvm.yml --ref <branch>A failed kvm fails suite and CI gate, so a nightly failure reaches
whoever GitHub notifies about the scheduled run: the person who last changed
its cron line.
To rerun only the failed jobs of a session: gh run rerun <session-id> --failed.
Running the Docker suites inside a session
You can work on Cloudfield from inside a Cloudfield run (an agent's container on a Cloudfield server). Two things differ there.
A session sets AETHER_* variables (AETHER_RUN_ID, AETHER_HARNESS,
AETHER_WORKSPACE_ID and others) that some tests read. internal/coordcli
fails with them set. Unset them for the test command:
env $(env | grep -o '^AETHER_[A-Z_]*' | sed 's/^/-u /') make testThe Docker suites need a Docker daemon inside the session. On a server with
Sysbox (environments.md) each session and
environment terminal has its own dockerd; check with docker info. Paths
the tests bind-mount resolve because the daemon and the test share one
filesystem, which is not true of the host's socket mounted into a container.
make test-integration # Go and git are in the standard image
make android # needs only the docker CLI and network
curl -fsSL https://bun.sh/install | bash # make test-e2e also needs Bun on PATH
(cd web && bun install --frozen-lockfile && bunx playwright install --with-deps chromium)
make test-e2eTo check Docker inside sessions on a Sysbox host, install Sysbox with
sudo sh scripts/install-sysbox.sh, then:
docker build -f images/standard/Dockerfile -t aether-standard:ci .
sudo env "PATH=$PATH" "HOME=$HOME" \
AETHER_NESTED_DOCKER_TEST_IMAGE=aether-standard:ci \
make test-integration INTEGRATION_PKGS=./internal/runtime \
INTEGRATION_RUN='^TestDockerNestedDocker$'
sudo sh scripts/nested-docker-smoke.sh aether-standard:ciTestDockerNestedDocker starts dockerd in a session container and runs a
container inside it; it skips unless AETHER_NESTED_DOCKER_TEST_IMAGE names
an image that has dockerd. scripts/nested-docker-smoke.sh runs the
resource limit tests against a nested daemon. It pulls busybox and Python
images, so it needs network access.