Skip to the content
docs

Troubleshooting and failure handling

Cloudfield was named Aether until November 2026. The commands, paths and hostnames in this guide still use that name. View this guide on GitHub.

Browse guides / Troubleshooting

What Cloudfield does when the host reboots, a disk fills, an agent hangs or a connection drops, and what you do next, for whoever runs the server and for members who find a session in an unexpected state. Terms such as environment, Enhanced mode and subsession are defined in Concepts.

A session is one agent working on one task in its own sandbox on the server. While it is live its status is running or needs-attention; when it ends it is completed, failed, interrupted, merged or abandoned. The reason shown beside the status says why it is there.

What happens when

When What you see What Cloudfield does What you do
The server restarts or is killed, and session sandboxes survive Sessions keep their status. Reattaches to every live sandbox. An Enhanced agent keeps running under its keeper, and the new server adopts its conversation, turn and open requests (Restarts and failures). A kill accepted before the crash is issued again. Nothing. When the keeper is gone, the conversation is restored in a fresh process and a turn that was cut off parks with turn interrupted: send a message to continue.
The host reboots, or Docker restarts without live-restore Live sessions read interrupted with server restarted: container exited <code> (or server restarted when the container is gone). Commits the partial work as wip: <task>, publishes the session branch and keeps the checkout. Pull the branch (aether pull <session-id>) or start a new session. An interrupted session cannot be reopened.
A server update is interrupted, or the new version cannot boot aether server update --status shows failed or rolled_back. Replaces nothing until every binary is verified. Rolls back a version that cannot finish booting. Recover from a failed server update.
Free disk falls toward the floor Launch refused with scheduler: insufficient disk headroom: .... Frees space on its own first, in a fixed order. Recover from disk pressure.
Host memory is low, or the server stalls on memory Launch refused with scheduler: insufficient host memory headroom: .... Starts nothing new. Running work is left alone. Recover from memory pressure.
An Enhanced Claude Code session is idle for --run-sleep-after (30 minutes) The Chat view shows Asleep since <time>; aether sessions shows (asleep). Stops the container with docker stop, freeing its memory and CPU, and keeps its disk and checkout. Nothing. The next message wakes it (Idle sleep).
A Standard or Enhanced session in a VM is idle for --run-sleep-after (5 minutes) As above. Saves the whole VM to disk with every process in it, freeing its memory and CPU. Nothing. The next message restores it (Sessions in VMs).
An agent goes silent needs-attention, reason stalled: no output or file changes for <duration>, state Waiting. An Enhanced session is Possibly stalled: stalled: possibly — no progress for <duration>, <phase>, transport open. Keeps the session supervised. It returns to running as soon as the agent writes output, makes progress or changes a file. Attach (aether attach <session-id>) or send a message. See Picking a stall threshold.
A Standard agent exits The session stays live and its terminal shows a login shell. Opens a shell in place of the agent. It never restarts the agent. Start the agent again in that shell, or close the session.
A Background agent exits completed, or failed with agent exited <code>: <last line of output>. Commits and publishes the branch, then removes the container. Read the full output in the session's transcript.
The agent hits its account's usage limit Usage limit reached on <owner>'s account; ... when it cannot continue. Moves the session to another account that has usage left. Usage limit mid-session.
The owner of the session's shared account revokes the share account share revoked: <owner> revoked access to their account; ... when it cannot continue. Moves the session to another account the launcher may use, or fails it and removes its container. Account sharing.
An agent update fails, or the vendor is unreachable Timeline note could not update <agent> from <version>: the updater exited <code>: <output>. Launches whatever is installed without waiting for the update. Tries again from that home after 15 minutes. Usually nothing. sudo aether-server config set agent-update false turns updates off.
An agent update an older Cloudfield interrupted left the agent's npm package missing The launch that starts the repair may not find the agent. Restores the copy in ~/.local/lib/.<agent>-update.*/previous in the background, before contacting npm. Launch again. The next launch finds the agent.
An SSH connection drops during an attach The CLI or dashboard disconnects. Nothing changes for the session: its terminal lives on the server. Input that never reached the server is dropped whole. Attach again. The full transcript replays before live output.
An aether forward connection drops The local port stops answering. Closes both ends of the tunnel. Run `aether forward <run-id
A launch cannot refresh the workspace mirror The launch fails with gitengine: mirror <kind> for branch "<branch>": <diagnostic>. No session is created. Never falls back to an older base on its own. Launch and mirror failures.
Your environment's main shell exits The environment terminal closes. Stops the terminal and keeps the stopped container. Open the environment again. It starts the container, or rebuilds it from your saved image.
An Enhanced session fails Disconnected, with enhanced session failed: <error> or enhanced session ended: .... Keeps the session up on its login shell. Enhanced sessions.
A session runs out of memory Timeline note Out of memory: the kernel killed a process in this session at its <n> GiB memory limit (peak <n> GiB). Nothing automatic. The killed process may be a tool, the agent or the shell. Check whether the agent is still running. Raise --run-memory if it keeps happening.
The kernel's file-watch limits are spent Server log gitengine: cannot watch checkout for file changes; polling for diff snapshots instead. Launches the session anyway and checks its files every 60 seconds. Raise file-watch limits.
Another process holds the database at startup store: begin migration <n>: database is locked (5) (SQLITE_BUSY) The server exits. Find and stop the other writer, then start again. Never delete the database.
The first start after an upgrade takes minutes Clients cannot connect and the server logs nothing yet. sudo aether-server update prints aether-server is migrating the database at schema <n> to <m>; waiting for it. Migrates the database before it serves. Migration 109 reads every stored event to build three indexes: on a 9 GB event log that took 1-2 minutes in testing, longer on a busy disk. Wait. A restart starts the migration over, and an update rolls back a release that starts twice without finishing. Before the update, free the database's size again plus a tenth of it: the update copies the database into server-update/, and the migration adds indexes of about 4% of the event log's size (400 MB on 9 GB in testing), as much again in aether.db-wal, and sort files in /var/tmp.
sudo aether-server update takes minutes while many sessions are open The update waits after the new server starts. The server log shows scheduler: recovery: reconnecting sessions sessions=<n> at_once=4, then scheduler: recovery: session started run=<id> and scheduler: recovery: session done run=<id> took=<duration> for each session, then scheduler: recovery: sessions done. Reconnects every open session before it counts as started, 4 at a time, so one slow session does not hold up the rest. An update still rolls back if a session has not finished 5 minutes after the database migration. Wait. sudo journalctl -u aether-server -f shows each session; a session with session started and no session done is the one still reconnecting, and a long took names a slow one.
The server is down when a schedule is due Nothing. Skips the missed occurrence and waits for the next one. Launch the session by hand if it matters.

A lost connection does not prove that a command failed. Before you retry something, check its files and processes. After a crash Cloudfield does not replay an agent's tool calls, but a Background Enhanced session is asked to continue its turn, so an action can run twice.

Session activity

A live session carries a state that says what it is doing. The dashboard and aether sessions show its label.

State Label Meaning
working Working A turn or tool is running.
waiting-agents Waiting for agents No turn runs, but the session waits on its own subagents, subsessions (waiting for <n> subsessions), or a waiting --for agents report.
waiting-ci Waiting for CI The agent reported waiting --for ci.
waiting-tools Waiting for process The agent waits on a background task it holds, or reported waiting --for process.
answered Answered The agent reported answered: it replied or discussed, with nothing pending. Never needs you and sends no notification.
waiting Turn ended, or Stale wait for a stale wait (aether sessions: Waiting) The turn ended with no report (turn ended without a report), the session stalled, a turn was cancelled, or a wait on agents went stale (no update for <duration>). Nothing is asked of you.
needs-you Needs you A person is needed: a needs-input or blocked report, an open permission request or question, a usage limit Cloudfield cannot continue from, an Enhanced turn that waits for a reply, or a broken wait.
finished Finished: review An interactive session reported success. Asks nothing of you.
failed Failed The session reported failure, or its turn failed.
disconnected Disconnected The Enhanced session failed or ended while the session is open.

When several apply, the first in this order wins: disconnected, needs-you, working, failed, finished, waiting-agents, waiting-ci, waiting-tools, answered, waiting. An answer never goes stale. A wait the agent reported with no condition goes stale after three times --stall-threshold (30 minutes by default) with no activity, unless the session still holds background tasks or, as a parent, has a live subsession or unread subsession mail. Its reason becomes no update for <duration>: <summary>. A CI or process wait keeps its state; an agents wait shows waiting, or waiting-agents while a subsession is live again. A wait goes stale once; the next turn starts it over. A wait Cloudfield infers from subsessions or background tasks lasts exactly as long as they do.

Waits Cloudfield can check

A waiting report can name conditions Cloudfield checks for the agent, such as a pull request's checks finishing, so it wakes the agent when one resolves. While the session shows such a wait, aether sessions <session-id> prints a wait line, the session record carries wait {kind, detail, verified_at, error}, and the state is judged by the evidence:

wait.kind When What you see Needs you
condition Every condition is pending and its source was read within twice its poll interval, or one resolved and its wake is on its way. The state, and <condition> (checked 2m ago). No. The wait never goes stale on inactivity.
source_unknown A pending condition was not read within twice its poll interval, or its reads fail. Waiting for CI (status unknown) and <condition>: status unknown (last read 12m ago). No. Cloudfield keeps retrying.
system_fault A condition errored or expired, for example the pull request is gone, and the agent is being told. Waiting for CI (wait fault) and the condition. No. The agent decides what to do next.
system_fault A condition resolved, errored or expired, and its wake was not delivered within 5 minutes. Needs you, reason wait broken: <condition> resolved, but the agent was not woken, and the delivery error. Yes. The agent will not move on its own. Send it a message.
system_fault A pending condition's source went unread for an hour. Needs you, reason wait broken: cannot read <condition> for over 1h, so the agent will not be woken. Yes.
system_fault Every read of the session's conditions failed for an hour, counted from the first failure or from the wait's report if later. A read that succeeds starts the hour over, and so does a server restart. Before that the wait is source_unknown. Needs you, reason wait broken: cannot read this session's wait conditions for over 1h, so the agent will not be woken, and the read error. Yes.
unverified The agent named no condition, or only ones that left pending before its current turn or whose wake was delivered. Once delivered, including a condition that resolved inside the agent's own report, the session waits on its agent, not on the condition. The state, as described above. It goes stale like a wait with no condition. No.

The dashboard words the same evidence as "Waiting on <condition> · checked 2m ago", "Waiting on <condition>: status unknown since 01:10" or "Wait fault: <condition>" (dashboard-frontend.md). wait reaches every workspace member, so it never holds what a read found or its error: those come from the owner's GitHub login and are in the session's wait_conditions, which only its owner and admins read in full (session waits). Session details lists each condition under Waiting until with what the server last read, when it checks next, its failures and whether the wake reached the agent, and Cancel ends a pending one (dashboard-frontend.md).

A broken wait is the only one that needs you, because only then can nothing but a person move the session. While another of its conditions is still pending or its wake is on its way, or the session still holds background tasks or, as a parent, has a live subsession or unread subsession mail, those can wake it: the session keeps its waiting state, marked (wait fault), and needs you only once they end. Its reason holds no time or source status, and names broken conditions in a fixed order, so it is announced once. A paused session is judged again when it resumes.

Each run.status event for a live session carries a cause naming what chose its state: turn, report (the agent's own report), turn-end, stall, wait-expired, wait-fault (a broken wait), workers, tasks-held, human, or disconnect. While the session shows a reported wait the event carries wait too. A wait whose source was read again at least a minute later is published as a same-status event with reread: true; aether timeline and the dashboard's activity feed leave it out unless filtered to run.status, changes.list leaves it out, and the dashboard's lanes and trace bars never draw it as a state change. A live session's record carries its latest cause too. aether timeline <session-id> prints it after the state, for example running -> needs-attention (waiting for 2 subsessions) · waiting-agents by workers. The dashboard ends each state's tooltip with it, for example Waiting for 2 subsessions since 3 minutes ago. Mail from its agents can wake it. Why: its subsessions are live or have unread mail for it.

An agent's blocked report reads blocked: <summary>, and its question reads needs input: <question>.

Review

A reported success puts the session up for review. The session page asks Your agent reports this task is finished. Mark complete?

  • Mark complete closes a live session as completed: it commits, publishes the branch and keeps the container like Close.
  • Not yet says the work is not finished: the session leaves review and reads Answered, "Not finished yet". Once the server accepts it, the session's Chat view opens with the composer focused so you can say what is left.
  • Review later keeps the session open and in review without a new notification; the prompt folds to one line, "Finished, kept for later", with Review now to open it again.

Any new turn clears the review, and a later success report asks again.

Before a close

Closing a session that others still depend on asks first. Close session…, Mark complete, Open in Standard and aether close read run.close_check (local-gateway.md) and list what the close affects: protection, an open pull request, commits that may not be merged, mail from other sessions, sessions waiting for this one to finish, subsessions that close with it and what closing each of those affects, and pending cleanup. With any of these closing turns into a danger button (Close as abandoned, or Close anyway for Mark complete) and Cancel holds the focus. A check that fails shows its error and still lets you close. Closing as merged skips the unmerged-commits warning: you are saying the work is merged.

aether close prints the warnings to stderr and asks Close anyway? [y/N] on a terminal; --force skips the question. Without a terminal it prints them and closes, so scripts keep working. A failed check prints its error. On a terminal it then asks too, unless the server is older than the CLI and has no check; there the close goes ahead:

CODE
Before closing run-d6ay38tj8j:
  pull_request: Pull request acme/api#7 was open, not merged, when Cloudfield last checked 2026-11-03T18:20:00Z
    https://github.com/acme/api/pull/7
Close anyway? [y/N]:

Needs you

One rule decides whether a session needs a person. The Needs you list, aether sessions --needs-you, aether wait (exit 3, except that a finished success exits 0, a failure exits 1 and a broken wait exits 5), the Insights count of sessions that need you, attention.list (local-gateway.md), push notifications and the assistant's notices all use it. The session status needs-attention alone does not: it means the agent is parked, which includes waiting on agents, CI or a process and a reported success.

A reported success is never a need. While its review is ready or later the session shows in Finished: review, its own group with its own count below Needs you, and the label Finished: review in the dashboard and aether sessions. attention.list still returns it, as kind ready with the reason Finished: review the result, after every item that asks something.

Only a top-level session can be Needs you or Finished: review. An attached subsession's reported success or failure is for its parent agent only, which hears of it through its wait on the subsession: it never counts as Needs you or Finished: review, never notifies and never moves the family, whether the parent is working, waiting, stopped, paused or ended. The parent carries the family: what surfaces is the parent's own state. A subsession's requests of a person (a permission, question, needs-input, blocked, a disconnect) still surface, listed with its parent. A detached subsession is a session of its own and follows the rules above. A subsession has no Mark complete of its own; it finishes by reporting to its parent. A watch someone asked for on a subsession still hears of its finish; the assistant's own follow of a session it launched or messaged does not. The dashboard reads a subsession's success as "Finished: reported to its parent" in a muted check, and aether sessions as Finished: reported to its parent.

Session Needs you Notifies
needs-you Yes Once, when it enters the state.
disconnected Yes Once.
failed, result not yet seen by the owner Yes, while the session is live Once: Failed, review the result.
failed, result seen No No.
Review ready, live or completed No: Finished: review Once per success, quietly: Finished: review the result, at normal urgency, never in the desktop badge.
Review marked Review later, Not yet or complete No No.
A broken wait (wait-fault) Yes Once: Wait broken, send a message to continue: <condition> ....
working or any waiting state, including turn cancelled No No.
Any state while the session is paused No Checked again on resume.

A need is announced once: opening the session or restarting the server does not repeat it. When it ends, or the session is handed to another member, each browser that showed it gets a silent No longer needs you (No longer awaiting review for a finished session) in place of the notification, and the desktop app closes its own. A server restart forgets which browsers showed a need, so one announced before it is not replaced.

Recover from disk pressure

Every launch checks that the Cloudfield data filesystem and Docker's storage keep a reserve free: by default the larger of 5 GiB and 5% of the filesystem, capped at 20 GiB, plus 1 GiB for this launch and each one already starting. When free space drops below the reserve plus 4 GiB, Cloudfield frees space on its own, on every poll and before refusing a launch, in this order, until it is back above:

  1. The saved memory of sleeping Enhanced session VMs whose agent restores its conversation, the longest asleep first (Sessions in VMs). The timeline notes Deleted this sleeping session's saved memory because disk space was low; it starts again from its disk when it next wakes, and restores its conversation. Sessions that could not start again from their disk, Standard ones among them, keep theirs.
  2. Containers of closed sessions, the soonest to expire first. The timeline notes Removed this closed session's container early because disk space was low; the session can no longer be reopened, and its branch and history are kept.
  3. Checkouts of every finished session, whatever their age. Skipped when --checkout-ttl is negative.
  4. Other members' stopped environments. Anything they installed and did not save with aether env save is lost; their homes are kept.
  5. Caches and unused standard and browser images.
  6. Leftover scratch directories.

If that is not enough, the launch is refused:

CODE
scheduler: insufficient disk headroom: <filesystem> has <n> free bytes; reserve <n> bytes plus <n> bytes per provisioning (<n> already in flight)

With session-runtime vm, a session's disks are sparse files that grow as it writes, so a running session can take the filesystem below the reserve. When the Cloudfield data filesystem first falls below the reserve, the server notes each VM's disk use, and every 5 seconds after that pauses every session whose disks have grown by 64 MiB or more since that point and stops its whole VM, so root in it cannot write either. Measuring against the point it fell below, not the previous check, catches a session filling the disk slowly as well as one filling it fast. Its timeline notes Paused: the server is low on disk, and this session's disks were filling it. It resumes on its own once there is space again. A session that grew less is never paused. A VM that is not a session, such as an environment terminal, is only logged (a VM that is not a session is filling the data filesystem). Once free space is back above the reserve plus 4 GiB, the sessions it paused resume and note Resumed: the server has disk space again.; one you paused, resumed, closed or woke yourself is yours, and the guard leaves it. A session that fills the disk again within 10 minutes of such a resume stays paused until you resume it (Paused again: ...). While a session is held, its Files view and closing it wait for the resume. Files a session deletes give their space back to the host. What a session writes to its home, which lives on the server rather than on a VM disk, is not counted, as it is not for a Docker session.

Attaching, messaging, pulling, closing, killing and deleting sessions keep working below the reserve, and so does reopening a closed session whose container is still there. To free space yourself:

  1. Open Insights → System to see what uses the space.
  2. Delete sessions you no longer need with Delete run or aether delete <session-id>. That removes the checkout, transcript and records; the published branch, the timeline and the cost totals stay.
  3. Free retained sandboxes with Free retained sandboxes… (admins).
  4. Retry the launch.
What grows What frees it
checkouts/ The checkout cleanup after --checkout-ttl, or deleting the session.
transcripts/ Deleting the session.
aether.db Deleting runs or the workspace.
repos/ Nothing. Every push and session branch is kept.

Archiving a session (Archive finished sessions… or aether archive <session-id>) only hides it and frees nothing. Archive finished sessions… also offers live sessions idle and done for over a day; it closes each one you leave checked before archiving it.

scheduler: host capacity unavailable: ... means Cloudfield could not measure a filesystem or host memory, for example a Docker daemon whose storage it cannot see or a probe that timed out. Fix what the error names; free space in df on a different machine does not count. When Docker does not answer, the error ends with the server's operator can check Docker with: sudo systemctl status docker. If a filesystem cannot be read at all, the disk check lets the launch through.

Recover from memory pressure

Every launch checks two things. Free memory must stay above the larger of 2 GiB and a tenth of the host's memory, plus 512 MiB for each launch already starting; otherwise the error ends <n> available bytes; reserve <n> bytes plus .... The server's own cgroup must also not be stalling on memory: its memory.pressure some avg60 must stay below 10%, which means tasks in the server's cgroup were blocked on memory for under 6 s of the last minute. Otherwise:

CODE
scheduler: insufficient host memory headroom: the server stalled on memory for <n>% of the last 60s (its cgroup's memory.pressure, some avg60); launches are refused at 10%

Only the server's cgroup counts. A session that thrashes inside its own memory limit reclaims within its own cgroup and does not refuse launches, although it raises the host-wide pressure that Insights → System charts (some avg10 of /proc/pressure/memory). To read the number the gate uses, for the service:

CODE
cat /sys/fs/cgroup"$(systemctl show -p ControlGroup --value aether-server)"/memory.pressure

For a foreground server, take its PID from the aether-server serve process (sessions run aether-server mcp processes that the host also sees):

CODE
cat /sys/fs/cgroup"$(cut -d: -f3 /proc/"$(pgrep -of 'aether-server serve')"/cgroup)"/memory.pressure

A foreground server shares its cgroup with the shell that started it, so memory-heavy jobs in that shell count toward the server's stall. Without cgroup v2, with the server in the root cgroup, or without memory.pressure for its cgroup, the stall check is skipped and the server logs that once. A file it cannot read or parse refuses launches with scheduler: host capacity unavailable, like the other probes.

To recover, close or kill sessions. A closed session's paused container still holds its memory; Free sandbox… releases it. An idle Enhanced Claude Code session, open or closed, frees its memory on its own after --run-sleep-after (Idle sleep), and so does any idle session in a VM, which also frees it as soon as it is closed. Running work is never stopped to make room.

Capture an unresponsive host

If the server stops answering while CPU and memory look free, capture evidence before you restart anything, reboot, prune Docker or kill processes. Run this in a root shell (sudo -s) on the server host. It writes to a private file and gives each Docker command five seconds:

CODE
umask 077
capture=$(mktemp /tmp/aether-capture.XXXXXX)
{
  date -Is
  systemctl show aether-server docker -p Id -p MainPID -p TasksCurrent -p TasksMax \
    -p LimitNOFILE -p LimitNPROC
  systemctl status aether-server --no-pager -n 80
  journalctl -u aether-server -u docker -b --no-pager -n 200
  journalctl -k -b --no-pager -n 200
  journalctl -u aether-server -u docker -b -1 --no-pager -n 200
  journalctl -k -b -1 --no-pager -n 200
} >"$capture" 2>&1

pid=$(systemctl show aether-server -p MainPID --value)
if [ "${pid:-0}" -gt 0 ]; then
  {
    printf '\n--- service process ---\n'
    ps -L -p "$pid" -o pid,tid,stat,nlwp,comm
    cat "/proc/$pid/limits"
    cat "/proc/$pid/status"
    printf 'fd_count='
    find "/proc/$pid/fd" -maxdepth 1 -type l | wc -l
    ls -l "/proc/$pid/fd" | sed -n '1,100p'
  } >>"$capture" 2>&1
fi

{
  printf '\n--- host pressure and capacity ---\n'
  cat /proc/pressure/cpu /proc/pressure/memory /proc/pressure/io
  cat /proc/loadavg /proc/sys/kernel/pid_max /proc/sys/kernel/threads-max
  cat /proc/sys/fs/file-nr /proc/sys/fs/inotify/max_user_watches
  cat /proc/sys/fs/inotify/max_user_instances
  ps -e -o stat= | sort | uniq -c
  printf 'host_threads='
  ps -eLf --no-headers | wc -l
  df -hT
  df -ih
  printf '\n--- bounded Docker probes ---\n'
  timeout 5s docker info
  timeout 5s docker ps --filter label=aether.managed=true \
    --format 'table {{.ID}}\t{{.State}}\t{{.Names}}' | sed -n '1,50p'
  timeout 5s docker stats --no-stream \
    --format 'table {{.Name}}\t{{.PIDs}}\t{{.CPUPerc}}\t{{.MemUsage}}'
} >>"$capture" 2>&1
printf '%s\n' "$capture"

The previous boot's logs need a persistent journal. Keep the capture out of the repository and redact it before you share it: log lines, command lines, paths and socket names can be sensitive. Do not send SIGQUIT to get a stack dump; it stops the server.

Recover from a failed server update

aether server update downloads and verifies both binaries before it replaces either, so a bad tag or a failed download changes nothing. A scheduled update that fails on the network or the release host is retried at the next idle moment, up to three times, then recorded as failed.

After the swap, the new version must open and migrate the database, then bind its SSH listener and recover the sessions within 5 minutes of migrating. If it fails to finish booting twice, or misses those 5 minutes, Cloudfield rolls back: the previous binaries go back into place, and if the schema already changed, the database copy taken before the update is restored. Every change made after that copy is lost.

  1. Read the result:

    CODE
    aether server update --status
    CODE
    update            rolled_back v0.8.0 -> v0.9.0
    error             v0.9.0 started 2 times without finishing its boot: store: apply migration 91: ...; database restored to its snapshot of 2026-10-16T09:12:03Z; changes made after it are lost
  2. Fix the cause the error names.

  3. Update again.

An error ending in rollback incomplete: <error> means a file could not be put back, and the server exits without starting either binary. A binary that cannot start at all cannot roll itself back either. Restore the previous binaries by hand as described in install.md.

Launch and mirror failures

A workspace with a mirror refreshes it from the upstream before every launch. If that fails, the launch is refused and no session is created; Cloudfield never starts from the previously accepted commit on its own. When an accepted commit exists, the CLI prints a one-off retry from it:

CODE
retry from cached <commit> with:
  aether session --agent <agent> --mode <mode> --workspace <workspace> --account <member-id> --cached-base <commit> -- <task>

--cached-base applies to that launch only, and is refused when the base has moved since. To repair the mirror itself (admins):

  1. Read the diagnostic:

    CODE
    aether workspace mirror status --workspace <workspace>
  2. Fix the cause from the server, not from your laptop. Check DNS and outbound access from the server host. For TLS errors, fix the server's git certificate trust; never turn verification off. For a deploy key, check that the key is still installed upstream.

  3. Refresh with aether workspace mirror refresh --workspace <workspace>.

A new or reconfigured mirror stays pending until an admin adopts a revision of its current generation, and every launch is refused until then. The refusal ends with the adopt command, and aether workspace mirror status and aether workspace mirror refresh print it with the workspace's values filled in:

CODE
sessions cannot launch until an admin adopts revision <sha>
check that it is the upstream commit you expect on <branch>, then run:
  aether workspace mirror adopt --workspace <workspace-id> --generation <n> --commit <sha> --yes

Compare <sha> with the upstream branch, for example with git ls-remote <source> <branch>, before you adopt it. In the dashboard the same step is Repository → Advanced → Review source mirror → Adopt candidate.

An upstream that was rewritten or diverged stays held (rewritten or diverged) until an admin reviews it and adopts the new revision with aether workspace mirror adopt --workspace <workspace> --generation <n> --commit <reviewed-sha> --yes. Published session branches stay fetchable while a mirror is failing.

Enhanced sessions

An Enhanced session talks to its agent through a separate process in the session sandbox. When that process fails, the container and its login shell stay up.

Failure What you see What you do
The session cannot start, for example because the agent is not signed in Disconnected, enhanced session failed: <error>. Messages are refused with scheduler: the enhanced session is not running: <error>. Fix the cause, then press Retry Enhanced in the Chat view.
The agent's process exits The turn in flight shows Turn interrupted; Disconnected, enhanced session ended: the agent's ACP server exited with code <n>, with ; stderr: <tail> when it printed any. Retry Enhanced.
The agent drops its conversation but keeps running, as Claude's adapter does when Claude Code dies Prompt failed with Session not found, then Agent conversation lost while Cloudfield restores the conversation. Lost again within 5 minutes: Disconnected, enhanced session failed: the agent lost its conversation again within 5 minutes: .... Send the failed message again. After a second loss, Retry Enhanced.
The agent refuses a turn with Authentication required Sign-in refused, or Agent signed out when the login file cannot sign the agent in, while Cloudfield restarts the agent in its container, then enhanced turn failed: ... Authentication required ...; <owner>'s <agent> login is signed out. Sign in, then send the message again: dashboard Agents page, or aether agent login <agent>, or ...; Cloudfield restarted the agent to reread <owner>'s login: send the message again. Refused again? Sign in: ... when the login file looks sound. The account owner's dashboard shows <agent> is signed out or <agent>'s sign-in was refused until the login works again. Send the message again when the reason says so. Otherwise the account's owner signs in on the Agents page or with aether agent login <agent>, then you send it again (harnesses.md).
run.launch answers <owner>'s <agent> login is signed out: <problem>. Sign in, then launch again: dashboard Agents page, or aether agent login <agent> The login file on the account cannot sign the agent in. Nothing was created. The account's owner signs in, then launch again. A workspace variable that signs the agent in, such as ANTHROPIC_API_KEY, lets the launch through (harnesses.md).
Root in one of your sessions took your shared home, for example a package install that ran systemd-tmpfiles Home restored with the owner and mode Cloudfield found, before the agent starts again. If restoring fails: Disconnected, enhanced session failed: the agent cannot read its home: .... Send the refused message again. Run sudo chown "$(id -u):$(id -g)" "$HOME" && sudo chmod u+rwx "$HOME" in the session's Terminal tab, which runs as the session's user, when restoring failed, or when the agent stays connected while its tools fail with permission denied or exit status 127, which no restore covers. After a failed restore, press Retry Enhanced.
The server restarts mid-turn The turn keeps running in the adopted conversation; a message queued behind it is sent again unless the agent already had it. Nothing.
The server restarts and cannot reattach to the agent's keeper, or the session's sandbox predates keepers Agent restarted with the reason, then Needs you, turn interrupted. Background tasks the agent was running are gone; commands in its aether-internal terminal terminals keep running. Send a message to continue.
You cancel a turn or press Stop Waiting, turn cancelled. Send a message when you want it to go on.
A Claude background task never ends running, working in background: <n> tasks. Press Stop.
The old session cannot be restored A new session starts; the item log says why. Repeat context the agent lost.

A Background Enhanced session has no one to retry it: any of these failures ends it as failed with agent exited 1: <cause>, for example agent exited 1: the agent's turn ended: cancelled. A server restart mid-turn leaves its turn running; when the server cannot reattach, it restores the session and asks the agent to continue.

Usage limit mid-session

When an agent reports that its account's usage limit stopped it, Cloudfield moves the session to another account that has usage left (usage fallback): the same session, checkout and branch continue on the new account's login, and a room note says so. When it cannot move, interactive sessions park as Needs you, a Background session fails, and the reason says why:

Reason Meaning
Usage limit reached on <owner>'s account; no other account is shared to fall back on No other account is shared with the launcher, or with the member whose home the session mounts, or listed in its --fallback order. Ask a teammate to share theirs, or wait for the limit to reset.
Usage limit reached on <owner>'s account; no fallback has usage left Every other account was read as exhausted or was already used by this session: the launcher's own, or a shared one whose login the server confirmed in the session's sandbox. Wait for the limit to reset, or ask a teammate to share theirs.
Usage limit reached on <owner>'s account; no account that may have usage left can run <agent> here Accounts with usage left, or whose usage could not be read, cannot launch the agent: their owner is not signed in to it, the agent is not installed for them, or the share no longer covers the session's home.
Usage limit reached on <owner>'s account; Cloudfield cannot read <agent>'s usage, so its sessions do not move to another account Only claude and codex sessions move (harnesses.md). Wait for the limit to reset, or relaunch on another account.
Usage limit reached on <owner>'s account; fallback is off for this session The session was launched with --no-fallback.
Usage limit reached on <owner>'s account; cannot continue on another account: <reason> The session cannot move: for example the agent reported no session to resume, it is a member's own agent definition, or the owner may no longer launch. Nothing was changed.
Usage limit reached on <A>'s account; continuing on <B>'s account failed: <error> The move started and failed. The session fails. start container: runtime: <path> in the container is inode <n>, not <login> in <B's home> ... means the container would have signed in with another login than B's; the rest of the error says whether Docker or the container runtime left B's login out.
Usage limit reached on <A>'s account; the session is on <B>'s account and its session could not be restored: <error> The session moved, but the Enhanced session did not come back. Send a message to start fresh.

A paused session, or one being closed or killed, is not moved. A move survives a server restart: the next start finishes it.

Raise file-watch limits

Cloudfield watches each session's checkout for changes with inotify. The kernel caps watches and instances per user, and a server running as root shares that budget with every root process in its session containers. When the kernel refuses, the session still launches and its file changes arrive up to 60 seconds late, which also delays stall detection.

no space left on device in the log means the watch limit is spent; too many open files means the instance limit, or the server's own file descriptors.

  1. Compare the server's open descriptors with its limit:

    CODE
    pid=$(systemctl show aether-server -p MainPID --value)
    sudo sh -c "ls /proc/$pid/fd | wc -l; grep 'open files' /proc/$pid/limits"

    If the count is at the limit, raise LimitNOFILE in the systemd unit instead.

  2. Otherwise raise the inotify limits:

    CODE
    printf 'fs.inotify.max_user_instances = 1024\nfs.inotify.max_user_watches = 1048576\n' |
      sudo tee /etc/sysctl.d/90-aether-inotify.conf
    sudo sysctl --system

Sessions launched afterwards watch again. A session that is already polling keeps polling until it is reopened.

Idle sleep

An Enhanced session of an agent whose conversation survives a container restart falls asleep after --run-sleep-after (default 30m) without activity: Cloudfield runs docker stop on its container, never docker rm. Memory and CPU are freed. Only Claude Code sleeps (harnesses.md, Idle sleep). Standard sessions never sleep, because starting their container again reruns the launch command with the original task; Background sessions are one-shot; Enhanced sessions of other agents stay awake until a test proves their conversation survives a restart. Sessions in VMs sleep differently: see Sessions in VMs.

Activity is any of these within the window: a turn, queued prompt, open permission request or question; agent output or a file change in the checkout; output or typing in the session's terminal or a shell tab; an aether ssh process or port forward still open; a Browser tab action such as navigating or clicking (an open Browser tab's own polling does not count); a member typing in the composer while another member watches the session, the only time the dashboard reports typing. A session that waits for you after its turn ended, or waits for CI or another session, is idle.

These keep a session awake however long it is quiet: a background task the agent started, such as a dev server; a command in the foreground of a terminal or shell tab, such as a dev server, an editor or a REPL (a shell at its prompt does not count); a session that waits for a process; a session you paused; a session that hit its usage limit; a session switching modes or accounts; and a session started in Standard and switched to Enhanced, whose container would rerun its launch command.

The session wakes when you send a message, change its model, mail reaches it, you type in its terminal, open a shell tab, use its Browser tab, start aether ssh or a port forward to it, run aether resume <session-id>, or choose Wake in the Chat view. Opening the session or its Browser tab without acting does not wake it. A port forward wakes the session, but a server that ran detached, such as one started with nohup, stopped with the sleep: the connection is refused until you start it again. The request that woke it waits while the container starts and the agent restores its conversation, then goes through. Two wakes at once start the container once. Waking counts against the same memory and disk checks as a launch. A refused or failed wake returns its error to the request that caused it, keeps the session asleep and shows the error under Asleep since. A wake never starts a new conversation: when the agent cannot restore its own, every wake fails with the agent's error; launch a new session on the session's branch, which is kept with its checkout.

A wake takes as long as starting the container and restoring the agent's conversation: in the real-agent test, a message to a sleeping Claude Code session was accepted about 1.6 seconds after it was sent on an idle host, and up to 23 seconds on a loaded one.

What a sleeping session keeps: the checkout and home on disk, including files the agent wrote but did not commit; the container's own disk layer; the conversation, the transcript and the branch.

What it loses: every process that was still running, which only a process Cloudfield cannot see can be, such as one started with nohup or &, or nested Docker containers such as those of docker compose up -d; open network connections; anything held only in memory; and the Claude Code session's scheduled wakeups (/loop, ScheduleWakeup, CronCreate), which live in the agent process. Shell tabs from before the sleep show as ended, and the terminal and the Chat view say processes were stopped. No turn is lost: a running turn, queued input or open request keeps the session awake.

The Browser tab's companion is removed when the session falls asleep; it shares the session container's network, which the stop ends. Its open pages, cookies and sign-ins, held only in its memory, are gone. The first Browser use after the wake starts a new companion.

Do not run docker container prune or docker system prune on the server: they delete the stopped containers of sleeping sessions. A sleeping session whose container was deleted ends on its next use, as a session whose container disappears does: its checkout and branch are kept.

A closed Enhanced Claude Code session's paused container is stopped the same way once it has been closed for --run-sleep-after. Relaunch or aether reopen <session-id> starts it again and restores the conversation. --run-container-ttl and --checkout-ttl work as before for sleeping sessions.

To keep every session awake, run sudo aether-server config set run-sleep-after 0, then sudo systemctl restart aether-server. Any other duration, such as 2h, changes the idle time the same way, for containers and VMs alike. There is no dashboard setting.

Sessions in VMs

With --session-runtime vm, a sleep saves the session's whole VM to disk, with every process in it, and a wake restores it where it was. Nothing is lost, so more sessions sleep, sooner:

  • Every agent sleeps, and Standard sessions too. Background sessions never do.
  • --run-sleep-after defaults to 5m for session VMs. A value you set applies to VMs and containers alike.
  • The conditions above keep a VM awake as they keep a container, and so do two more: a VM using more than a tenth of its CPU, such as a build or a test run left in the background, and a wakeup the agent set that has yet to fire. A save that fails leaves the session running; Cloudfield tries again after another --run-sleep-after.
  • A closed session is saved as soon as the close finishes, instead of staying paused in memory, unless --run-sleep-after is 0. It stays paused through the save, so nothing changes its files after the close's commit. Relaunch restores and resumes it. Closing or killing a sleeping session wakes it first, so its work is committed as for a running one.

A sleeping VM keeps everything: its processes, including dev servers, nested Docker containers such as a docker compose stack and the Browser tab's browser; its terminals and shell tabs, which reconnect on the wake; the agent process and its conversation; its disks. Connections to outside the session, such as a database client's or a port forward's, can end while it sleeps, and the programs holding them see them fail after the wake, as after a network outage. The VM's clock is set right on the wake.

Timers in a saved VM stop with it and count only time awake: on a test host, a 150-second sleep and a Node setTimeout started before a save fired 44 seconds after the wake, the time they still had left. So an Enhanced Claude Code session stays awake while a wakeup it set with ScheduleWakeup (as /loop does) or a one-shot CronCreate has yet to fire, and while a recurring CronCreate job runs, until CronDelete. Timers that Cloudfield cannot see, such as a Standard session's or a sleep in a script, wait for the next wake. Cloudfield's own wakes still wake a sleeping session, Standard or Enhanced: mail, aether-internal report --until conditions and watches reach it as any message does.

A Standard session's Chat and Terminal views, and an Enhanced session's Chat view, show Asleep since with a Wake button while it sleeps.

A wake takes a second or two on local NVMe: on a test host, a message to a sleeping Claude Code or Codex session running a dev server and a docker compose stack was accepted 0.7 to 1.5 seconds after it was sent, Relaunch of a closed one took 3 seconds, and a start from disk after a failed restore 5 seconds. The Chat view and the session's terminal say Cloudfield had saved this idle session to disk and restored it.

When the saved state cannot be restored, because the server now runs another Cloud Hypervisor or guest kernel, or the file is gone or damaged, an Enhanced session of an agent that restores its conversation starts again from its disk: processes that were running end, the conversation is restored, and the Chat view gives the restore's error. Any other session, a Standard one among them, stays asleep: the wake fails with the restore's error, and the VM keeps its disk and saved state. Launch a new session on its branch, or wake it again. A VM a failed save left running is taken up as it is on the next wake.

A saved VM takes about as much disk as the memory its processes use, after the guest drops its page cache: 0.6 to 1.3 GB for those test sessions in 2 GiB VMs. When disk runs low, Cloudfield deletes the saved memory of the longest-sleeping sessions first (Recover from disk pressure). A server restart or update leaves sleeping VMs asleep; the next use wakes them.

What stays after a session ends

Action Container Checkout Reopen
Close (aether close <session-id> --outcome merged or abandoned) Paused and kept for --run-container-ttl (7 days). It still holds its memory, until an Enhanced Claude Code session's container is stopped after --run-sleep-after (Idle sleep). A session VM is saved to disk at once instead. Kept while the container is kept. Removed after that, but no sooner than --checkout-ttl (72 hours) after the session ended: at 7 days with the defaults. Yes, until the deadline: Relaunch or aether reopen <session-id>. The deadline is cleared while the session is live; the next close sets a new 7 days.
A Background session finishes Removed at once. Kept for --checkout-ttl (72 hours) after the session ends. No.
Kill run (aether kill <session-id>) Stopped and removed. Kept for --checkout-ttl after the session ends. No.
Free sandbox… Removed. History stays. Kept until --checkout-ttl after the session ended. No.
Delete run (aether delete <session-id>) Removed. Removed with the transcript and records. No.

Every row removes the session's browser companion, the Chromium container behind its Browser tab. A relaunched session creates a new companion when its Browser tab is next used. A companion left behind by a crash or a failed cleanup step is removed at server start and then hourly: when its session has ended or is unknown to the server, or when the session container it shares a network with is gone. A companion an open session can still reach is never removed. The sweep reads only this server's journals under <data-dir>/scheduler/browser/; a companion with no journal there, such as one another server on the same Docker daemon started, is left alone.

The session branch is published before a checkout is removed, so cleanup never loses committed work. When disk runs low, Cloudfield removes both sooner (Recover from disk pressure). Relaunching resumes the same container and checkout; it restarts no process that had stopped. When retention ends, a closed session's reason reads retained container expired or retained container unavailable; when low disk space removes the container early it reads retained container reclaimed for disk space.

Cloudfield captures evidence before it removes a container or checkout, and retries a failed capture every --poll-interval. If the checkout is already gone, no retry can capture it: cleanup finishes without an evidence packet and the reason gains evidence unavailable: session checkout missing.

A checkout over the evidence storage limit cannot be captured, so the session gets no evidence packet. Cloudfield commits the checkout's files as a wip: commit, publishes the session branch, adds evidence unavailable: evidence storage limit exceeded to the session's reason (shown by aether sessions <session-id> and the dashboard), and only then removes the checkout. If the commit, the publish or that reason write fails, the checkout stays.

Removing a checkout first copies the session's diff history out of it into checkouts/<run-id>.diffsnap/, which stays; this reads the session's diff events from the event log. If any step fails, the checkout stays and the server logs scheduler: checkout gc: capture before cleanup with the session, the attempt, the next retry time and the error. Retries wait 15 minutes, doubling to at most 6 hours, and continue until cleanup succeeds; nothing is removed without its history. The wait restarts from 15 minutes when the server restarts.

A member's containers share one uid:gid mapping, so a launch under another mapping is refused while one of them is live. A closed session whose cleanup is still retrying counts only while its container exists: Cloudfield asks Docker before refusing, and the refusal names it as session <id> (cleanup pending) and includes any Docker error.

Tuning

aether-server serve reads these from /etc/aether/server.conf. Set one by its flag name, then restart. For durations and ceilings, 0 means the default.

CODE
sudo aether-server config set stall-threshold 20m
sudo systemctl restart aether-server
Flag Default What it changes When to change it
--stall-threshold 10m How long a live session may go with no agent output, file change or working report before it parks as stalled. Waiting states fall back to waiting after three times this. See below.
--first-event-grace 3m How long an Enhanced turn may wait for the agent's first update before it parks as stalled. At most --stall-threshold. Raise for agents that think for minutes before they stream anything.
--stream-quiet-grace 5m How long an Enhanced turn with no tool running may go without agent progress before it parks as stalled. At most --stall-threshold. As above.
--poll-interval 30s How often stalls, retention and disk are checked. Values above 1m act as 1m. Keep it at a third of the threshold or less.
--checkout-ttl 72h How long a finished session's checkout is kept after the session ends. It also stays while the session's container is kept. Negative keeps every checkout, and disk cleanup leaves them alone. Lower on a small disk.
--run-container-ttl 168h How long a closed session's container is kept for reopening. Negative removes it at once. Lower when paused containers hold too much memory.
--run-sleep-after 30m; 5m for session VMs How long an idle Enhanced session of an agent that can restore its conversation, open or closed, keeps its container running, and how long any idle session VM runs before it is saved (Idle sleep). 0 keeps every session awake. Lower on a host short of memory; 0 when processes in idle sessions must keep running.
--min-free-disk automatic The free-space reserve per filesystem, in bytes. Negative turns the disk check off; the memory check stays. See below.
--run-cpus 8, or the host's CPU count if lower CPU ceiling per session and environment. Lower to share a small host.
--run-memory 8 GiB Memory ceiling per session and environment, in bytes. Swap adds a quarter of it. Raise when sessions hit out-of-memory kills.
--run-pids 4096 Process and thread ceiling per container. Raise for builds that start many processes.
--agent-update true Update a shipped agent in the member's home, in the background, when a session launches. Turn off on a host without internet access, or to hold a version.

The CPU, memory and process ceilings apply to containers created after the change. A host-wide out-of-memory kill takes agent processes before the server (OOMScoreAdjust=-500 on the server, 500 in containers). Under CPU and disk contention the server gets a hundred shares to each container's one (install.md).

Picking a stall threshold

For agents that report their turns (claude, codex, opencode, pi and omp), a finished turn parks the session at once, and the threshold only catches an agent that hangs mid-turn. For other agents, silence is the only signal, and a long build or slow tool call looks the same as a hang.

  • Too low, and long tool calls park healthy sessions as stalled.
  • Too high, and a wedged agent wastes hours before anyone notices.

10 minutes suits interactive sessions. Raise it for Background sessions with long builds. Lower it to a minute or two for many short scripted sessions.

An Enhanced session is judged on its agent's progress: messages, thoughts, tool calls, plans, subagents, compaction, background task progress and permission requests. Cloudfield's own calls to the agent, the history a restored session replays, and mode, command and usage updates are not progress. How long it may stay quiet depends on where the turn is, its phase:

Phase Limit
awaiting-first-event: the prompt was sent, nothing came back --first-event-grace
streaming: the agent replied, no tool is running --stream-quiet-grace
tool-running: a tool call or a compaction is open --stall-threshold
tool-running between turns: background tasks hold the session three times --stall-threshold
awaiting-input, between-turns, or the turn that asks for a missing report --stall-threshold

A stall that names a phase is a possible stall: the dashboard shows Possibly stalled, never Needs you or Disconnected. Disconnected needs the agent process or its connection to end. An agent that thinks for minutes without streaming a thought parks at the short limit and returns to running with its next update. A resume or a server restart starts the quiet time over.

Cloudfield restarts nothing on its own. The session's Chat view shows the agent's last progress, the phase and the connection, and offers:

  • Stop turn cancels the turn. It costs nothing.
  • Open trace opens the turn's model calls and tool calls.
  • Resend last prompt… sends your last prompt again after you confirm. It costs tokens, and the agent may repeat tool actions it already took. A turn that is still running finishes first, so stop it to send now.

Messaging a stalled session usually gets it going. The echo of your message does not count as agent output, so a hung agent still reads as stalled afterwards.

Picking a disk floor

The automatic reserve suits most hosts; raise --min-free-disk for large checkouts or a filesystem shared with other services. Host memory is checked separately: a launch needs the larger of 2 GiB and 10% of RAM available, plus 512 MiB for it and each launch already starting. Memory ceilings are not reservations, so idle sessions can share a host. Refusals and the cleanup order are in Recover from disk pressure.

Limits

  • aether.db does not sync every write to disk. A killed server loses no committed write, but a power loss or kernel crash can drop the last few; the database stays consistent.
  • A Standard agent survives a server restart only if its own process survived. Extra terminals in a session lose their old output streams.
  • Cloudfield does not restart an agent killed for memory, and an exit code alone does not prove memory was the cause. A cgroup v1 host posts no out-of-memory note; check docker inspect and the kernel log there.